Resources and autoscaling

Vertical Pod Autoscaler

1.What is VPA ?

A feedback loop that moves a Pod's requests (and limits in proportion) toward its real usage. It never changes replicas. An add-on from kubernetes/autoscaler, not core : two CRDs (VerticalPodAutoscaler, ...Checkpoint) and three controllers you install. It reads usage from metrics-server. Beside it, HPA adds Pods and Karpenter adds nodes.

VPA changes Pods, not your DeploymentThe template keeps the old requests, so look at the Pod. Remove VPA and new Pods get the template values back.

VPA 1.8.0 runs on Kubernetes 1.36 to 1.38. API autoscaling.k8s.io/v1 only (v1beta2 not served since 1.3.0).

2.Three components, one loop

metrics-server usage, every 1m recommender Checkpoint history status.recommendation VPA object both read the target admission controller (webhook) updater patch on CREATE evict resize Pods
ComponentDoes
recommenderevery 1m reads metrics.k8s.io, writes status and checkpoints. One active, metrics :8942
updaterevery 1m evicts or resizes out-of-date Pods. Evicted Pods get new values from the webhook. One active, metrics :8943
admission controllerwebhook on Pod CREATE, failurePolicy: Ignore. Patches requests, limits. Several active, metrics :8944

3.The VPA object

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata: {name: my-app-vpa}
spec:
  targetRef:
    {apiVersion: apps/v1, kind: Deployment, name: my-app}
  updatePolicy: {updateMode: "Recreate"} # not "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: "*"
      minAllowed: {cpu: 100m, memory: 50Mi}
      maxAllowed: {cpu: 1, memory: 500Mi}
      controlledResources: ["cpu", "memory"]
      controlledValues: RequestsAndLimits
  • targetRef : any controller with a /scale subresource, or a well-known kind (Deployment, StatefulSet, DaemonSet, Job...). Its Pods must be owned directly by it, so operator Pods need a CR with /scale.
  • Per container mode: Off skips a container (its default, Auto, is not the deprecated one).
  • controlledValues: RequestsOnly leaves limits alone.
  • updatePolicy.minReplicas overrides the updater's floor. evictionRequirements filter by direction, for example only evict to scale up.

4.Update modes

ModeAt creationRunning PodsStatus
Offnonorecommend only
Initialyesnostable
Recreateyesevicts (PDBs apply)default when unset
InPlaceOrRecreateyesin place first, eviction as fallbackGA in VPA 1.6.0
InPlaceyesin place only, never evictsalpha in 1.7.0, gate InPlace
Autoyesevictsdeprecated in 1.5.0, same as Recreate

5.Install

  • Script : install metrics-server first, then ./hack/vpa-up.sh from a clone of kubernetes/autoscaler. It installs into kube-system and needs OpenSSL 1.1.1+. vpa-down.sh removes it.
  • Helm : chart 0.13.0, "not ready for production" in its README. It sets --in-place-skip-disruption-budget=true, unlike the binary. Recommender only : --set updater.enabled=false --set admissionController.enabled=false.
In short : VPA moves requests toward usage. The recommender only writes numbers; the updater and the webhook apply them, by updateMode and by which of them you installed.

1.Reading a VPA

kubectl get vpa : MODE, CPU, MEM (first container's target), PROVIDED; -o wide adds MinReplicas, OOMSeconds. Status has four numbers per container :

  • targetwhat the webhook applies.
  • lowerBound"not guaranteed to be sufficient".
  • upperBoundabove it is likely waste.
  • uncappedTargetbefore minAllowed / maxAllowed.
Conditions
  • RecommendationProvided
  • LowConfidence
  • NoPodsMatched
  • FetchingHistory

2.The recommender recipe

  1. Grouped per VPA and container name, not Pod.
  2. CPU : a decaying histogram, 24h half-life.
  3. Memory : one peak per 24h interval, 8 intervals.
  4. Target p90, lower p50, upper p95, then +15%.
  5. Short history widens bounds : upper x3 at 12h, x2 at 24h, x1.14 at a week; lower x0 with none.
  6. Per-Pod floor 25m / 250Mi, split across containers.
  7. Caps : min/maxAllowed, LimitRange.

An OOMKill adds a synthetic sample of max(used + 100Mi, used x 1.2). The floor unit is MiB, even though the flag says -mb.

Recommender flagDefaultEffect
--recommender-interval1mloop
--cpu-histogram-decay-half-life24hday-old weighs half
--memory-aggregation-interval x -count24h x 8a peak a day
--target-cpu-percentile, memory0.9target
--recommendation-margin-fraction0.15+15% on all
--pod-recommendation-min-cpu-millicores, -memory-mb25 / 250floor, in MiB

3.When the updater acts

lowerBound target upperBound belowact insideno action aboveact also : running 12h+ and a change of 10%+, or OOMKilled after running under 10 minutes
One replica is never evicted with defaults. Set updatePolicy.minReplicas: 1 if you mean it.
  • Candidate : a request outside [lowerBound, upperBound] (a missing request counts as outside), or one of the two rules in the box.
  • The largest summed change goes first.
  • Safety : at least minReplicas (2) live, at most half evicted at once (tolerance 0.5), PDBs through the Eviction API, an optional rate limit, and no evictions if the admission controller looks dead.

4.In place, and its fallback

InPlaceOrRecreate resizes all containers at once through pods/resize (In-Place Resize). It evicts instead :

  • at oncefailed PATCH (a QoS change is rejected), Infeasible, or reason Error.
  • later5 min Deferred, or 1 hour in progress (fixed, not flags).

The fallback still obeys minReplicas and tolerance. The kubelet refuses a memory-limit cut below current usage (reason Error, see In-Place Resize), so VPA evicts instead. InPlace (alpha) never evicts; it retries an infeasible size once the recommendation drops.

5.Limits and LimitRange

  • RequestsAndLimits keeps the request:limit ratio. Request 10m with limit 1 CPU (ratio 100) : at request 300m the limit becomes 30 CPU. Use RequestsOnly, or keep them close (Requests and Limits).
  • A limit of 0 is not "unlimited" : with a request set, it is rejected. Omit the limit.
  • Over a LimitRange max, the limit drops to max and the request in proportion.
  • If the VPA policy and a LimitRange conflict, VPA's docs say VPA wins.
In short : Targets are p90 of decayed usage plus 15%; bounds widen on short history. The updater acts outside the bounds, after 12h with a 10% change, or on a quick OOM, and by default never below two replicas.

1.VPA with HPA

Both on CPU : a loop VPA raisesCPU request HPA utilizationdrops HPA scales in per-Pod usagerises Safe : split by metric VPA : memory HPA : CPU Deployment requests replicas
PatternSafe ?Why
VPA on CPU, HPA on CPU utilizationnothe loop
VPA controlledResources: ["memory"], HPA on CPUyesupstream pattern
HPA on custom or external metrics (RPS, queue depth, KEDA), VPA on bothyesdifferent signals
VPA Off, any HPAyesnothing applied

HPA utilization is usage divided by request, and VPA moves the request. More in HPA in Depth.

2.Recommender only

  • Install only the recommender, give every workload a VPA with updateMode: "Off", and change requests by hand in Git : no evictions, no webhook.
  • Goldilocks : a dashboard on top of it.
  • kube-state-metrics removed its VPA metrics in v2.9.0 : export status.recommendation with CustomResourceState.
max_over_time(container_memory_rss
  {namespace="NS",container="C"}[7d])
max_over_time(container_memory_working_set_bytes
  {namespace="NS",container="C"}[7d])

PromQL : close together, the target is honest. RSS far below : page cache inflates it.

3.Where the numbers mislead

  • Memory follows working set, page cache included : file-heavy workloads look far too big.
  • Bursty, short-lived workloads get undersized : p90 + 15% misses rare peaks (Jobs).
  • A recommender restart forgets the current interval's peak so far.
  • Identical tiny values are the floor, not data.
  • Labels matching stray Pods pollute the aggregate. Two VPAs on one Pod : undefined.
  • Pod-level resources are not supported.
  • Bigger than any node or quota : Pods stay Pending. Set maxAllowed (Karpenter).

4.Operating it

  1. Read release notes first : the 1.7.0 recommender reset recommendations from checkpoints (fixed in 1.7.1).
  2. Back up verticalpodautoscalercheckpoints first.
  3. After the roll, prove history loaded : log line Fetching VPA checkpoints ... count=N. PROVIDED=True proves nothing (old status stays).
  4. Do not resize from numbers within 24h of a recommender restart.

The webhook's failurePolicy: Ignore silently skips mutation while it is down.

Seen in practice
  • Memory limits copied from VPA on a file-heavy service came out many times its resident memory. Bursty proxies sized from VPA came out too small : VPA saw only steady state.
  • Pitfall : treating the VPA target as the size.
  • Do better : use VPA as one input. Check memory against peak RSS, size bursty and data-path Pods from peaks. Keep a CPU limit off the recommender : its own throttling skews every number.

5.Commands

CommandWhat it shows
kubectl get vpa -AMODE, CPU, MEM, PROVIDED for every VPA
kubectl describe vpa web -n shoprecommendation, conditions, eviction events
kubectl get vpa web -n shop -o jsonpath='{.status.recommendation.containerRecommendations[*].target}'just the targets
kubectl get pod <p> -o jsonpath='{.metadata.annotations}'what the webhook applied (vpaUpdates)
kubectl get verticalpodautoscalercheckpoints -A -o json > checkpoints.jsonback up before an upgrade
kubectl -n kube-system logs deploy/vpa-recommender | grep -E 'Fetching VPA checkpoints|Error while loading checkpoint'did history load
In short : Keep VPA and HPA on different signals, start with recommender-only and updateMode: "Off", and check memory targets against resident memory before you trust them.