Resources and autoscaling
Vertical Pod Autoscaler
1.What is VPA ?
A feedback loop that moves a Pod's requests (and limits in proportion) toward its real usage. It never changes replicas. An add-on from kubernetes/, not core : two CRDs (VerticalPodAutoscaler, ...Checkpoint) and three controllers you install. It reads usage from metrics-server. Beside it, HPA adds Pods and Karpenter adds nodes.
VPA 1.8.0 runs on Kubernetes 1.36 to 1.38. API autoscaling. only (v1beta2 not served since 1.3.0).
2.Three components, one loop
| Component | Does |
|---|---|
| recommender | every 1m reads metrics., writes status and checkpoints. One active, metrics :8942 |
| updater | every 1m evicts or resizes out-of-date Pods. Evicted Pods get new values from the webhook. One active, metrics :8943 |
| admission controller | webhook on Pod CREATE, failurePolicy: Ignore. Patches requests, limits. Several active, metrics :8944 |
3.The VPA object
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata: {name: my-app-vpa}
spec:
targetRef:
{apiVersion: apps/v1, kind: Deployment, name: my-app}
updatePolicy: {updateMode: "Recreate"} # not "Auto"
resourcePolicy:
containerPolicies:
- containerName: "*"
minAllowed: {cpu: 100m, memory: 50Mi}
maxAllowed: {cpu: 1, memory: 500Mi}
controlledResources: ["cpu", "memory"]
controlledValues: RequestsAndLimits
targetRef: any controller with a/scalesubresource, or a well-known kind (Deployment, StatefulSet, DaemonSet, Job...). Its Pods must be owned directly by it, so operator Pods need a CR with/scale.- Per container
mode: Offskips a container (its default,Auto, is not the deprecated one). controlledValues: RequestsOnlyleaves limits alone.updatePolicy.minReplicasoverrides the updater's floor.evictionRequirementsfilter by direction, for example only evict to scale up.
4.Update modes
| Mode | At creation | Running Pods | Status |
|---|---|---|---|
Off | no | no | recommend only |
Initial | yes | no | stable |
Recreate | yes | evicts (PDBs apply) | default when unset |
InPlaceOrRecreate | yes | in place first, eviction as fallback | GA in VPA 1.6.0 |
InPlace | yes | in place only, never evicts | alpha in 1.7.0, gate InPlace |
Auto | yes | evicts | deprecated in 1.5.0, same as Recreate |
5.Install
- Script : install metrics-server first, then
./hack/vpa-up.shfrom a clone ofkubernetes/. It installs intoautoscaler kube-systemand needs OpenSSL 1.1.1+.vpa-down.shremoves it. - Helm : chart 0.13.0, "not ready for production" in its README. It sets
--in-place-skip-, unlike the binary. Recommender only :disruption-budget=true --set updater.enabled=false --set admissionController.enabled=false.
updateMode and by which of them you installed.1.Reading a VPA
kubectl get vpa : MODE, CPU, MEM (first container's target), PROVIDED; -o wide adds MinReplicas, OOMSeconds. Status has four numbers per container :
- targetwhat the webhook applies.
- lowerBound"not guaranteed to be sufficient".
- upperBoundabove it is likely waste.
- uncappedTargetbefore
minAllowed/maxAllowed.
RecommendationProvidedLowConfidenceNoPodsMatchedFetchingHistory
2.The recommender recipe
- Grouped per VPA and container name, not Pod.
- CPU : a decaying histogram, 24h half-life.
- Memory : one peak per 24h interval, 8 intervals.
- Target p90, lower p50, upper p95, then +15%.
- Short history widens bounds : upper x3 at 12h, x2 at 24h, x1.14 at a week; lower x0 with none.
- Per-Pod floor 25m / 250Mi, split across containers.
- Caps :
min/maxAllowed, LimitRange.
An OOMKill adds a synthetic sample of max(used + 100Mi, used x 1.2). The floor unit is MiB, even though the flag says -mb.
| Recommender flag | Default | Effect |
|---|---|---|
--recommender- | 1m | loop |
--cpu-histogram- | 24h | day-old weighs half |
--memory- x -count | 24h x 8 | a peak a day |
--target-cpu-, memory | 0.9 | target |
--recommendation- | 0.15 | +15% on all |
--pod-recommendation-, -memory-mb | 25 / 250 | floor, in MiB |
3.When the updater acts
updatePolicy.minReplicas: 1 if you mean it.- Candidate : a request outside [lowerBound, upperBound] (a missing request counts as outside), or one of the two rules in the box.
- The largest summed change goes first.
- Safety : at least
minReplicas(2) live, at most half evicted at once (tolerance 0.5), PDBs through the Eviction API, an optional rate limit, and no evictions if the admission controller looks dead.
4.In place, and its fallback
InPlaceOrRecreate resizes all containers at once through pods/resize (In-Place Resize). It evicts instead :
- at oncefailed PATCH (a QoS change is rejected),
Infeasible, or reasonError. - later5 min
Deferred, or 1 hour in progress (fixed, not flags).
The fallback still obeys minReplicas and tolerance. The kubelet refuses a memory-limit cut below current usage (reason Error, see In-Place Resize), so VPA evicts instead. InPlace (alpha) never evicts; it retries an infeasible size once the recommendation drops.
5.Limits and LimitRange
RequestsAndLimitskeeps the request:limit ratio. Request 10m with limit 1 CPU (ratio 100) : at request 300m the limit becomes 30 CPU. UseRequestsOnly, or keep them close (Requests and Limits).- A limit of
0is not "unlimited" : with a request set, it is rejected. Omit the limit. - Over a LimitRange
max, the limit drops to max and the request in proportion. - If the VPA policy and a LimitRange conflict, VPA's docs say VPA wins.
1.VPA with HPA
| Pattern | Safe ? | Why |
|---|---|---|
| VPA on CPU, HPA on CPU utilization | no | the loop |
VPA controlledResources: ["memory"], HPA on CPU | yes | upstream pattern |
| HPA on custom or external metrics (RPS, queue depth, KEDA), VPA on both | yes | different signals |
VPA Off, any HPA | yes | nothing applied |
HPA utilization is usage divided by request, and VPA moves the request. More in HPA in Depth.
2.Recommender only
- Install only the recommender, give every workload a VPA with
updateMode: "Off", and change requests by hand in Git : no evictions, no webhook. - Goldilocks : a dashboard on top of it.
- kube-state-metrics removed its VPA metrics in v2.9.0 : export
status.with CustomResourceState.recommendation
max_over_time(container_memory_rss
{namespace="NS",container="C"}[7d])
max_over_time(container_memory_working_set_bytes
{namespace="NS",container="C"}[7d])
PromQL : close together, the target is honest. RSS far below : page cache inflates it.
3.Where the numbers mislead
- Memory follows working set, page cache included : file-heavy workloads look far too big.
- Bursty, short-lived workloads get undersized : p90 + 15% misses rare peaks (Jobs).
- A recommender restart forgets the current interval's peak so far.
- Identical tiny values are the floor, not data.
- Labels matching stray Pods pollute the aggregate. Two VPAs on one Pod : undefined.
- Pod-level
resourcesare not supported. - Bigger than any node or quota : Pods stay Pending. Set
maxAllowed(Karpenter).
4.Operating it
- Read release notes first : the 1.7.0 recommender reset recommendations from checkpoints (fixed in 1.7.1).
- Back up
verticalpodautoscalercheckpointsfirst. - After the roll, prove history loaded : log line
Fetching VPA checkpoints ... count=N.PROVIDED=Trueproves nothing (old status stays). - Do not resize from numbers within 24h of a recommender restart.
The webhook's failurePolicy: Ignore silently skips mutation while it is down.
- Memory limits copied from VPA on a file-heavy service came out many times its resident memory. Bursty proxies sized from VPA came out too small : VPA saw only steady state.
- Pitfall : treating the VPA target as the size.
- Do better : use VPA as one input. Check memory against peak RSS, size bursty and data-path Pods from peaks. Keep a CPU limit off the recommender : its own throttling skews every number.
5.Commands
| Command | What it shows |
|---|---|
| kubectl get vpa -A | MODE, CPU, MEM, PROVIDED for every VPA |
| kubectl describe vpa web -n shop | recommendation, conditions, eviction events |
| kubectl get vpa web -n shop -o | just the targets |
| kubectl get pod <p> -o | what the webhook applied (vpaUpdates) |
| kubectl get verticalpodautoscalercheckpoints -A -o json > checkpoints.json | back up before an upgrade |
| kubectl -n kube-system logs deploy/ | did history load |
updateMode: "Off", and check memory targets against resident memory before you trust them.