Resources and autoscaling
Horizontal Pod Autoscaler in Depth
1.Beyond the basics
- The formula :
desired = ceil(current x value / target); a ratio of 0.9 to 1.1 changes nothing (tolerance 0.1). Deployments and Scaling works an example. Everything else here stops that formula from flapping. autoscaling/v2is GA since 1.23 (v2beta2not served since 1.26).autoscaling/v1is still served, CPU only; v2-only fields survive as annotations.- It scales anything with a
scalesubresource : Deployment, StatefulSet, ReplicaSet, CRDs. Never a DaemonSet. - A controller loop in kube-controller-manager, every 15s. In 1.37 a new HPA, or one whose spec changed (
HPAGeneration, beta, on), is reconciled at once.
2.One sync
- Read the target's
/scale: replicas and selector. - Out of
min..maxtoday ? Clamp, without reading metrics. currentin the formula isscale.spec.replicas, the desired count, not the Pods running now.- Every metric failed, or some failed and the rest say "fewer" : an error, no change.
- Some failed and the rest say "more" : scale up, and still report the error.
3.Which Pods count
- Pods being deleted are ignored, Failed Pods discarded (phases : Pods).
- Pending Pods (every metric type) and Pods with no sample are set aside.
- CPU only : a not-yet-ready Pod is set aside. In its first 5 minutes if NotReady or its last sample predates its readiness change; after that only if it was never Ready.
- The ratio comes from the rest. Inside tolerance : stop.
- Recompute conservatively. If that flips direction or lands inside tolerance : no change (status still shows the plain average).
On a scale down a missing Pod counts as 100% of its request (or the target if higher), on a scale up as 0%. Set-aside Pods : 0% on a scale up. Pods metrics : the target, going down.
Object, External : a Value target multiplies by Running and Ready Pods; AverageValue gives ceil(total / target), checked against status.replicas.
4.kube-controller-manager flags
Flag ("..." = --horizontal-) | Default |
|---|---|
...-sync- | 15s |
...-tolerance | 0.1 |
...-downscale- | 5m |
...-cpu- | 5m |
...-initial- | 30s |
--concurrent- | 5 |
Cluster-wide; a per-HPA tolerance overrides it. -upscale-delay and -downscale-delay are gone since 1.31 : set one and the process exits (unknown flag).
5.Utilization needs requests
Utilizationis usage as a percent of the request (see Requests, Limits and QoS).- If any container in a Pod (often a sidecar) has no request for that resource, the HPA takes no action on that metric.
- Fix : request on every container, or
ContainerResourcefor one named container (next sheet). - With Pod-level requests set (
PodLevelResources, beta, on) and no container named, the Pod-level request is the denominator.
1.Five metric types
| Type | Reads | Targets | From |
|---|---|---|---|
Resource | cpu or memory per Pod, summed over containers | Utilization, AverageValue | metrics. |
ContainerResource | one named container (GA 1.30) | Utilization, AverageValue | metrics. |
Pods | a per-Pod custom metric, averaged | AverageValue only | custom. |
Object | one object in the namespace (describedObject) | Value, AverageValue | custom. |
External | not tied to an object; matching series are summed | Value, AverageValue | external. |
No metric set : CPU Utilization 80%. Several metrics : each one proposes a count and the largest wins.
2.Utilization, AverageValue, Value
| Target | Compares | Replicas |
|---|---|---|
Utilization | mean usage / request, in percent, over the counted Pods | ceil(current x mean% / target%) |
AverageValue | mean per Pod (Object, External : total / status.) | ceil(current x mean / target), or ceil(total / target) |
Value | one raw series value, as is | ceil(readyPods x value / target) |
AverageValue on CPU or memory needs no request (raw usage, 500m). 1500m means 1.5.
3.Where the numbers come from
metrics.k8s.io/v1is GA in 1.37, yet the 1.37 HPA readsv1beta1of both the resource and external APIs : keep serving it.custom.: the version the server prefers (discovery).metrics.k8s.io - Installing KEDA takes
external.from prometheus-adapter. See metrics-server, Prometheus.metrics.k8s.io - The docs prefer custom over external : the external API can expose anything.
4.The behavior field
behavior:
scaleDown:
stabilizationWindowSeconds: 300 # kcm flag
policies:
- {type: Percent, value: 100, periodSeconds: 15}
scaleUp:
stabilizationWindowSeconds: 0
policies:
- {type: Percent, value: 100, periodSeconds: 15}
- {type: Pods, value: 4, periodSeconds: 15}
selectPolicy: Max
- Each sync records a recommendation. Scale down takes the highest one in the window (0 to 3600s) : a rolling max.
- A policy caps change over its
periodSeconds(1 to 1800).selectPolicy:Max= most change,Min= least,Disabled= off. - Per-direction
tolerance, GA in 1.37 : 0.05 down and 0.01 up on a 100Mi target scale down below 95Mi, up above 101Mi. - No
behaviorat all (everyautoscaling/v1HPA) : the legacy path, scale up capped atmax(2 x current, 4)per sync (ScaleUpLimit).
5.A policy at work
Pods 4 / 60s and Percent 10 / 60s (Max) : above 40, 10% allows more, so it removes 8, then 8 (7.2 rounded up); below 40, 4 at a time."Scale down feels slow" : that is the 300s window, by design. Change it per HPA or with the kcm (kube-controller-manager) flag.
behavior : no wait up, a 5-minute rolling max down, policies cap each step.1.Reading describe hpa
| Condition | True reasons | False reasons |
|---|---|---|
AbleToScale | ReadyForNewScale, ScaleUpStabilized, ScaleDownStabilized, SucceededRescale ... | FailedGetScale, FailedUpdateScale |
ScalingActive | ValidMetricFound | ScalingDisabled, FailedGet...Metric ... |
ScalingLimited | TooManyReplicas, TooFewReplicas, ScaleUpLimit, ScaleDownLimit | DesiredWithinRange |
ScaledToZero | ScaledToZero | NotScaledToZero |
Events : SuccessfulRescale, FailedComputeMetricsReplicas, AmbiguousSelector (two HPAs, same Pods : no scaling).
2.Rollouts and GitOps
- The HPA owns
replicas; the Deployment splits them across ReplicaSets, surge Pods included. - Drop
spec.replicasfrom the manifest, or every apply resets it. Naively it drops to 1 once : usekubectl apply edit-last-applied(client-side) or transfer field ownership (server-side, API Machinery, Flux). - Readiness that passes only when warm keeps startup CPU from scaling up (Rollouts and Probes). CPU target with headroom :
(1 - buffer) / (1 + growth); 10% and 30% give about 70%.
3.Scale to zero in 1.37
- Gate
HPAScaleToZero: beta, on in 1.37, in kube-apiserver and kube-controller-manager. Upgrade both first;minReplicas >= 1before a downgrade. minReplicas: 0needs anObjectorExternalmetric; CPU or memory alone is rejected.ScaledToZerocame in 1.36. RaisingminReplicaswakes an HPA-held zero. From zero : no tolerance or ready-Pod count; on the way to zero the 300s window applies.
4.Beside VPA, KEDA and Karpenter
| Tool | Changes | Signal | See |
|---|---|---|---|
| HPA | replica count | resource, custom, external metrics | this chapter, Deployments and Scaling |
| VPA | container requests | usage history | Vertical Pod Autoscaler |
| KEDA | creates and drives an HPA, adds 0 to 1 | event sources | KEDA |
| Karpenter | nodes | unschedulable Pods | Karpenter |
- Never VPA and HPA on the same resource metric (VPA). KEDA owns
keda-hpa-<name>: add no HPA of your own to that workload. min = max : a no-op. - CPU limit = request : utilization tops out near 100%, so a sync grows replicas by at most about
100 / target(1.25x at 80%); throttling hides demand (Requests and Limits).
KEDA-scaled services that never go to zero, with a tuned cooldownPeriod. Pitfall : it only governs scaling to zero; N to 1 follows the HPA's 300s window. Do better : set behavior in advanced.; check the metric with kubectl get --raw.
5.Commands
| Command | Shows |
|---|---|
| kubectl get hpa -A | TARGETS as current / target |
| kubectl describe hpa web | metrics, conditions, events |
| kubectl get events --field-selector involvedObject.kind= | rescale history and failures |
| kubectl get --raw /apis/ | the numbers the HPA reads |
replicas, and give scale to zero an Object or External metric. Never let two autoscalers drive the same workload on the same signal.