Resources and autoscaling

Horizontal Pod Autoscaler in Depth

1.Beyond the basics

  • The formula : desired = ceil(current x value / target); a ratio of 0.9 to 1.1 changes nothing (tolerance 0.1). Deployments and Scaling works an example. Everything else here stops that formula from flapping.
  • autoscaling/v2 is GA since 1.23 (v2beta2 not served since 1.26). autoscaling/v1 is still served, CPU only; v2-only fields survive as annotations.
  • It scales anything with a scale subresource : Deployment, StatefulSet, ReplicaSet, CRDs. Never a DaemonSet.
  • A controller loop in kube-controller-manager, every 15s. In 1.37 a new HPA, or one whose spec changed (HPAGeneration, beta, on), is reconciled at once.

2.One sync

  1. Read the target's /scale : replicas and selector.
  2. Out of min..max today ? Clamp, without reading metrics.
  3. current in the formula is scale.spec.replicas, the desired count, not the Pods running now.
  4. Every metric failed, or some failed and the rest say "fewer" : an error, no change.
  5. Some failed and the rest say "more" : scale up, and still report the error.
HPA object controller, every 15s read /scale : replicas, selector list Pods by selector metrics APIs ratio per metric, largest wins behavior : window, policies write scale.spec.replicas Deployment splits across ReplicaSets 0 replicas and not ScaledToZero : stop (paused) some failed : scale up only then clamp to min and max

3.Which Pods count

  1. Pods being deleted are ignored, Failed Pods discarded (phases : Pods).
  2. Pending Pods (every metric type) and Pods with no sample are set aside.
  3. CPU only : a not-yet-ready Pod is set aside. In its first 5 minutes if NotReady or its last sample predates its readiness change; after that only if it was never Ready.
  4. The ratio comes from the rest. Inside tolerance : stop.
  5. Recompute conservatively. If that flips direction or lands inside tolerance : no change (status still shows the plain average).
No metric ?

On a scale down a missing Pod counts as 100% of its request (or the target if higher), on a scale up as 0%. Set-aside Pods : 0% on a scale up. Pods metrics : the target, going down.

Object, External : a Value target multiplies by Running and Ready Pods; AverageValue gives ceil(total / target), checked against status.replicas.

4.kube-controller-manager flags

Flag ("..." = --horizontal-pod-autoscaler)Default
...-sync-period15s
...-tolerance0.1
...-downscale-stabilization5m
...-cpu-initialization-period5m
...-initial-readiness-delay30s
--concurrent-horizontal-pod-autoscaler-syncs5

Cluster-wide; a per-HPA tolerance overrides it. -upscale-delay and -downscale-delay are gone since 1.31 : set one and the process exits (unknown flag).

5.Utilization needs requests

  • Utilization is usage as a percent of the request (see Requests, Limits and QoS).
  • If any container in a Pod (often a sidecar) has no request for that resource, the HPA takes no action on that metric.
  • Fix : request on every container, or ContainerResource for one named container (next sheet).
  • With Pod-level requests set (PodLevelResources, beta, on) and no container named, the Pod-level request is the denominator.
In short : Every 15s the HPA reads one number per metric, divides it by the target, multiplies by the current replicas and rounds up. It ignores or sets aside Pods it cannot trust and recomputes conservatively, so it moves only when the evidence is clear.

1.Five metric types

TypeReadsTargetsFrom
Resourcecpu or memory per Pod, summed over containersUtilization, AverageValuemetrics.k8s.io
ContainerResourceone named container (GA 1.30)Utilization, AverageValuemetrics.k8s.io
Podsa per-Pod custom metric, averagedAverageValue onlycustom.metrics.k8s.io
Objectone object in the namespace (describedObject)Value, AverageValuecustom.metrics.k8s.io
Externalnot tied to an object; matching series are summedValue, AverageValueexternal.metrics.k8s.io

No metric set : CPU Utilization 80%. Several metrics : each one proposes a count and the largest wins.

2.Utilization, AverageValue, Value

TargetComparesReplicas
Utilizationmean usage / request, in percent, over the counted Podsceil(current x mean% / target%)
AverageValuemean per Pod (Object, External : total / status.replicas)ceil(current x mean / target), or ceil(total / target)
Valueone raw series value, as isceil(readyPods x value / target)

AverageValue on CPU or memory needs no request (raw usage, 500m). 1500m means 1.5.

3.Where the numbers come from

metrics-server metrics.k8s.io prometheus-adapter custom.metrics.k8s.io KEDA (or an adapter) external.metrics.k8s.io one server per cluster kube-apiserver aggregation HPA controller
  • metrics.k8s.io/v1 is GA in 1.37, yet the 1.37 HPA reads v1beta1 of both the resource and external APIs : keep serving it.
  • custom.metrics.k8s.io : the version the server prefers (discovery).
  • Installing KEDA takes external.metrics.k8s.io from prometheus-adapter. See metrics-server, Prometheus.
  • The docs prefer custom over external : the external API can expose anything.

4.The behavior field

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300 # kcm flag
    policies:
    - {type: Percent, value: 100, periodSeconds: 15}
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
    - {type: Percent, value: 100, periodSeconds: 15}
    - {type: Pods, value: 4, periodSeconds: 15}
    selectPolicy: Max
  • Each sync records a recommendation. Scale down takes the highest one in the window (0 to 3600s) : a rolling max.
  • A policy caps change over its periodSeconds (1 to 1800). selectPolicy : Max = most change, Min = least, Disabled = off.
  • Per-direction tolerance, GA in 1.37 : 0.05 down and 0.01 up on a 100Mi target scale down below 95Mi, up above 101Mi.
  • No behavior at all (every autoscaling/v1 HPA) : the legacy path, scale up capped at max(2 x current, 4) per sync (ScaleUpLimit).

5.A policy at work

From 80 replicas, scale down with Pods 4 / 60s and Percent 10 / 60s (Max) : above 40, 10% allows more, so it removes 8, then 8 (7.2 rounded up); below 40, 4 at a time.

"Scale down feels slow" : that is the 300s window, by design. Change it per HPA or with the kcm (kube-controller-manager) flag.

In short : Pick the metric type by what you can measure, the target type by what it means; the largest proposal wins. behavior : no wait up, a 5-minute rolling max down, policies cap each step.

1.Reading describe hpa

ConditionTrue reasonsFalse reasons
AbleToScaleReadyForNewScale, ScaleUpStabilized, ScaleDownStabilized, SucceededRescale ...FailedGetScale, FailedUpdateScale
ScalingActiveValidMetricFoundScalingDisabled, FailedGet...Metric ...
ScalingLimitedTooManyReplicas, TooFewReplicas, ScaleUpLimit, ScaleDownLimitDesiredWithinRange
ScaledToZeroScaledToZeroNotScaledToZero

Events : SuccessfulRescale, FailedComputeMetricsReplicas, AmbiguousSelector (two HPAs, same Pods : no scaling).

2.Rollouts and GitOps

  • The HPA owns replicas; the Deployment splits them across ReplicaSets, surge Pods included.
  • Drop spec.replicas from the manifest, or every apply resets it. Naively it drops to 1 once : use kubectl apply edit-last-applied (client-side) or transfer field ownership (server-side, API Machinery, Flux).
  • Readiness that passes only when warm keeps startup CPU from scaling up (Rollouts and Probes). CPU target with headroom : (1 - buffer) / (1 + growth); 10% and 30% give about 70%.

3.Scale to zero in 1.37

N replicas HPA scales down : minReplicas 0 + Object / External metric rises : ceil(usage ratio) 0, ScaledToZero=True someone scales to 0 0, paused : ScalingActive False, ScalingDisabled the HPA will not wake it
  • Gate HPAScaleToZero : beta, on in 1.37, in kube-apiserver and kube-controller-manager. Upgrade both first; minReplicas >= 1 before a downgrade.
  • minReplicas: 0 needs an Object or External metric; CPU or memory alone is rejected.
  • ScaledToZero came in 1.36. Raising minReplicas wakes an HPA-held zero. From zero : no tolerance or ready-Pod count; on the way to zero the 300s window applies.

4.Beside VPA, KEDA and Karpenter

ToolChangesSignalSee
HPAreplica countresource, custom, external metricsthis chapter, Deployments and Scaling
VPAcontainer requestsusage historyVertical Pod Autoscaler
KEDAcreates and drives an HPA, adds 0 to 1event sourcesKEDA
Karpenternodesunschedulable PodsKarpenter
  • Never VPA and HPA on the same resource metric (VPA). KEDA owns keda-hpa-<name> : add no HPA of your own to that workload. min = max : a no-op.
  • CPU limit = request : utilization tops out near 100%, so a sync grows replicas by at most about 100 / target (1.25x at 80%); throttling hides demand (Requests and Limits).
Seen in practice

KEDA-scaled services that never go to zero, with a tuned cooldownPeriod. Pitfall : it only governs scaling to zero; N to 1 follows the HPA's 300s window. Do better : set behavior in advanced.horizontalPodAutoscalerConfig; check the metric with kubectl get --raw.

5.Commands

CommandShows
kubectl get hpa -ATARGETS as current / target
kubectl describe hpa webmetrics, conditions, events
kubectl get events --field-selector involvedObject.kind=HorizontalPodAutoscaler,involvedObject.name=webrescale history and failures
kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/<ns>/podsthe numbers the HPA reads
In short : Read the conditions first, let the HPA own replicas, and give scale to zero an Object or External metric. Never let two autoscalers drive the same workload on the same signal.