Resources and autoscaling

In-Place Pod Resize

1.What is in-place resize ?

  • Change the CPU and memory requests and limits of a running Pod's containers without a new Pod : same node, same IP, warm caches.
  • KEP-1287, gate InPlacePodVerticalScaling : alpha 1.27, beta (on) 1.33, GA 1.35, locked on, due for removal in 1.38.
  • Do not set the gate : =true only logs a warning, =false stops the component at startup.
Two ways to change resources
  • Template change in a Deployment = new Pods (a rollout).
  • A /resize patch on the live Pod = the same Pod, new numbers.
  • The QoS class never changes : a resize that would change it is rejected.

2.Desired, allocated, actual

ViewFieldWritten by
desiredspec.containers[*].resourcesyou, through /resize
allocatedstatus.containerStatuses[*].allocatedResources (requests only)the kubelet, when it admits the resize
actualstatus.containerStatuses[*].resourcesthe kubelet, from what the runtime applied
progressstatus.conditions : PodResizePending, PodResizeInProgressthe kubelet

status.resize is deprecated and no longer set since 1.33. Scripts must read the two conditions instead. Pod spec and status basics are in Pods.

3.The resize subresource

  • Only --subresource resize on patch, edit or apply (kubectl 1.32+) changes resources. A plain update cannot.
  • RBAC resource : pods/resize (RBAC). Grant patch on it to whoever resizes.
  • It can change regular containers, sidecars, running non-sidecar init containers (GA 1.37), Pod-level spec.resources (beta 1.36), and with an alpha gate a memory-backed emptyDir sizeLimit.
$ kubectl patch pod resize-demo \
  --subresource resize --patch \
  '{"spec":{"containers":[{"name":"app","resources":
    {"requests":{"cpu":"800m"},"limits":{"cpu":"800m"}}}]}}'

4.One resize, end to end

patch --subresource resize API server validation + admission 403 Forbidden spec = desired kubelet : does it fit ? others' allocated + new <= allocatable not now : Deferred, retried never here : Infeasible yes allocated + InProgress event ResizeStarted CRI update : cgroups failed : reason Error status = actual, Completed

Admission : PodResizeValidator, LimitRanger, ResourceQuota. One resize is atomic; patches sent before the kubelet acts merge.

5.Conditions and events

  • PodResizePending, reason Deferred : fits the node's allocatable but not its free room. Retried.
  • PodResizePending, reason Infeasible : this node can never do it. Not retried.
  • PodResizeInProgress : allocated, being applied. Reason Error when applying failed.
  • Both at once : a new resize arrived while the last one is still applying.
  • Events : ResizeStarted, ResizeCompleted, ResizeDeferred, ResizeInfeasible, and ResizeError as a Warning.
conditions:  # a resize that waits
- type: PodResizePending
  status: "True"
  reason: Deferred
  message: "... cpu, requested: 6000,
    used: 3950, capacity: 8000"
In short : Desired lives in spec, actual in status. You change desired only through /resize; the kubelet catches up in the same Pod and reports progress in PodResizePending and PodResizeInProgress.

1.resizePolicy

  • Per container and resource : NotRequired (resize live if possible) or RestartContainer (restartCount goes up). Only these two; never PreferNoRestart.
  • No entry = NotRequired. An entry with an empty restartPolicy fails validation.
  • Both resources change and either is RestartContainer : the container restarts.
  • Pods with restartPolicy: Never must use NotRequired.
  • Fixed after creation (recreate the Pod). Since 1.36 non-sidecar init containers may not use RestartContainer.
containers:  # Pod resize-demo
- name: app
  image: registry.k8s.io/pause:3.8
  resizePolicy:
  - resourceName: cpu
    restartPolicy: NotRequired
  - resourceName: memory
    restartPolicy: RestartContainer
  resources:  # Guaranteed : change
    # request and limit together
    requests: {cpu: 500m, memory: 128Mi}
    limits:   {cpu: 500m, memory: 128Mi}

2.How the kubelet applies it

Node-side admission : other Pods' allocated requests + this Pod's new requests <= allocatable. Then the cgroups are written (cpu.weight, cpu.max, memory.max; see Requests, Limits and QoS).

Growing : pod cgroup first, then containers. Shrinking : containers first, pod cgroup last. Decreases before increases. Memory before CPU. On any error the kubelet stops and retries on a later sync.

node pod cgroup container A container B growing 1. pod cgroup 2. containers shrinking 1. containers 2. pod cgroup memory first, then CPU; decreases before increases

3.Shrinking memory

  • Allowed with NotRequired since 1.34.
  • Usage at or above the new limit : the kubelet refuses, sets PodResizeInProgress reason Error with a Warning ResizeError, and retries later.
  • A race remains : usage can rise between check and write. On cgroup v2 the kernel then reclaims, or OOM-kills.
  • Common mistake : expecting a shrink below usage to "stay pending". It is an Error, not Pending.

4.Waiting and accounting

  • Retry order : no request increase, then higher PriorityClass, then Guaranteed before Burstable, then the one waiting longest.
  • Scheduler and quota count max(desired, allocated, actual) (not desired if Infeasible).
  • Race : a new Pod placed from cached data can be rejected OutOfCPU (open issue).
  • 1.37 alpha (...SchedulerPreemption on API server, scheduler, kubelet) : preempt lower-priority Pods for a Deferred resize, same node only.

5.What happens when

SituationOutcome
fits the node nowPodResizeInProgress, then done
fits allocatable, but other Pods use the roomDeferred, retried
above the node's allocatable403 at admission since 1.36 (PodResizeValidator; it ignores other Pods); was Infeasible
static or Windows Pod, QoS change, removing a set value, not CPU or memory, resizePolicy changerejected by API validation (403/422 at the API server); QoS case : "Pod QOS Class may not change as a result of resizing"
memory change with swap and NotRequiredInfeasible
Guaranteed Pod on a static CPU (Memory) manager node, CPU (memory) changeInfeasible unless an alpha gate; the other resource resizes
memory limit at or below usagePodResizeInProgress Error, retried

Burstable Pods on static manager nodes are not blocked. Ephemeral containers are never resizable.

In short : resizePolicy sets restart per resource, fixed at creation. The kubelet admits against allocatable, writes cgroups in a safe order and refuses memory shrinks below usage.

1.Who resizes in place ?

kubectl --subresource VPA updater (in place) operator calling it pods/resize same Pod same IP not in place : edit in Git template new Pods
  • A resources change in a Deployment, StatefulSet or DaemonSet template still rolls new Pods. Job template resources cannot change.
  • So a template's resizePolicy matters only when something patches the live Pod. Workload-level in-place resize is only a design sketch.
  • A manual patch is out of band from Git : when the Pod is replaced, the template value returns.
  • Operators resize in place only if they call pods/resize. CloudNativePG recreates its Pods instead.

2.Pod-level resize

  • Beta and on since 1.36, still beta in 1.37. Patch Pod-level spec.resources (the API) through /resize.
  • No Pod-level resizePolicy. A container without its own limit inherits the Pod limit, so it resizes too, and restarts if its own policy says RestartContainer.
  • Rules : Pod requests >= sum of container requests; each container limit <= Pod limit.
  • Needs cgroup v2, Linux, containerd 2.0+ or CRI-O.

3.VPA uses it

  • updateMode: InPlaceOrRecreate (GA in VPA 1.6.0) resizes all containers through pods/resize and falls back to eviction. InPlace (alpha, VPA 1.7.0) never evicts. Auto is deprecated.
  • The updater needs patch on pods/resize. VPA does not support Pods with Pod-level resources. Fallback rules : VPA.

4.Runtimes

RuntimeCPU live ?Memory live ?
Go, Rust, Cyesyes, more room is usable at once
JVM (HotSpot)mostly : availableProcessors() re-reads cgroups, but GC and pool threads are sized at startno : max heap is fixed at start
Node.jsyesheap capped by --max-old-space-size, set at start
Pythonyesthe GA blog says it needs a restart

Rule of thumb : CPU NotRequired; memory RestartContainer for runtimes with a fixed heap.

Seen in practice
  • Pitfall : memory NotRequired in the Git-managed template of a fixed-heap JVM service, "so future bumps land live". A Git edit still rolls the Pod, and a live memory raise adds only off-heap room, no heap.
  • Do better : treat resizePolicy as the contract for /resize callers (VPA, kubectl), use memory RestartContainer for fixed heaps, and write manual resizes back to Git.

5.Commands

CommandWhat it does
kubectl get pod POD -o jsonpath='{.status.containerStatuses[0].allocatedResources.cpu}'The CPU request the kubelet allocated
kubectl get pod POD -o jsonpath='{.status.conditions[?(@.type=="PodResizePending")]}'Why a resize waits (reason and message)
kubectl get events --field-selector involvedObject.name=PODThe five Resize* events
kubectl exec POD -- cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/memory.maxApplied cgroup v2 values (max 100000 = no CPU limit)
kubectl auth can-i patch pods/resizeRBAC check for the subresource

Kubelet metrics (alpha) for Prometheus : kubelet_pod_pending_resizes, kubelet_pod_infeasible_resizes_total{reason_detail}, kubelet_pod_resize_duration_milliseconds.

In short : Templates still roll. Only a /resize caller (kubectl, VPA, an operator) resizes in place, until the next rollout. CPU resizes live almost everywhere; fixed heaps need a restart.