Resources and autoscaling
In-Place Pod Resize
1.What is in-place resize ?
- Change the CPU and memory requests and limits of a running Pod's containers without a new Pod : same node, same IP, warm caches.
- KEP-1287, gate
InPlacePodVerticalScaling: alpha 1.27, beta (on) 1.33, GA 1.35, locked on, due for removal in 1.38. - Do not set the gate :
=trueonly logs a warning,=falsestops the component at startup.
- Template change in a Deployment = new Pods (a rollout).
- A /resize patch on the live Pod = the same Pod, new numbers.
- The QoS class never changes : a resize that would change it is rejected.
2.Desired, allocated, actual
| View | Field | Written by |
|---|---|---|
| desired | spec. | you, through /resize |
| allocated | status. (requests only) | the kubelet, when it admits the resize |
| actual | status. | the kubelet, from what the runtime applied |
| progress | status. : PodResizePending, PodResizeInProgress | the kubelet |
status. is deprecated and no longer set since 1.33. Scripts must read the two conditions instead. Pod spec and status basics are in Pods.
3.The resize subresource
- Only
--subresource resizeonpatch,editorapply(kubectl 1.32+) changes resources. A plain update cannot. - RBAC resource :
pods/(RBAC). Grantresize patchon it to whoever resizes. - It can change regular containers, sidecars, running non-sidecar init containers (GA 1.37), Pod-level
spec.(beta 1.36), and with an alpha gate a memory-backedresources emptyDirsizeLimit.
$ kubectl patch pod resize-demo \
--subresource resize --patch \
'{"spec":{"containers":[{"name":"app","resources":
{"requests":{"cpu":"800m"},"limits":{"cpu":"800m"}}}]}}'
4.One resize, end to end
Admission : PodResizeValidator, LimitRanger, ResourceQuota. One resize is atomic; patches sent before the kubelet acts merge.
5.Conditions and events
PodResizePending, reason Deferred : fits the node's allocatable but not its free room. Retried.PodResizePending, reason Infeasible : this node can never do it. Not retried.PodResizeInProgress: allocated, being applied. Reason Error when applying failed.- Both at once : a new resize arrived while the last one is still applying.
- Events :
ResizeStarted,ResizeCompleted,ResizeDeferred,ResizeInfeasible, andResizeErroras a Warning.
conditions: # a resize that waits
- type: PodResizePending
status: "True"
reason: Deferred
message: "... cpu, requested: 6000,
used: 3950, capacity: 8000"
spec, actual in status. You change desired only through /resize; the kubelet catches up in the same Pod and reports progress in PodResizePending and PodResizeInProgress.1.resizePolicy
- Per container and resource : NotRequired (resize live if possible) or RestartContainer (
restartCountgoes up). Only these two; neverPreferNoRestart. - No entry =
NotRequired. An entry with an emptyrestartPolicyfails validation. - Both resources change and either is
RestartContainer: the container restarts. - Pods with
restartPolicy: Nevermust useNotRequired. - Fixed after creation (recreate the Pod). Since 1.36 non-sidecar init containers may not use
RestartContainer.
containers: # Pod resize-demo - name: app image: registry.k8s.io/pause:3.8 resizePolicy: - resourceName: cpu restartPolicy: NotRequired - resourceName: memory restartPolicy: RestartContainer resources: # Guaranteed : change # request and limit together requests: {cpu: 500m, memory: 128Mi} limits: {cpu: 500m, memory: 128Mi}
2.How the kubelet applies it
Node-side admission : other Pods' allocated requests + this Pod's new requests <= allocatable. Then the cgroups are written (cpu.weight, cpu.max, memory.max; see Requests, Limits and QoS).
Growing : pod cgroup first, then containers. Shrinking : containers first, pod cgroup last. Decreases before increases. Memory before CPU. On any error the kubelet stops and retries on a later sync.
3.Shrinking memory
- Allowed with
NotRequiredsince 1.34. - Usage at or above the new limit : the kubelet refuses, sets
PodResizeInProgressreason Error with a WarningResizeError, and retries later. - A race remains : usage can rise between check and write. On cgroup v2 the kernel then reclaims, or OOM-kills.
- Common mistake : expecting a shrink below usage to "stay pending". It is an
Error, notPending.
4.Waiting and accounting
- Retry order : no request increase, then higher PriorityClass, then Guaranteed before Burstable, then the one waiting longest.
- Scheduler and quota count max(desired, allocated, actual) (not desired if
Infeasible). - Race : a new Pod placed from cached data can be rejected
OutOfCPU(open issue). - 1.37 alpha (
...Scheduleron API server, scheduler, kubelet) : preempt lower-priority Pods for aPreemption Deferredresize, same node only.
5.What happens when
| Situation | Outcome |
|---|---|
| fits the node now | PodResizeInProgress, then done |
| fits allocatable, but other Pods use the room | Deferred, retried |
| above the node's allocatable | 403 at admission since 1.36 (PodResizeValidator; it ignores other Pods); was Infeasible |
static or Windows Pod, QoS change, removing a set value, not CPU or memory, resizePolicy change | rejected by API validation (403/422 at the API server); QoS case : "Pod QOS Class may not change as a result of resizing" |
memory change with swap and NotRequired | Infeasible |
| Guaranteed Pod on a static CPU (Memory) manager node, CPU (memory) change | Infeasible unless an alpha gate; the other resource resizes |
| memory limit at or below usage | PodResizeInProgress Error, retried |
Burstable Pods on static manager nodes are not blocked. Ephemeral containers are never resizable.
resizePolicy sets restart per resource, fixed at creation. The kubelet admits against allocatable, writes cgroups in a safe order and refuses memory shrinks below usage.1.Who resizes in place ?
- A
resourceschange in a Deployment, StatefulSet or DaemonSet template still rolls new Pods. Job template resources cannot change. - So a template's
resizePolicymatters only when something patches the live Pod. Workload-level in-place resize is only a design sketch. - A manual patch is out of band from Git : when the Pod is replaced, the template value returns.
- Operators resize in place only if they call
pods/. CloudNativePG recreates its Pods instead.resize
2.Pod-level resize
- Beta and on since 1.36, still beta in 1.37. Patch Pod-level
spec.(the API) throughresources /resize. - No Pod-level
resizePolicy. A container without its own limit inherits the Pod limit, so it resizes too, and restarts if its own policy saysRestartContainer. - Rules : Pod requests >= sum of container requests; each container limit <= Pod limit.
- Needs cgroup v2, Linux, containerd 2.0+ or CRI-O.
3.VPA uses it
updateMode: InPlaceOrRecreate(GA in VPA 1.6.0) resizes all containers throughpods/and falls back to eviction.resize InPlace(alpha, VPA 1.7.0) never evicts.Autois deprecated.- The updater needs
patchonpods/. VPA does not support Pods with Pod-level resources. Fallback rules : VPA.resize
4.Runtimes
| Runtime | CPU live ? | Memory live ? |
|---|---|---|
| Go, Rust, C | yes | yes, more room is usable at once |
| JVM (HotSpot) | mostly : availableProcessors() re-reads cgroups, but GC and pool threads are sized at start | no : max heap is fixed at start |
| Node.js | yes | heap capped by --max-old-space-size, set at start |
| Python | yes | the GA blog says it needs a restart |
Rule of thumb : CPU NotRequired; memory RestartContainer for runtimes with a fixed heap.
- Pitfall : memory
NotRequiredin the Git-managed template of a fixed-heap JVM service, "so future bumps land live". A Git edit still rolls the Pod, and a live memory raise adds only off-heap room, no heap. - Do better : treat
resizePolicyas the contract for/resizecallers (VPA, kubectl), use memoryRestartContainerfor fixed heaps, and write manual resizes back to Git.
5.Commands
| Command | What it does |
|---|---|
| kubectl get pod POD -o jsonpath='{.status. | The CPU request the kubelet allocated |
| kubectl get pod POD -o jsonpath='{.status. | Why a resize waits (reason and message) |
| kubectl get events --field-selector involvedObject. | The five Resize* events |
| kubectl exec POD -- cat /sys/fs/ | Applied cgroup v2 values (max 100000 = no CPU limit) |
| kubectl auth can-i patch pods/resize | RBAC check for the subresource |
Kubelet metrics (alpha) for Prometheus : kubelet_, kubelet_, kubelet_.
/resize caller (kubectl, VPA, an operator) resizes in place, until the next rollout. CPU resizes live almost everywhere; fixed heaps need a restart.