Resources and autoscaling

Requests, Limits and QoS

1.Reservation and ceiling

Basics are in Quick Revision. Here : under the hood.

  • Scheduler : per resource, requests on the node plus the new Pod must fit node.status.allocatable. Usage is ignored. A miss : FailedScheduling, insufficient cpu (plugins).
  • For scoring only, a container with no request counts as 100m CPU and 200Mi, so zero-request Pods spread out. Fit still uses zero.
  • HPA utilization is a percentage of the request.
Pod spec : resources requests limits kube-scheduler fits allocatable ? cpu.weight share when contended kubelet + runtime cpu.max throttled memory.max OOM-killed, exit 137

2.Units and traps

CPUMemory
unitcores, m = millibytes, Mi, Gi, M
requestscheduling, cpu.weight sharescheduling, eviction ranking, Memory QoS hints
limitcpu.max quota per 100 msmemory.max hard cap
over limitthrottled, never killedOOM-killed, exit 137, restarted
kindcompressiblenot compressible
idleusable above request if no limitusable, but first to be evicted
Unit traps
  • memory: 400m is 0.4 bytes.
  • CPU is absolute, not a share of the node.
  • Limit only : the request copies it, even over a LimitRange defaultRequest.
  • Smallest useful CPU limit : 10m (1 ms quota).

3.What lands in cgroup v2

Fieldcgroup v2 fileExample
requests.cpucpu.weight, from shares = mCPU x 1024 / 1000 (2 to 262144)1 CPU = 100 (runc 1.3.2+, crun 1.23+), 39 before
limits.cpucpu.max = "<quota> 100000"500m : 50000 100000; none : max 100000
limits.memorymemory.maxmax when unset
requests.memorynothing hardmemory.min, .low, .high only with Memory QoS
  • The kubelet passes values through the CRI; the OCI runtime writes the files. The kubelet still writes Pod and QoS-tier cgroups with the old formula, so their cpu.weight uses another scale.
  • An OOM kills every process in the container together (memory.oom.group). cgroup v1 : the kubelet refuses it by default since 1.35 (failCgroupV1: true), removal not before 1.38.

4.Throttling below the average

  • Throttling shows while average usage looks far below the limit. Check nr_throttled in cpu.stat, not averages.
  • Runtimes read the limit : Go 1.25 (when go.mod says 1.25+) sets GOMAXPROCS from it (rounded up, minimum 2, never above the CPU count) and re-checks it after a resize.
limit 100m = 10 ms per 100 ms throttled throttled throttled run 10 ms : 30 ms of work needs 3 periods, about 210 ms 4 threads, limit 400m all stall all stall all stall 4 x 10 ms burns the 40 ms quota, all stall about 90 ms

5.Node allocatable

Allocatable = Capacity - kubeReserved - systemReserved - hard eviction threshold. By default both reserves are empty, so only memory.available<100Mi is held back. Docs example : 16 CPU, 32Gi minus 1 CPU / 2Gi kube, 500m / 1Gi system and <500Mi eviction gives 14.5 CPU, 28.5Gi. systemReserved should cover kernel memory, which is not charged to Pods.

In short : A request is a reservation counted against allocatable; a limit is a ceiling the kernel enforces with cpu.max and memory.max. Over it, CPU is throttled and memory is OOM-killed.

1.QoS beyond the rules

Rules in Quick Revision. The class is derived and fixed; an in-place resize that would change it is rejected.

Classoom_score_adjEviction position
Guaranteed-997last, by priority
Burstablemin(max(3, 1000 - 1000 x memRequest / nodeMemory), 999), so 3 to 999first if over its request, else late
BestEffort1000first : usage is always over a zero request
  • system-node-critical Pods also get -997. A sidecar's score is capped at that of the app container with the smallest memory request, so it is not killed first.

2.Two ways to die of memory

memory use rises hits its limit node runs short kernel OOM killer inside that cgroup OOMKilled, exit 137 restarted in the same Pod kubelet signal under threshold, every 10 s rank Pods, evict Pod Failed controller makes a new one PDBs ignored, hard : 0 s grace spike

3.Who is evicted first

  1. Usage above request ? (not for PID pressure)
  2. Then lower priority.
  3. Then usage relative to request.
  • memory.available = capacity minus working set. Page cache in active_file can fake pressure; the documented fix is memory request = limit.
  • Disk pressure : dead containers and unused images go first.
  • MemoryPressure, DiskPressure, PIDPressure become taints (scheduling).
  • HugepageAwareEviction (beta, on in 1.37) subtracts huge pages from memory.available.

4.Default hard thresholds (Linux)

memory.availablenodefs.availableimagefs.availablenodefs.inodesFreeimagefs.inodesFree
<100Mi<10%<15%<5%<5%
Setting one evictionHard value zeroes the others, unless mergeDefaultEvictionSettings: true. List every one you want. Soft thresholds need a grace period; evictionPressureTransitionPeriod is 5m.

5.Commands, and Memory QoS in v1.37

CommandWhat it shows
kubectl describe node <node>capacity, allocatable, allocated requests
kubectl get pod <p> -o jsonpath='{.status.qosClass}'the QoS class
kubectl describe pod <p>OOMKilled, exit 137, or FailedScheduling
kubectl top pod <p> --containerslive usage (needs metrics-server)
stat -fc %T /sys/fs/cgroup/prints cgroup2fs on cgroup v2
cat cpu.weight cpu.max memory.max cpu.statapplied values, in the container's cgroup
Memory QoS
  • Beta and on, but inert until configured (memoryThrottlingFactor defaults to null). With a factor, memory.high = request + factor x (limit - request) : 0.9, 256Mi, 1Gi gives about 947Mi. Never for Guaranteed or a container with request = limit.
  • memoryReservationPolicy: TieredReservation sets memory.min (Guaranteed) or memory.low (Burstable) to the request; a cache-heavy Guaranteed Pod may then OOM. Kernel 5.9+.
Seen in practice

A cache or JVM service crash-loops with OOMKilled. Pitfall : its own memory cap (-m, -Xmx, GOMEMLIMIT) equals the container limit, but buffers and off-heap memory live above that cap. Do better : keep the app cap clearly below the limit; raise the limit only once you know why memory grew.

In short : QoS sets the OOM score; the kubelet evicts by usage over request, then priority. A fast spike meets the OOM killer first, eviction ignores PDBs, one custom threshold zeroes the rest.

1.Guardrails at admission

Pod create LimitRanger : apply defaults LimitRanger : min, max, ratio ResourceQuota : sum vs hard stored, then scheduler 403 Forbidden
  • LimitRange defaulting : a missing default takes max; a missing defaultRequest takes default, then min.
  • Request only : gets the default limit. Limit only : request = its own limit.
  • default.cpu: 500m and a 700m request with no limit : rejected (must be less than or equal to cpu limit).
  • Two LimitRanges : the winning default is not deterministic. maxLimitRequestRatio >= 1. Admission only : running Pods never change (plugin order).

2.ResourceQuota

KeyWhat it sums
requests.cpu / .memorysum of requests
limits.cpu / .memorysum of limits
requests.ephemeral-storage, hugepages-<size>scratch, huge pages
requests.<extended>extended, requests. only
<class>.deviceclass.resource.k8s.io/devicesDRA device count
pods, count/<resource>.<group>, requests.storageobjects, PVC storage
  • Scopes : BestEffort, NotBestEffort, Terminating, NotTerminating, PriorityClass, CrossNamespacePodAffinity, VolumeAttributesClass.
  • Only non-terminal Pods count.
  • Quota on cpu or memory : Pods must set it (LimitRange).
  • Over quota : Deployment, no Pods.
  • Write access can delete it : RBAC.

3.Effective request : init, sidecars, Pod level

  • sidecar proxyindex 0 : 100m
  • init migrateindex 1 : 1 CPU + sidecar = 1.1
  • app500m; running sum 600m
  • effective1.1 CPU, for the Pod's whole life

Init and sidecars : Pods. The exact KEP-753 rule (short form : Advanced Scheduling). RuntimeClass overhead is added after.

Pod-level spec.resources (beta, on since 1.34) : only cpu, memory, hugepages-<size>. Pod values win for scheduling, the Pod cgroup and QoS. Container limits must be <= Pod limits; the rest share the budget. Not used by the CPU, memory or topology managers in 1.37, nor on Windows. An empty Pod-level block no longer changes QoS.

4.The CPU-limits question

PolicyGood forCost
request onlylatency-sensitive servicesneeds accurate requests; apps lean on spare CPU
request = limitbenchmarks, predictable capacityidle CPU wasted; throttled if sized low
request + headrooma middle groundstill throttles at the cap
whole-core limitgreedy batch Jobsthrottling is intended

Mainstream answer : CPU requests everywhere, sized from data (VPA); CPU limits only for a hard cap (multi-tenant, greedy batch, benchmarks); memory request = limit where it matters. The Kubernetes blog carries a dissent, for predictability.

Seen in practice

A gateway proxy's TLS throughput sat far below line rate, throttled at low average CPU. Pitfall : a small CPU limit on a data-path Pod. Do better : no CPU limit on proxies, a request from measured peak, a throttle-ratio alert.

5.Commands

CommandWhat it shows
kubectl describe quota -n <ns>used vs hard, per key
kubectl describe limitrange -n <ns>defaults, min, max, ratio
sum by (namespace,pod) (rate(container_cpu_cfs_throttled_periods_total[1h])) / sum by (namespace,pod) (rate(container_cpu_cfs_periods_total[1h]))throttle ratio by Pod (Prometheus). Aggregate by Pod, not container name
In short : LimitRange fills in and checks each container, ResourceQuota caps the namespace sum, both at admission. Init and sidecars count in the effective request; CPU limits only for hard caps.