Resources and autoscaling
Requests, Limits and QoS
1.Reservation and ceiling
Basics are in Quick Revision. Here : under the hood.
- Scheduler : per resource, requests on the node plus the new Pod must fit
node.. Usage is ignored. A miss :status. allocatable FailedScheduling,insufficient cpu(plugins). - For scoring only, a container with no request counts as
100mCPU and200Mi, so zero-request Pods spread out. Fit still uses zero. - HPA utilization is a percentage of the request.
2.Units and traps
| CPU | Memory | |
|---|---|---|
| unit | cores, m = milli | bytes, Mi, Gi, M |
| request | scheduling, cpu.weight share | scheduling, eviction ranking, Memory QoS hints |
| limit | cpu.max quota per 100 ms | memory.max hard cap |
| over limit | throttled, never killed | OOM-killed, exit 137, restarted |
| kind | compressible | not compressible |
| idle | usable above request if no limit | usable, but first to be evicted |
memory: 400mis 0.4 bytes.- CPU is absolute, not a share of the node.
- Limit only : the request copies it, even over a LimitRange
defaultRequest. - Smallest useful CPU limit :
10m(1 ms quota).
3.What lands in cgroup v2
| Field | cgroup v2 file | Example |
|---|---|---|
requests.cpu | cpu.weight, from shares = mCPU x 1024 / 1000 (2 to 262144) | 1 CPU = 100 (runc 1.3.2+, crun 1.23+), 39 before |
limits.cpu | cpu.max = "<quota> 100000" | 500m : 50000 100000; none : max 100000 |
limits.memory | memory.max | max when unset |
requests.memory | nothing hard | memory.min, .low, .high only with Memory QoS |
- The kubelet passes values through the CRI; the OCI runtime writes the files. The kubelet still writes Pod and QoS-tier cgroups with the old formula, so their
cpu.weightuses another scale. - An OOM kills every process in the container together (
memory.oom.group). cgroup v1 : the kubelet refuses it by default since 1.35 (failCgroupV1: true), removal not before 1.38.
4.Throttling below the average
- Throttling shows while average usage looks far below the limit. Check
nr_throttledincpu.stat, not averages. - Runtimes read the limit : Go 1.25 (when
go.modsays 1.25+) setsGOMAXPROCSfrom it (rounded up, minimum 2, never above the CPU count) and re-checks it after a resize.
5.Node allocatable
Allocatable = Capacity - kubeReserved - systemReserved - hard eviction threshold. By default both reserves are empty, so only memory. is held back. Docs example : 16 CPU, 32Gi minus 1 CPU / 2Gi kube, 500m / 1Gi system and <500Mi eviction gives 14.5 CPU, 28.5Gi. systemReserved should cover kernel memory, which is not charged to Pods.
cpu.max and memory.max. Over it, CPU is throttled and memory is OOM-killed.1.QoS beyond the rules
Rules in Quick Revision. The class is derived and fixed; an in-place resize that would change it is rejected.
| Class | oom_score_adj | Eviction position |
|---|---|---|
| Guaranteed | -997 | last, by priority |
| Burstable | min(max(3, 1000 - 1000 x memRequest / nodeMemory), 999), so 3 to 999 | first if over its request, else late |
| BestEffort | 1000 | first : usage is always over a zero request |
system-node-criticalPods also get -997. A sidecar's score is capped at that of the app container with the smallest memory request, so it is not killed first.
2.Two ways to die of memory
3.Who is evicted first
- Usage above request ? (not for PID pressure)
- Then lower priority.
- Then usage relative to request.
memory.= capacity minus working set. Page cache inavailable active_filecan fake pressure; the documented fix is memory request = limit.- Disk pressure : dead containers and unused images go first.
MemoryPressure,DiskPressure,PIDPressurebecome taints (scheduling).HugepageAwareEviction(beta, on in 1.37) subtracts huge pages frommemory..available
4.Default hard thresholds (Linux)
memory. | nodefs. | imagefs. | nodefs. | imagefs. |
|---|---|---|---|---|
| <100Mi | <10% | <15% | <5% | <5% |
evictionHard value zeroes the others, unless mergeDefaultEvictionSettings: true . List every one you want. Soft thresholds need a grace period; evictionPressureTransitionPeriod is 5m.5.Commands, and Memory QoS in v1.37
| Command | What it shows |
|---|---|
| kubectl describe node <node> | capacity, allocatable, allocated requests |
| kubectl get pod <p> -o | the QoS class |
| kubectl describe pod <p> | OOMKilled, exit 137, or FailedScheduling |
| kubectl top pod <p> --containers | live usage (needs metrics-server) |
| stat -fc %T /sys/fs/cgroup/ | prints cgroup2fs on cgroup v2 |
| cat cpu.weight cpu.max memory.max cpu.stat | applied values, in the container's cgroup |
- Beta and on, but inert until configured (
memoryThrottlingdefaults to null). With a factor,Factor memory.high = request + factor x (limit - request): 0.9, 256Mi, 1Gi gives about 947Mi. Never for Guaranteed or a container with request = limit. memoryReservationsetsPolicy: Tiered Reservation memory.min(Guaranteed) ormemory.low(Burstable) to the request; a cache-heavy Guaranteed Pod may then OOM. Kernel 5.9+.
A cache or JVM service crash-loops with OOMKilled. Pitfall : its own memory cap (-m, -Xmx, GOMEMLIMIT) equals the container limit, but buffers and off-heap memory live above that cap. Do better : keep the app cap clearly below the limit; raise the limit only once you know why memory grew.
1.Guardrails at admission
- LimitRange defaulting : a missing
defaulttakesmax; a missingdefaultRequesttakesdefault, thenmin. - Request only : gets the default limit. Limit only : request = its own limit.
default.cpu: 500mand a700mrequest with no limit : rejected (must be less than or equal to cpu limit).- Two LimitRanges : the winning default is not deterministic.
maxLimitRequestRatio>= 1. Admission only : running Pods never change (plugin order).
2.ResourceQuota
| Key | What it sums |
|---|---|
requests. / .memory | sum of requests |
limits. / .memory | sum of limits |
requests., hugepages- | scratch, huge pages |
requests. | extended, requests. only |
<class>. | DRA device count |
pods, count/, requests. | objects, PVC storage |
- Scopes :
BestEffort,NotBestEffort,Terminating,NotTerminating,PriorityClass,CrossNamespace,PodAffinity VolumeAttributes.Class - Only non-terminal Pods count.
- Quota on
cpuormemory: Pods must set it (LimitRange). - Over quota : Deployment, no Pods.
- Write access can delete it : RBAC.
3.Effective request : init, sidecars, Pod level
- sidecar proxyindex 0 : 100m
- init migrateindex 1 : 1 CPU + sidecar = 1.1
- app500m; running sum 600m
- effective1.1 CPU, for the Pod's whole life
Init and sidecars : Pods. The exact KEP-753 rule (short form : Advanced Scheduling). RuntimeClass overhead is added after.
Pod-level spec.resources (beta, on since 1.34) : only cpu, memory, hugepages-. Pod values win for scheduling, the Pod cgroup and QoS. Container limits must be <= Pod limits; the rest share the budget. Not used by the CPU, memory or topology managers in 1.37, nor on Windows. An empty Pod-level block no longer changes QoS.
4.The CPU-limits question
| Policy | Good for | Cost |
|---|---|---|
| request only | latency-sensitive services | needs accurate requests; apps lean on spare CPU |
| request = limit | benchmarks, predictable capacity | idle CPU wasted; throttled if sized low |
| request + headroom | a middle ground | still throttles at the cap |
| whole-core limit | greedy batch Jobs | throttling is intended |
Mainstream answer : CPU requests everywhere, sized from data (VPA); CPU limits only for a hard cap (multi-tenant, greedy batch, benchmarks); memory request = limit where it matters. The Kubernetes blog carries a dissent, for predictability.
A gateway proxy's TLS throughput sat far below line rate, throttled at low average CPU. Pitfall : a small CPU limit on a data-path Pod. Do better : no CPU limit on proxies, a request from measured peak, a throttle-ratio alert.
5.Commands
| Command | What it shows |
|---|---|
| kubectl describe quota -n <ns> | used vs hard, per key |
| kubectl describe limitrange -n <ns> | defaults, min, max, ratio |
| sum by (namespace,pod) (rate(container_ | throttle ratio by Pod (Prometheus). Aggregate by Pod, not container name |