Scaling and metrics
Karpenter
1.What is Karpenter ?
A node autoscaler. It launches nodes for Pods that cannot be scheduled, then removes or replaces nodes that are empty, underused, outdated or too old.
Versions and Kubernetes 1.37
- v1.14.1 core and AWS provider (2026-08-21), LTS on AWS until Jul 2027. Azure provider v1.14.3.
karpenter.sh/v1since v1.0. Tutorials with Provisioner, AWSNodeTemplate or v1beta1 are out of date.- 1.37 is not declared yet : v1.14 covers 1.26 to 1.36, these notes target 1.37. On 1.37 the AWS provider logs "not compatible" once and runs on, unsupported. Expect support in v1.15.
2.From Pending Pod to node
- kube-scheduler gives up :
PodScheduled=False, Unschedulable. - Karpenter skips Pods with a nodeName, a nomination, or a DaemonSet owner.
- Batch : 1s with no new Pod, 10s max.
- Biggest Pods first : existing nodes, planned nodes, then NodePools by weight.
- Size = requests + DaemonSet requests. Preferences relax one by one.
- Node Ready, kube-scheduler binds.
3.NodePool, NodeClass, NodeClaim
NodePool = what nodes may look like. NodeClass = how to build them on this cloud. NodeClaim = one machine.
| Kind | Holds |
|---|---|
| NodePool karpenter.sh/v1 | Requirements, taints, expiry, disruption, limits, weight |
| EC2NodeClass karpenter.k8s.aws/v1 | AMI, subnets, security groups, IAM role, kubelet |
| AKSNodeClass karpenter.azure.com | v1beta1. Image family, OS disk, maxPods |
| NodeClaim karpenter.sh/v1 | One machine. Spec is immutable |
NodeClaim lifecycle
Launchedinstance up (5 min)
Registeredkubelet in (15 min)
InitializedReady, taints gone
Readyall three true
Miss a timeout and the claim and instance are deleted. Consolidation waits for Initialized.
4.How Karpenter reads a Pod
| Pod field | What Karpenter does |
|---|---|
| requests | Sizes the node (containers, init, sidecars, overhead). Limits are ignored |
| nodeSelector, required affinity | A NodePool must allow it, custom labels included, or the Pod stays Pending |
| preferred affinity, ScheduleAnyway | Treated as required, then relaxed one at a time |
| tolerations | Must tolerate the NodePool taints. startupTaints need no toleration |
| PVC, WaitForFirstConsumer | Launches in the zone of the StorageClass or bound PV |
| DaemonSet Pods | Never trigger a node on their own. Counted as overhead |
Same rules as in Advanced Scheduling : Karpenter simulates them before it buys a node.
In short : Pending Pods in, NodeClaims out. The NodePool says what, the NodeClass says how.
1.Six ways a node goes away
Replacing nodes is the job. Graceful methods can be paused or slowed, forceful ones only get a drain deadline.
| Method | Fires when | Budget reason | do-not-disrupt |
|---|---|---|---|
| Consolidation graceful | Pods fit elsewhere (delete), or elsewhere plus one cheaper node (replace) | Empty, Underutilized | Blocks it |
| Drift graceful | NodePool template or NodeClass changed, or a new AMI resolved | Drifted | Blocks it, if no TGP |
| Expiration forceful | Older than expireAfter (default 720h). No wait for a replacement | None, ignored | Does not stop it |
| Interruption forceful | AWS Spot warning, health event, instance stopped or terminated | None | Does not stop it |
| Node repair forceful | Alpha. Ready False or Unknown for 30 min. Skipped if over 20% unhealthy | None | Does not stop it |
| Manual delete | kubectl delete nodeclaim or node, drained through the finalizer | None | Does not stop it |
2.The graceful loop
- Order : Emptiness, static drift, Drift, multi-node, single-node consolidation. One method per pass.
- Draining uses the Eviction API, so PDBs hold it back.
Consolidation policy
- WhenEmpty : only nodes whose Pods cost nothing to disrupt (DaemonSet Pods).
- WhenEmptyOrUnderutilized (default) : any move that lowers cost.
- Balanced (v1.14+) : scores savings against disruption cost, moves only at score >= 0.5. Stops near-zero-savings churn.
- consolidateAfter : quiet time after the last Pod change.
Never= off.
DriftTemplate and NodeClass hashes sit on each NodeClaim. A new AMI from the selectors drifts too. Changing weight, limits or disruption never drifts. No off switch : pace it with a Drifted budget.
3.What keeps it safe
apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: default }
spec:
template:
spec:
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: default
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand"]
expireAfter: 720h # forceful
terminationGracePeriod: 48h # deadline
disruption:
consolidateAfter: 1m # required in here
budgets:
- nodes: "20%" # any reason
- nodes: "0" # pause
schedule: "@daily" # UTC
duration: 10m
reasons: ["Underutilized"]
- Budgets : max nodes disrupted at once per pool (default 10%). By reason, with a UTC cron window.
- PDBs : every drain evicts, so a PDB can hold it.
- do-not-disrupt : on a Pod,
"true"or a duration like"30m". On a Node, only"true". Blocks graceful methods. Forceful drains skip those Pods until the TGP deadline. - terminationGracePeriod : near its end, leftover Pods are deleted, past PDBs and do-not-disrupt. Each still gets its own grace seconds.
TrapexpireAfter with no TGP, plus a do-not-disrupt Pod or a blocking PDB : the node can sit half drained forever. Set both.
In short : Budgets, PDBs and do-not-disrupt slow graceful methods. The TGP caps every drain.
1.Spot and interruptions
- Several capacity types allowed ? It prefers reserved, then spot, then on-demand.
- AWS Spot : price-capacity-optimized, up to 60 instance types per CreateFleet call.
- No capacity ? That type and zone is skipped for 3 min.
- 2-minute notice : the replacement starts and the drain runs in parallel.
- Spot nodes are consolidated by deletion only, unless the alpha SpotToSpotConsolidation gate is on.
- Do not also run AWS Node Termination Handler.
Azure is differentProvider v1.14.3 has no interruption controller. An evicted Spot VM is deleted, then a new node comes. No drain.
2.Karpenter vs Cluster Autoscaler
| Aspect | Cluster Autoscaler | Karpenter |
|---|---|---|
| Scales | Pre-built node groups, an expander picks one | Single instances, types picked per launch |
| Scale up | Unschedulable Pods, scan every 10s | Unschedulable Pods, batch 1s idle, 10s max |
| Scale down | Unneeded for 10 min, under 0.5 utilization | Consolidation policy and consolidateAfter (default 0s) |
| Lifecycle | Autoscaling only | Also expiry, drift (image upgrades), interruption, repair |
| Timeouts | Provision 15 min | Launch 5 min, registration 15 min |
| Pod opt-out | safe-to-evict: "false" | karpenter. |
| Versions | Use the release that matches the control-plane minor | One release spans a range of minors |
Migrating ? The guide scales Cluster Autoscaler to 0 replicas. The FAQ says the two can run side by side.
3.Providers
| Provider | Notes |
|---|---|
| AWS v1.14.1 | Reference, LTS. Helm, or managed as EKS Auto Mode (336h expiry, 21-day cap) |
| Azure v1.14.3 | Helm, or managed as AKS NAP (GA, last announced on v1.14.0). No Windows |
| Cluster API v0.2.0 | Experimental proof of concept, create and delete only. See Cluster API |
| IBM Cloud v1.0.5 | SIG owned |
| GCP v0.7.0 | CloudPilot AI, pre-1.0 |
Oracle ships an official OCI provider. The core README lists about a dozen more.
4.Gotchas
- A
disruption:block without consolidateAfter is rejected in v1.14. Many doc snippets fail on it. - A new DaemonSet never triggers a node. Give it a high, preempting PriorityClass.
- Agent taints (Cilium's) belong in startupTaints, or Karpenter keeps launching nodes.
- Editing expireAfter or TGP drifts the whole pool. So does an
@latestAMI alias, on every release. - Limits count node capacity, not requests, and can overrun in fast scale-outs.
5.Commands
| Command | Does |
|---|---|
helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version 1.14.1 -n kube-system | CRDs as their own release. Helm never upgrades a chart's crds/ folder |
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version 1.14.1 -n kube-system --set settings.clusterName=$CLUSTER_NAME --set settings.interruptionQueue=$CLUSTER_NAME | Controller : 2 replicas in kube-system, a PDB, metrics on :8080. Needs two non-Karpenter nodes in different zones |
kubectl get nodepools -o wide | NODECLASS, NODES, READY. wide adds WEIGHT, CPU, MEMORY |
kubectl get nodeclaims -o wide | TYPE, CAPACITY, ZONE, NODE, READY. wide adds IMAGEID, ID, NODEPOOL, NODECLASS, DRIFTED |
kubectl describe nodeclaim NAME | Which condition is False, DisruptionBlocked events |
kubectl logs -f -n kube-system -l app. | Follow the controller's decisions |
kubectl annotate node NODE karpenter. | Shield one node from graceful methods |
In short : AWS Spot needs the SQS queue, CRDs get their own chart, 1.37 is not declared yet.