Scaling and metrics

Karpenter

1.What is Karpenter ?

A node autoscaler. It launches nodes for Pods that cannot be scheduled, then removes or replaces nodes that are empty, underused, outdated or too old.

  • SIG Autoscaling subproject. A core library, one provider per cloud.
  • No node groups : it picks instance types per launch, sized from Pod requests.
  • Never scales Pods. An HPA or KEDA adds replicas, Karpenter adds the nodes.
Versions and Kubernetes 1.37
  • v1.14.1 core and AWS provider (2026-08-21), LTS on AWS until Jul 2027. Azure provider v1.14.3.
  • karpenter.sh/v1 since v1.0. Tutorials with Provisioner, AWSNodeTemplate or v1beta1 are out of date.
  • 1.37 is not declared yet : v1.14 covers 1.26 to 1.36, these notes target 1.37. On 1.37 the AWS provider logs "not compatible" once and runs on, unsupported. Expect support in v1.15.

2.From Pending Pod to node

  1. kube-scheduler gives up : PodScheduled=False, Unschedulable.
  2. Karpenter skips Pods with a nodeName, a nomination, or a DaemonSet owner.
  3. Batch : 1s with no new Pod, 10s max.
  4. Biggest Pods first : existing nodes, planned nodes, then NodePools by weight.
  5. Size = requests + DaemonSet requests. Preferences relax one by one.
  6. Node Ready, kube-scheduler binds.
Pending Pod PodScheduled=False Karpenter watches, batches Unschedulable batch : 1s idle, 10s max Simulate nodes, then NodePools create NodeClaim one new machine launch Cloud API AWS : EC2 CreateFleet boots New Node kubelet joins Ready kube-scheduler binds, not Karpenter binds the Pod

3.NodePool, NodeClass, NodeClaim

NodePool = what nodes may look like. NodeClass = how to build them on this cloud. NodeClaim = one machine.

KindHolds
NodePool
karpenter.sh/v1
Requirements, taints, expiry, disruption, limits, weight
EC2NodeClass
karpenter.k8s.aws/v1
AMI, subnets, security groups, IAM role, kubelet
AKSNodeClass
karpenter.azure.com
v1beta1. Image family, OS disk, maxPods
NodeClaim
karpenter.sh/v1
One machine. Spec is immutable
NodePool what nodes may be NodeClass cloud settings nodeClassRef owns N shapes NodeClaim one machine Instance EC2 / Azure VM launches Node registers Node kubelet joins Claim, instance and Node map one to one. Delete the pool and its claims go too.
NodeClaim lifecycle
Launchedinstance up (5 min)
Registeredkubelet in (15 min)
InitializedReady, taints gone
Readyall three true

Miss a timeout and the claim and instance are deleted. Consolidation waits for Initialized.

4.How Karpenter reads a Pod

Pod fieldWhat Karpenter does
requestsSizes the node (containers, init, sidecars, overhead). Limits are ignored
nodeSelector, required affinityA NodePool must allow it, custom labels included, or the Pod stays Pending
preferred affinity, ScheduleAnywayTreated as required, then relaxed one at a time
tolerationsMust tolerate the NodePool taints. startupTaints need no toleration
PVC, WaitForFirstConsumerLaunches in the zone of the StorageClass or bound PV
DaemonSet PodsNever trigger a node on their own. Counted as overhead

Same rules as in Advanced Scheduling : Karpenter simulates them before it buys a node.

In short : Pending Pods in, NodeClaims out. The NodePool says what, the NodeClass says how.

1.Six ways a node goes away

Replacing nodes is the job. Graceful methods can be paused or slowed, forceful ones only get a drain deadline.

MethodFires whenBudget reasondo-not-disrupt
Consolidation gracefulPods fit elsewhere (delete), or elsewhere plus one cheaper node (replace)Empty, UnderutilizedBlocks it
Drift gracefulNodePool template or NodeClass changed, or a new AMI resolvedDriftedBlocks it, if no TGP
Expiration forcefulOlder than expireAfter (default 720h). No wait for a replacementNone, ignoredDoes not stop it
Interruption forcefulAWS Spot warning, health event, instance stopped or terminatedNoneDoes not stop it
Node repair forcefulAlpha. Ready False or Unknown for 30 min. Skipped if over 20% unhealthyNoneDoes not stop it
Manual deletekubectl delete nodeclaim or node, drained through the finalizerNoneDoes not stop it

2.The graceful loop

Pick a method fixed order Budget left ? per pool, per reason yes Simulate Pods fit elsewhere ? Taint the node disrupted:NoSchedule launch new first Replacement Ready fail : untaint, retry Terminate evict, detach, delete acted : go again now. idle : wait 10s
  • Order : Emptiness, static drift, Drift, multi-node, single-node consolidation. One method per pass.
  • Draining uses the Eviction API, so PDBs hold it back.
Consolidation policy
  • WhenEmpty : only nodes whose Pods cost nothing to disrupt (DaemonSet Pods).
  • WhenEmptyOrUnderutilized (default) : any move that lowers cost.
  • Balanced (v1.14+) : scores savings against disruption cost, moves only at score >= 0.5. Stops near-zero-savings churn.
  • consolidateAfter : quiet time after the last Pod change. Never = off.
DriftTemplate and NodeClass hashes sit on each NodeClaim. A new AMI from the selectors drifts too. Changing weight, limits or disruption never drifts. No off switch : pace it with a Drifted budget.

3.What keeps it safe

apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: default }
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["on-demand"]
      expireAfter: 720h           # forceful
      terminationGracePeriod: 48h # deadline
  disruption:
    consolidateAfter: 1m  # required in here
    budgets:
    - nodes: "20%"        # any reason
    - nodes: "0"          # pause
      schedule: "@daily"  # UTC
      duration: 10m
      reasons: ["Underutilized"]
  • Budgets : max nodes disrupted at once per pool (default 10%). By reason, with a UTC cron window.
  • PDBs : every drain evicts, so a PDB can hold it.
  • do-not-disrupt : on a Pod, "true" or a duration like "30m". On a Node, only "true". Blocks graceful methods. Forceful drains skip those Pods until the TGP deadline.
  • terminationGracePeriod : near its end, leftover Pods are deleted, past PDBs and do-not-disrupt. Each still gets its own grace seconds.
launched expireAfter + TGP normal life forceful drain max node life = expireAfter + TGP
TrapexpireAfter with no TGP, plus a do-not-disrupt Pod or a blocking PDB : the node can sit half drained forever. Set both.
In short : Budgets, PDBs and do-not-disrupt slow graceful methods. The TGP caps every drain.

1.Spot and interruptions

  • Several capacity types allowed ? It prefers reserved, then spot, then on-demand.
  • AWS Spot : price-capacity-optimized, up to 60 instance types per CreateFleet call.
  • No capacity ? That type and zone is skipped for 3 min.
  • 2-minute notice : the replacement starts and the drain runs in parallel.
  • Spot nodes are consolidated by deletion only, unless the alpha SpotToSpotConsolidation gate is on.
  • Do not also run AWS Node Termination Handler.
EventBridge EC2 + Health events SQS queue interruptionQueue Karpenter reads the queue CordonAndDrain Spot, stop, scheduled ForcefulTermination status check failed NoAction rebalance : event only No queue set ? Interruption handling is off.
Azure is differentProvider v1.14.3 has no interruption controller. An evicted Spot VM is deleted, then a new node comes. No drain.

2.Karpenter vs Cluster Autoscaler

AspectCluster AutoscalerKarpenter
ScalesPre-built node groups, an expander picks oneSingle instances, types picked per launch
Scale upUnschedulable Pods, scan every 10sUnschedulable Pods, batch 1s idle, 10s max
Scale downUnneeded for 10 min, under 0.5 utilizationConsolidation policy and consolidateAfter (default 0s)
LifecycleAutoscaling onlyAlso expiry, drift (image upgrades), interruption, repair
TimeoutsProvision 15 minLaunch 5 min, registration 15 min
Pod opt-outsafe-to-evict: "false"karpenter.sh/do-not-disrupt
VersionsUse the release that matches the control-plane minorOne release spans a range of minors

Migrating ? The guide scales Cluster Autoscaler to 0 replicas. The FAQ says the two can run side by side.

3.Providers

ProviderNotes
AWS v1.14.1Reference, LTS. Helm, or managed as EKS Auto Mode (336h expiry, 21-day cap)
Azure v1.14.3Helm, or managed as AKS NAP (GA, last announced on v1.14.0). No Windows
Cluster API v0.2.0Experimental proof of concept, create and delete only. See Cluster API
IBM Cloud v1.0.5SIG owned
GCP v0.7.0CloudPilot AI, pre-1.0

Oracle ships an official OCI provider. The core README lists about a dozen more.

4.Gotchas

  • A disruption: block without consolidateAfter is rejected in v1.14. Many doc snippets fail on it.
  • A new DaemonSet never triggers a node. Give it a high, preempting PriorityClass.
  • Agent taints (Cilium's) belong in startupTaints, or Karpenter keeps launching nodes.
  • Editing expireAfter or TGP drifts the whole pool. So does an @latest AMI alias, on every release.
  • Limits count node capacity, not requests, and can overrun in fast scale-outs.

5.Commands

CommandDoes
helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version 1.14.1 -n kube-systemCRDs as their own release. Helm never upgrades a chart's crds/ folder
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version 1.14.1 -n kube-system --set settings.clusterName=$CLUSTER_NAME --set settings.interruptionQueue=$CLUSTER_NAMEController : 2 replicas in kube-system, a PDB, metrics on :8080. Needs two non-Karpenter nodes in different zones
kubectl get nodepools -o wideNODECLASS, NODES, READY. wide adds WEIGHT, CPU, MEMORY
kubectl get nodeclaims -o wideTYPE, CAPACITY, ZONE, NODE, READY. wide adds IMAGEID, ID, NODEPOOL, NODECLASS, DRIFTED
kubectl describe nodeclaim NAMEWhich condition is False, DisruptionBlocked events
kubectl logs -f -n kube-system -l app.kubernetes.io/name=karpenter -c controllerFollow the controller's decisions
kubectl annotate node NODE karpenter.sh/do-not-disrupt=trueShield one node from graceful methods
In short : AWS Spot needs the SQS queue, CRDs get their own chart, 1.37 is not declared yet.