KUBERNETESNOTES

A comprehensive collection of Kubernetes concepts, explained in simple words with clear diagrams and examples. Perfect for quick revision and interview preparation!

Kubernetes Core Concepts Pods & Containers Clusters &Nodes Deployments &Services Security &Best Practices
What's Inside?
  • Core Concepts
  • Architecture
  • Pods and Deployments
  • Services and Ingress
  • Config and Storage
  • GitOps and Certificates
  • Autoscaling, Gateway API
  • Serverless, Identity
  • Observability
  • Controllers, Operators, RBAC
Why This Notes?
  • Easy to understand
  • Well-structured
  • Interview focused
  • Quick revision
  • Handwritten style for better retention
“Learn, Deploy, and Scale with Kubernetes!”

Start here

Contents

How it plugs in Git repoFlux CD v2 syncs it API serverRBAC, admission, CRDs controllers and operators cert-managerExternal Secrets CloudNativePGKEDA, metrics Gateway, MetalLBSPIRE, OTel Your workloads

Part 2.Beyond Core

GitOps and delivery
CI and builds
Certificates and secrets
Data
Cloud infrastructure
Scaling and metrics
Workload identity
Networking and traffic
Serverless

Part 2.Beyond Core (continued)

Observability

Part 3.Advanced Concepts

Under the hood: the machinery that core objects and every add-on rely on, grouped by topic.

Extending the API
API machinery
Security
Where each topic fits CI: build, sign, attestImage Signing, SBOM and Provenance kubectl apply, or FluxGitOps with Flux API server who are you, may you ?OIDC Auth, RBAC apply: merge, own fieldsAPI Machinery mutating admissionAdmission schema checkCRDs validating admissionAdmission, Pod Security etcd saves the object watch control loops compare, act, repeatControllers app know-how in codeOperators build and upgrade clustersCluster API owners and finalizersFinalizers and GC pick a node and devicesScheduling, DRA, Volume Topology attach and mount its volumesCSI Drivers, Volume Operations pull and check the imageImages and Pulls, Image Policy kubelet runs the Pod traffic reaches the PodPod Networking, Service DataplaneCluster DNS, Network Policy

Top to bottom is the path of one change. Blue names are the topics that explain each step.

In short : Part 2 ends with observability: metrics, alerts, logs, events, traces and profiles. Part 3 starts with the API itself: how it is extended, how each request is checked, and who may do what.

Part 3.Advanced Concepts (continued)

Who you are and what you trust, trusted images, where Pods run and how they are sized and scaled, then the network and storage under every Pod.

Identity and trust
Supply chain security
Scheduling
Resources and autoscaling
Networking internals
Storage internals
One Pod, end to end name to Service IPCluster DNS Service IP to a PodService Dataplane allowed ?Network Policy a request comes in Pod gets an IP and a routePod Networking its PVC provision in the right zoneVolume Topology attach, stage, publishCSI Drivers later resize, detachVolume Operations snapshot, backupVolume Snapshots

Top half: traffic on its way to the Pod. Bottom half: the life of its volume.

In short : Part 3 continues under the hood: identity and trust, trusted images, placement, sizing and autoscaling, then the network and storage under every Pod.

Part 3.Advanced Concepts (continued)

Cluster management
GitOps

Tip.Start from a question

Start with Part 1. Then read each topic's In short first.

SituationRead
A Pod cannot reach a Service by nameCluster DNS, Service Dataplane
Only some Pods should reach a databaseNetwork Policy
A Pod is Pending: its volume is in another zoneVolume Topology
A volume is stuck attaching or detachingVolume Operations, CSI Drivers
You need yesterday's data backSnapshots and Backup
Keep a password in Git safelySecrets in Git with SOPS, External Secrets
A GitHub Actions job waits, no runner picks it upARC
A Gitea runner rollout cancels running jobsGitea Runners
CI must build images, but privileged Pods are bannedImage Builds, Pod Security
Only signed images may runImage Signing, Image Policy
An object will not deleteFinalizers and GC
Two tools fight over the same fieldAPI Machinery
Pods need GPUs or other devicesDRA, Scheduling
A ServiceMonitor is ignored, no targets appearPrometheus
Too many alert messages, or none at allAlertmanager
Logs are rejected or slow to queryLoki and Alloy
A Pod failed an hour ago and its Events are goneEvents
One request is slow, but which service and which line ?Tempo, Pyroscope
kubectl should log you in with your company accountOIDC Auth, Keycloak
Every app asks for its own passwordKeycloak
A Pod needs a database password that expiresOpenBao, External Secrets
Internal TLS fails with "unknown authority"Private PKI
Each Pod needs its own short-lived certificatePod Certificates, SPIRE
A container is OOMKilled, or slow under a CPU limitRequests and Limits
A running Pod needs more CPU, without a restartIn-Place Resize
Nobody knows what requests to setVPA, Requests and Limits
The HPA scales too late, too often, or not at allHPA in Depth
In short : Part 1 teaches the core of Kubernetes in 13 topics.
Part 2 adds 29 topics, grouped by what they do.
Part 3 looks under the hood, with 34 topics on how it all fits together.

Core Kubernetes · 01

What is Kubernetes

1.What is Kubernetes ?

Kubernetes is an open-source platform that runs containers across a cluster of machines.

You do not give it a list of steps. You declare the state you want as API objects, and control loops keep pulling the cluster toward it.

  • ManagedEKS, GKE, AKS run the control plane.
  • Distributionsk3s, RKE2, OpenShift, Talos package it with defaults.
  • Localkind, minikube, k3d on a laptop.
  • Extendyour own object types and controllers.
Fact file
  • NameGreek for helmsman or pilot. K8s = K, 8 letters, s.
  • RootsIdeas from Borg, Google's in-house cluster manager.
  • 2014First commit June 6. Google open-sources it.
  • 2015v1.0 on July 21. Google pledges it to the new CNCF.
  • OSLinux control plane. Linux or Windows workers.
  • HomeCNCF hosts it.
  • Nowv1.37, patch v1.37.1.

2.Why does it exist ?

An image packages your app. Production needs more : keep it running on many machines, with no downtime. By hand, you are the scheduler, health checker, load balancer and upgrade script.

Runtime vs orchestratorA runtime starts containers on one host. An orchestrator picks hosts, keeps the count and replaces what fails.
By hand, one host You SSH, scripts Host A 3 web containers crash : restart policy, same host Host dies all 3 stay down until you rebuild With an orchestrator You replicas: 3 apply Control plane API, scheduler, controllers Node 1 Node 2 lost Node 3 want 3, have 2 : new Pods start on Node 1 and 3
ChoreOne host, by handKubernetes
PlacementYou pick a host, then SSH inScheduler packs Pods by CPU / memory requests
Crashed containerRestart policy, same hostRestart, kill on failed checks, no traffic until ready
Dead machineDown until you rebuild itDeployment replaces the Pods on healthy nodes, after about 5 min
More copiesStart more, edit the proxykubectl scale, or an HPA fed by metrics-server
UpgradeStop old, start new, undo by handRolling update, kubectl rollout undo to go back
Finding the appHard-coded host IP:portDNS name or IP, load-balanced across copies
Config, secretsEnv files on the host, or in the imageConfigMaps and Secrets, no image rebuild
Source of truthShell history and memoryAPI objects, plus manifests kept in Git

3.Desired vs current state

Every object is a record of intent. You write its spec (what you want). Kubernetes writes its status (what is true now).

A controller is a never-ending loop, like a thermostat. It watches, compares spec with status, and asks the API server for changes until they match.

Loops, not a scriptNot a workflow engine : no fixed A, then B, then C, and no central brain. Many small loops each fix their own part. A busy cluster may never look stable, and that is fine.
You or CI kubectl apply The real cluster Pods crash, nodes fail writes spec watch, report API server every read and write Deployment web spec : 3 replicas status : 2 ready watch Controller compares spec, status differ 2 vs 3 Act ask for 1 more Pod request match : keep watching
In short : Kubernetes runs your containers across a cluster. You declare the state you want, and never-ending control loops keep pushing the real state toward it.

1.What you get, and what you do not

Built in
  • Service discovery and load balancing (Services)
  • Rollouts and rollbacks at a set pace
  • Bin packing : Pods placed by CPU / memory requests
  • Self-healing : restart, replace, kill on failed checks
  • Storage, batch jobs, secrets and config
  • Horizontal scaling, IPv4 / IPv6 dual-stack
Not included
  • Not an all-in-one PaaS : no code builds, no CI/CD
  • No built-in databases, caches or middleware
  • No required logging, monitoring or config language
  • Does not set up or manage your machines
  • Upstream ships no metrics-server (an HPA needs one)

2.Everything is an API object

Pods, Namespaces, ConfigMaps, Events : each is an object behind one HTTP API on the API server. kubectl and controllers all go through it. You write YAML, kubectl sends JSON.

Your manifest becomes the object apiVersionapps/v1 kindDeployment metadatanamespace, name specwhat you want statuswhat is true Identity API group + resource type + namespace + name not the version Controllers read spec, write status
apiVersion: apps/v1   # core kinds use just v1
kind: Deployment
metadata:
  name: web
spec:                 # desired state, yours
  replicas: 3         # 1 if left out
  selector:           # required, immutable
    matchLabels: {app: web}
  template:           # the Pod template
    metadata:
      labels: {app: web}   # = selector
    spec:
      containers:
      - name: web
        image: nginx:1.30
# no status here : Kubernetes writes it

3.Three ways to manage objects

TechniqueExampleWorks onBest for
Imperative commandskubectl create deployment web --image=nginx:1.30Live objectsDev, trying things
Imperative object configkubectl create -f web.yaml, then replace -fSingle filesProduction
Declarative object configkubectl diff -f configs/, then kubectl apply -f configs/Folders of filesProduction

apply keeps changes made by other writers. Use one technique per object, mixing them is undefined. replace drops whatever the file leaves out, even an HPA's changes.

4.First commands

CommandWhat it does
kind create clusterLocal cluster, nodes are containers (or minikube start)
kubectl version --clientClient version. Stay within 1 minor of the server
kubectl config get-contextsContexts (cluster, user, namespace) in ~/.kube/config
kubectl api-resourcesEvery type this cluster serves, with its group/version
kubectl explain pods.spec.containersField docs from the cluster's own schema
kubectl create deployment web --image=nginx:1.30 --dry-run=client -o yamlStarter manifest, nothing sent to the cluster
kubectl diff -f web.yamlPreview apply. Exit 1 = changes found, above 1 = error
kubectl apply -f web.yamlCreate or update objects to match the file
kubectl get deploy web -o yamlSpec and status, side by side
kubectl scale --replicas=5 deployment/webImperative. The next apply undoes it if the file sets replicas
kubectl get pods -wWatch live. Delete a Pod, see a new one appear
In short : Everything is an API object : you write the spec, Kubernetes writes the status. Manage each object one way, ideally kubectl diff then kubectl apply.

Core Kubernetes · 02

Architecture

1.What is in a cluster ?

Two halves. The control plane stores the state you asked for and keeps steering toward it. The nodes run your Pods. You need at least one.

  • Control plane : kube-apiserver, etcd, kube-scheduler, kube-controller-manager, plus cloud-controller-manager on a cloud.
  • Node : kubelet, a container runtime, usually kube-proxy.
  • Add-ons : a CNI network plugin (required) and cluster DNS.
  • Cloud : in-tree cloud code is gone since v1.31, so clouds run cloud-controller-manager.

2.Old words, new words

  • masterSay control plane. kubeadm switched its node label in v1.24.
  • minionSay node, or worker node.
  • client-serverNo. Hub-and-spoke around kube-apiserver.
  • dockershimGone in v1.24. Nodes use a CRI runtime.
  • kube-dnsCoreDNS since v1.13, behind a Service still named kube-dns.
  • /healthzDeprecated (v1.16), like componentstatuses (v1.19). Use /livez, /readyz.

3.Who talks to whom

kube-apiserver is the hub. The others watch it and write back to it. Only kube-apiserver touches etcd.

kubectl / clients HTTPS :6443 load balancer (HA) cloud provider API Control plane any healthy copy kube-apiserver the hub etcd all cluster state :2379 read / write cloud-controller-manager kube-controller-manager kube-scheduler LBs, routes, nodes watch my Pods, status, Lease logs, exec :10250 watch Services Every node Pods container runtime kubelet CRI kube-proxy (optional) kubectl :6443 load balancer (HA) kube-apiserver, the hub rest of the control plane etcd read / write on :2379 kube-scheduler kube-controller-manager cloud-controller-manager cloud provider API every node kube-proxy (optional) kubelet CRI container runtime Pods

Arrows point from the caller. Dashed : kube-apiserver calls the kubelet for logs, exec and port-forward. A CNI plugin such as Cilium can replace kube-proxy.

4.The parts, one by one

PartRuns onJobPort
kube-apiserverControl planeHTTP API, authn, authz, admission. Only etcd client.6443
etcdControl planeHolds every API object, the cluster's only source of truth. Back it up.2379-2380
kube-schedulerControl planeGives each unscheduled Pod a node10259
kube-controller-managerControl planeBuilt-in loops (Deployment, ReplicaSet, Node ...)10257
cloud-controller-managerCloud onlyNode, route and load balancer loops, cloud API10258
kubeletEvery nodeRegisters the node, runs its Pods via CRI10250
container runtimeEvery nodePulls images, makes Pod sandboxes and containers for the kubelet (CRI v1)socket
kube-proxy (optional)Every nodePacket rules for Service virtual IPs10256
CNI plugin (add-on)Every nodePod IPs and Pod-to-Pod traffic-
CoreDNS (add-on)kube-system PodsCluster DNS for Services and Pods-

Ports are TCP. NodePort range : see Services.

In short : The control plane stores the desired state and reconciles toward it, each node's kubelet makes it real through a CRI runtime, and every part goes through kube-apiserver, the only one that touches etcd.

1.From apply to running

  • 1-3, API : kubectl apply sends the Deployment over HTTPS. kube-apiserver checks it (authn, authz, mutating admission, validation, validating admission) and saves it in etcd.
  • 4-5, controllers : the Deployment controller makes a ReplicaSet, which makes Pods with no node yet (Pending).
  • 6, scheduler : filters and scores the nodes, then posts a binding.
  • 7-11, node : the kubelet sees the Pod and asks the runtime for a sandbox. The runtime calls CNI for the Pod IP, pulls the image, starts the containers. The kubelet reports Running.
kubectl kube-controller-manager Deploymentcontroller ReplicaSetcontroller kube-apiserver2 : runs checks etcd kube-scheduler 1 3 4 5 6 kubelet runtime (CRI) CNI plugin containers upimage pulled first 7, 11 8 9 10

No one calls the next step. Each part watches the API and reacts to a change.

2.Inside a node

  • The kubelet registers its Node and renews a Lease in kube-node-lease as its heartbeat.
  • It drives the runtime over CRI gRPC on a local socket. CRI v1 only : containerd, CRI-O, or Docker via cri-dockerd.
  • Static Pods : files in the kubelet's manifest folder run with no API server. kubeadm puts the control plane in /etc/kubernetes/manifests.
  • Since v1.35 the kubelet refuses cgroup v1 unless overridden. v1.36+ needs containerd 2.0+.

3.kube-proxy is not the network

Key point
  • Pod IPs and Pod-to-Pod traffic come from the CNI plugin.
  • kube-proxy only turns Services and EndpointSlices into rules for Service virtual IPs.
  • Modes : iptables (default), nftables (GA v1.33), ipvs (deprecated v1.35). Pick one explicitly.
  • Cilium and some other CNIs replace it, so no kube-proxy Pods.

4.HA control plane

  • 3+ control plane nodes, odd count for etcd quorum : 3 survive 1 loss, 5 survive 2.
  • kube-apiserver : all copies active behind a load balancer (TCP check, 6443).
  • Scheduler, controller-manager : one leader per Lease, the rest wait.
  • Stacked etcd (kubeadm default) : fewer hosts, but a dead node takes an apiserver and an etcd member.
  • External etcd : 3+ own hosts, failures stay apart.
workers + kubectl load balancer :6443 all active CP node 1 apiserver etcd sched leader CP node 2 apiserver etcd kcm leader CP node 3 apiserver etcd standby

Bottom row : scheduler and controller-manager, each with its own leader Lease.

5.Commands

CommandShows
kubectl get nodes -o wideNodes, kubelet version, CONTAINER-RUNTIME column
kubectl get pods -n kube-system -o wideStatic control plane Pods, CoreDNS, kube-proxy, CNI, and their nodes
kubectl get --raw='/readyz?verbose'API server checks one by one. /livez for liveness
kubectl -n kube-system get lease -l apiserver.kubernetes.io/identity=kube-apiserverOne Lease per running kube-apiserver
sudo crictl psOn the node : containers via CRI, even with the API server down
In short : Each step from apply to running is a loop reacting to an API change, the kubelet makes Pods real through CRI and CNI, and HA means 3+ control plane nodes behind a load balancer.

Core Kubernetes · 03

Pods

1.What is a Pod ?

The smallest thing Kubernetes deploys : one or more containers that are always placed and scheduled together on one node, like a small logical host for one app. API : apiVersion: v1, kind: Pod.

  • One container per Pod is the usual case. A second one only for a tightly coupled helper, e.g. one that refreshes the web server's files.
  • Never add containers to get copies. Copies are more Pods, made by a workload (Deployments and Scaling).

2.Network and storage

  • One IP per address family (status.podIPs : max one IPv4 + one IPv6).
  • Hostname = Pod name by default. Other Pods must use IP networking to reach it.
  • spec.volumes is declared once. Each container mounts what it needs with its own volumeMounts (Volumes and Storage).
Shared in the PodSet per container
IP address and portsImage and command
localhost and IPCEnv variables
Volumes it mountsProbes, resources
Node the Pod is bound here once and never moves Pod : web one IP, one port space localhost seed-content init, exited 0 refresher native sidecar nginx :80 app container writes once every 10s reads html emptyDir : survives container restarts, deleted with this Pod (its UID) Other Pod to Pod IP:80 over the network (no localhost, no shared IPC)

3.Four kinds of container

KindDeclared inWhen it runsProbesRestarts
Appspec.containersAll together once init is done, for the whole Pod lifeliveness, readiness, startupPod restartPolicy (per-container rules beta v1.35)
Initspec.initContainersBefore the apps, one at a time, each must exit 0None, no lifecycle hooks eitherRetried, unless restartPolicy is Never
Native sidecarinitContainers + restartPolicy: AlwaysStarts in init order, stays up the whole Pod life, stops lastYes, its readiness counts for the PodAlways, on any exit (GA v1.33)
Ephemeralvia kubectl debugAdded to a running Pod to troubleshootNot allowed, no ports eitherNever (GA v1.25)

4.Example : init, sidecar and app

apiVersion: v1
kind: Pod
metadata: {name: web}     # DNS label, becomes the hostname
spec:
  volumes: [{name: html, emptyDir: {}}]
  initContainers:
  - name: seed-content    # init : must exit 0
    image: busybox:1.28
    command: [sh, -c, 'echo hello > /html/index.html']
    volumeMounts: [{name: html, mountPath: /html}]
  - name: refresher       # native sidecar
    image: busybox:1.28
    restartPolicy: Always
    command: [sh, -c, 'while sleep 10; do date > /html/time.txt; done']
    volumeMounts: [{name: html, mountPath: /html}]
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports: [{containerPort: 80}]
    volumeMounts: [{name: html, mountPath: /usr/share/nginx/html}]
Start order
Sandbox readynetwork and volumes, phase Pending
seed-contentmust exit 0 (STATUS Init:0/2)
refresherstarted (or startupProbe ok)
nginxapps start together, phase Running
Stop order is the reverse : apps get TERM first, sidecars only after the last app exits.
In short : A Pod is one or more containers that share a node, one IP and the volumes they mount. Init runs first, sidecars run beside the app, and a controller usually creates the Pods.

1.Pod phases (status.phase)

Pendingnot set up yet Runningon a node Unknownlost contact Succeededterminal Failedterminal bound container restarts all exit 0 one failed,no restart node lost init fails and restartPolicy is Never
  • A rough summary, not a state machine. An in-place Pod restart can go back to Pending. STATUS is not the phase.
PhaseMeans (STATUS you may see)
PendingAccepted, but not every container is set up : no node yet, image pulls, init steps. Init:0/1, ContainerCreating, ImagePullBackOff
RunningOn a node, all containers created, at least one running or restarting. Running, CrashLoopBackOff
SucceededEvery container exited 0 and none will restart. Completed
FailedAll stopped, at least one failed, no restart coming. Pods of a lost node end here too. Error
UnknownState cannot be read, usually because the node does not answer.

2.Container states and restarts

Waitingnot started yet pull retry (max 300s) RunningpostStart done container starts Terminatedreason + exit code exits or is killed(preStop first) Restart ?restartPolicy yes : CrashLoopBackOffat once, then 10s, 20s ... 300s no : stays Terminated OOMKilled = exit 137
  • ImagePullBackOff : bad name or tag, or a private registry without an imagePullSecret.
PolicyExit 0Exit 1+Where
Always (default)restartrestartOnly value in a Deployment
OnFailuredonerestartJobs
NeverdonedoneJobs, one-off Pods
SidecarrestartrestartIgnores the Pod policy
  • Restarts happen in the same Pod, on the same node. RESTARTS goes up, the phase stays Running. Backoff resets after 10 min of clean running.

3.Why not a bare Pod ?

  • Scheduled once. Node failure, eviction or delete ends it for good.
  • Deployment, StatefulSet, DaemonSet and Job build Pods from a template and replace lost ones (new UID).
  • An HPA scales a workload, never a single Pod.
  • Delete : 30s grace, TERM (after preStop), then KILL.
Gotchas
  • Two containers in one Pod cannot both use :80.
  • A helper in spec.containers is not a sidecar : no start or stop order.
  • Pod fields are mostly immutable. Change the template instead.
  • --grace-period=0 --force does not wait for the kubelet. The process may still run.

4.Commands

CommandWhat it does
kubectl run web --image=nginx --dry-run=client -o yamlPrints a Pod manifest skeleton, creates nothing
kubectl get pods --field-selector=status.phase=RunningFilters on the real phase, not on STATUS
kubectl describe pod webState, Last State, exit code, Conditions, then Events. Conditions in kubelet order : PodReadyToStartContainers (sandbox ready, GA v1.37), Initialized, Ready, ContainersReady, PodScheduled
kubectl logs web -c nginx --previousLogs of the crashed run of one container
kubectl debug -it web --image=busybox:1.28 --target=nginxAdds an ephemeral debug container
In short : The phase is only a summary. The kubelet restarts containers in place with a growing backoff, but only a controller replaces a Pod that is gone.

Core Kubernetes · 04

Deployments and Scaling

1.What is a Deployment ?

A Deployment keeps N identical Pods running for a stateless app. It never creates Pods itself: it owns ReplicaSets, and they own the Pods.

  • One ReplicaSet per Pod template version, named web-<pod-template-hash>.
  • Old ones stay at 0 replicas as revision history (see Rollouts and Probes).
  • Links live in metadata.ownerReferences. Edit the Deployment, not its ReplicaSets.
Deployment webapps/v1, replicas: 3 owns, sets replicas owns, keeps at 0 ReplicaSet web-7c9fcurrent template ReplicaSet web-5d4bprevious template keeps 3 Pods web-7c9f-<random> 0 Pods, old template kept

2.Labels and Selectors

Labels are key/value tags, a selector is a query over them. A ReplicaSet finds its Pods by selector, never by name.

  • Key : optional prefix (DNS subdomain, 253 max), /, name (63 max). kubernetes.io/ and k8s.io/ are reserved.
  • Value : 63 max, may be empty.
  • Match : =, ==, !=, or set-based in, notin, exists (key), not exists (!key).
  • YAML : matchLabels and matchExpressions, all ANDed. No OR.
Deployment selectormatchLabels {app: web} Pod template labels{app: web} mustmatch copied,hash added ReplicaSet selectorapp=web, pod-template-hash=7c9f selects, owns Pod labelsapp=web, pod-template-hash=7c9f Other controllerselector {app: web} not blocked,they fight

A selector the template does not satisfy is rejected. In apps/v1 it is never defaulted and is immutable.

3.A Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata: {name: web}
spec:
  replicas: 3
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      containers:
      - name: web
        image: nginx:1.29
        resources: {requests: {cpu: 100m}}
  • selectorMust match template labels (extras ok). Immutable.
  • replicasDefault 1. Leave out under an HPA.
  • templateA change makes (or reuses) a ReplicaSet.
What brokeWho fixes itYou see
Container crashedkubelet, in place, with backoffRESTARTS climbs, CrashLoopBackOff
Pod deleted or evictedReplicaSet makes a new PodNew name, new IP
Node failedReplicaSet, after node grace + 300s tolerationNothing for about 6 min, then Pods on other nodes
Stray Pod matchesReplicaSet adopts it, deletes extrasBack to replicas

4.Which workload controller ?

ControllerUse it forGives youWatch out
Deployment apps/v1Stateless web apps, APIs, workersInterchangeable Pods, declarative updatesPod names are random. Use a StatefulSet for stable ones
StatefulSet apps/v1Databases, queues, quorum systemsdb-0, db-1, ... : stable DNS, own PVC, in orderYou make the headless Service. PVCs stay
DaemonSet apps/v1Log shippers, node exporters, CNI agentsOne Pod per eligible nodeNo HPA. Control-plane nodes need tolerations
Job batch/v1Migrations, one-off batch runsRuns to completion, backoffLimit 6restartPolicy Never or OnFailure
CronJob batch/v1Backups, nightly reportsA new Job per schedule tickRuns may overlap (Allow). Use spec.timeZone
In short : Deployment owns ReplicaSets, ReplicaSets own Pods by label, kubelet restarts crashes.

1.Scaling by hand

Scaling only edits spec.replicas. No rollout, no new revision.

$ kubectl scale deployment/web --replicas=5

Undone by an apply that sets replicas, or an HPA.

Scale-down : who goes first
  1. Not yet on a node
  2. Pending, Unknown
  3. Not ready
  4. Lower deletion cost
  5. Doubled up on a node
  6. Ready for less time
  7. More restarts
  8. Newer

The first rule that differs decides (v1.37). The cost is the controller.kubernetes.io/pod-deletion-cost annotation: the one sure lever.

2.HorizontalPodAutoscaler

An HPA (autoscaling/v2) edits replicas for you. Every 15s its controller reads metrics and writes the target's /scale.

The mathdesired = ceil( current x metric / target )
4 Pods at 90% CPU, target 70% : ceil(4 x 90 / 70) = ceil(5.14) = 6, kept within min..max.
kubeletevery node Metrics adaptercustom / external metrics scraped 15s custom APIs metrics-serveradd-on serves kube-apiservermetrics.k8s.io/v1beta1 metrics HPA webautoscaling/v2 spec HPA controllerkube-controller-manager sets replicas Deployment web/scale subresource owns ReplicaSet, Podsmore or fewer Pods new load

metrics-server is an add-on that kubeadm does not install, see metrics-server. Custom or external metrics need an adapter, such as KEDA.

3.HPA manifest and defaults

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: {name: web}
spec:
  scaleTargetRef: {apiVersion: apps/v1,
                   kind: Deployment, name: web}
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target: {type: Utilization,
               averageUtilization: 70}
FieldDefaultNote
minReplicas10 needs an Object or External metric (beta)
maxReplicasrequiredAt least minReplicas
metrics80% CPUHighest result wins
scaleUpno wait+100% or +4 Pods per 15s
scaleDown300sHighest pick in that window
tolerance10%Per-HPA field GA in v1.37
sync15skube-controller-manager
On the DeploymentTarget the Deployment, not its ReplicaSet. Leave out replicas or each apply resets it (removing it later can drop to 1 once). Set requests.cpu : 100m, so 70% is 70m.

4.Commands

CommandWhat it does
kubectl create deployment web --image=nginx:1.29 --replicas=3Create one with 3 Pods
kubectl get deploy,rs,pods -l app=web --show-labelsThe whole chain, with pod-template-hash
kubectl get pods -l 'env in (prod, qa)'Set-based selector query
kubectl scale --current-replicas=2 --replicas=3 deployment/webScale only if it is still at 2
kubectl autoscale deployment web --min=2 --max=10 --cpu=70%Make an HPA (--cpu-percent is deprecated)
kubectl get hpa web --watchCurrent vs target metric, replicas
kubectl describe hpa webConditions (ScalingActive, ...) and events
kubectl delete deployment web --cascade=orphanRemove it, keep ReplicaSets and Pods
In short : kubectl scale sets replicas once. An HPA recomputes them every 15s from metrics.

Core Kubernetes · 05

Services

1.Why a Service ?

Pods come and go, and each new Pod gets a new IP. A Service puts one stable DNS name, virtual IP and port in front of the Pods its label selector matches.

  • Clients use the name. Traffic goes to ready Pods (if none is ready, to Pods still serving while they stop).
  • L4 only (TCP, UDP, SCTP). HTTP routing and TLS : Ingress or Gateway API.
Not a box in the pathNo Service process exists. The ClusterIP is virtual : no host owns it and nothing answers on it. Rules on the client's own node rewrite each new connection to a real Pod IP. No Pod to pick ? DNS still answers, and a REJECT refuses the call after about a second. internalTrafficPolicy Local with no Pod on that node is a DROP : a timeout.

2.How it works

Two watch loops. No traffic goes through either.

  1. The EndpointSlice controller (kube-controller-manager) watches the selector and the matching Pods.
  2. It writes Pod IPs, ports and ready state into EndpointSlices labelled kubernetes.io/service-name=web (100 per slice by default).
  3. kube-proxy on every node watches both and programs kernel rules for 10.96.0.15:80.
Service web 10.96.0.15:80 Matching Pods listen on 8080 Slice controller kube-controller-manager watch watch, ready ? writes EndpointSlice PodIP:8080 + ready flag kubernetes.io/ service-name=web kube-proxy on every node watch watch programs node rules iptables or nftables
Each new connection
Client Podasks DNS for web.prod
CoreDNSanswers 10.96.0.15
Client Poddials 10.96.0.15:80
Its own nodeDNAT to one ready PodIP:8080
Podgets it on 8080

One Pod per connection. iptables walks a chain of random-probability jumps, nftables does one verdict-map lookup, ipvs uses a kernel IPVS virtual server (round robin). The ClusterIP never goes on the wire.

3.The types nest

Types are layers, not rivals. Each keeps what the smaller one gets, so NodePort and LoadBalancer Services still answer inside on their ClusterIP.

A LoadBalancer drops its NodePorts only with allocateLoadBalancerNodePorts: false. ExternalName is the odd one : only a DNS CNAME, nothing is proxied.

LoadBalancer + an external IP from an LB NodePort + the same port on every node ClusterIP (default) ExternalName DNS CNAME only Headless clusterIP: None
TypeWhat it getsReach it fromGood for
ClusterIPA virtual IP from the service rangeInside the cluster onlyService to service calls
HeadlessNo IP. DNS returns the Pod IPsInside, through DNSStatefulSets, peer discovery
NodePortClusterIP + a port on every node+ outside on NodeIP:nodePortYour own external LB, labs
LoadBalancerNodePorts + an external LB IP+ outside on the LB IPProduction L4 (cloud or MetalLB)
ExternalNameNo IP. A CNAME to externalNameInside, cluster DNS onlyAlias for an outside host

4.port, targetPort, nodePort

  • portwhat clients dial on the ClusterIP
  • targetPortthe Pod's port. Defaults to port, or names a containerPort
  • nodePortthe same port on every node
  • protocolTCP (default), UDP or SCTP
Client outside Client in a Pod NodeIP:30007 nodePort 10.96.0.15:80 port node rules kube-proxy DNAT PodIP:8080 targetPort range 30000-32767 by default fixed picks : 30000-30085
In short : A Service is a stable name and virtual IP over the Pods its selector picks. EndpointSlices track the Pods, kube-proxy turns them into node rules, and the types nest.

1.DNS, headless, ExternalName

Every Service gets <svc>.<ns>.svc.cluster.local.

  • Plain web only works in the same namespace. Elsewhere use web.prod.
  • SRV : _http._tcp.web... needs a named port.
  • Headless : clusterIP: None (the word, not an empty field). No VIP, kube-proxy skips it.
  • No selector : no EndpointSlices. You write them (outside DB, migrations).
Client Pod asks DNS ClusterIP also NodePort, LB Headless clusterIP: None ExternalName points at a host A = ClusterIP kube-proxy DNATs A = Pod IPs client picks one CNAME client goes direct

BusyBox 1.37 nslookup : a dotless name gets only the search list, a dotted one is sent as typed, exit 1 on any NXDOMAIN. An SRV query with no _http._tcp lists every port.

2.Traffic from outside

Kubernetes ships no load balancer. A cloud-controller-manager, or the controller named in loadBalancerClass, builds one and fills .status.loadBalancer. Until then EXTERNAL-IP says <pending>. Bare metal : MetalLB.

externalTrafficPolicy (NodePort and LB) :

  • Clusterdefault. Any node, SNAT, the app sees a node IP.
  • Localclient IP kept. Nodes with no local Pod fail the LB health check.
Client source IP C Load balancer cloud LB or MetalLB Cluster only nodePort Node A (no Pod) Local : health check fails, skipped Node B Pod Local : C kept Cluster : SNAT, client IP lost

Local drops only outside traffic : a Pod, or the node itself, calling a node with no local Pod is still answered.

3.Traffic knobs

  • sessionAffinity : None or ClientIP (timeout 3 h by default, 24 h max).
  • internalTrafficPolicy : Local = same-node endpoints only, or a drop (a timeout).
  • trafficDistribution : PreferSameZone or PreferSameNode. A hint, Local wins.
kube-proxyStatus in v1.37
iptablesDefault on Linux
nftablesStable since v1.33, a future default
ipvsDeprecated in v1.35, removal planned v1.43
kernelspaceThe only Windows mode

Pin the mode yourself before the default flips.

4.Commands

CommandWhat it does
kubectl expose deployment web --port=80 --target-port=8080Service using the Deployment's selector (add --type)
kubectl create service clusterip web --tcp=80:8080Selector app=web. --clusterip="None" = headless
kubectl create service externalname db --external-name my.database.example.comCNAME alias for an outside host
kubectl get svc web / kubectl describe svc webTYPE, CLUSTER-IP, EXTERNAL-IP, PORT(S), selector
kubectl get endpointslices -l kubernetes.io/service-name=webEvery endpoint, ready or not. Empty = bad selector or no Pods
kubectl get endpoints webReady ones only, like describe svc, after Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
kubectl run tmp -it --rm --restart=Never --image=busybox:1.28Throwaway Pod, then nslookup web.prod
curl http://localhost:10249/proxyModeOn a node : which mode kube-proxy runs

5.Gotchas & Good to know

Gotchas
  • One Pod per connection : gRPC and HTTP/2 stick.
  • A bare web from another namespace leaves the cluster : web is a public TLD.
  • ExternalName breaks Host and TLS names.
  • Env vars skip Pods older than the Service.
Good to know
  • Two or more ports : each needs a name.
  • nftables : NodePorts on the primary IP only.
  • Skip externalIPs (deprecated v1.36) and loadBalancerIP (deprecated v1.24).
  • v1.37 : names may start with a digit.
In short : DNS maps a Service to a ClusterIP, Pod IPs or a CNAME. Outside traffic needs an LB implementation, and externalTrafficPolicy decides if the client IP survives.

Core Kubernetes · 06

ConfigMaps and Secrets

1.What are they ?

Two core v1 objects that keep settings out of the container image. Same shape, different job.

  • ConfigMapPlain config, no secrecy at all. data holds UTF-8 text, binaryData holds bytes (base64 in YAML).
  • SecretA small confidential value (password, token, key), kept out of images and Pod specs.
  • BothNamespaced, max 1 MiB. A Pod only references ones in its own namespace.
  • Base64An encoding, not encryption. Y2hhbmdlLW1l is just change-me, one decode away.
ConfigMapSecret
Made forPlain settingsSmall confidential values
Value fieldsdata, binaryDatadata (base64), stringData (plain text, write-only)
typeNoneOpaque if not set
In etcdPlaintextPlaintext too, by default
At restEncryption is opt-inEncryption is opt-in
On nodesNothing specialOnly nodes whose Pods need it, in tmpfs, gone with the Pod
describePrints valuesByte counts only, but a token Secret's token in full
Volume fieldconfigMap.namesecret.secretName

2.How a Pod reads them

Four ways in : env vars, $(VAR) in command or args, files in a read-only volume, or app code calling the Kubernetes API. The kubelet watches each object and hands it to the Pod.

ConfigMap / Secret stored API server + etcd watch kubelet cache get / watchif RBAC allows envread once volumeup to ~90 s subPathread once APIlive watch restart Pod app re-reads restart Pod app reacts
containers:
- name: app
  image: nginx
  envFrom:               # every key, once
  - configMapRef:
      name: app-config
  volumeMounts:
  - name: creds          # files, refreshed
    mountPath: /etc/creds
volumes:
- name: creds
  secret:
    secretName: db-creds # not .name
    optional: true       # start if missing

3.What sees an edit ?

Read asPod spec fieldSees edits ?To pick up a change
One key as env varconfigMapKeyRef / secretKeyRefNoRestart the Pods
All keys as env varsenvFrom : configMapRef / secretRefNoRestart the Pods
Command / args$(VAR_NAME) from an env varNoRestart the Pods
Volume, whole mountvolumes[].configMap / .secretYes, within ~90 sApp re-reads the file
Volume with subPathvolumeMounts[].subPathNoRestart the Pods
Kubernetes APIApp code, get / watchYes, liveApp handles the event
Immutable objectimmutable: trueLockedNew name, then roll the Pods

Restart = kubectl rollout restart deployment/NAME (see Deployments). A container restart also re-reads env, and subPath once its volume has updated. Volumes follow in 60 to 90 s, Pod by Pod; a Pod annotation refreshes at once. The kubelet skips watching immutable objects; reusing one's name can leave a node on the old copy.

In short : ConfigMaps hold plain settings, Secrets hold small confidential values. Env vars are copied once, whole-volume files follow edits eventually, and base64 protects nothing.

1.Who can read a Secret ?

etcd or a backupplain unless encrypted get secretsreads that one list / watch secretsreads every value in ns Secretstored in etcd sent tothe node create Podsin that namespace Pod mounts itany Secret in its ns node tmpfs copydeleted with the Pod privileged containeron the same node

Default cluster, no hardening : every arrow is a way in. Close them one by one.

  • etcdEncryption at rest (this path only).
  • RBAClist or watch on secrets = read-all.
  • PodsPod-create = read. Split by namespace.
  • NodesNo privileged containers. Mount Secrets only where needed.
  • SourceExternal store, see External Secrets.

2.Encryption at rest

Off by default. Pass an EncryptionConfiguration with kube-apiserver --encryption-provider-config; edits need a restart unless its automatic reload flag is on. Use KMS v2 (stable v1.29), or secretbox (not aescbc) for a local key.

kind: EncryptionConfiguration
apiVersion: apiserver.config.k8s.io/v1
resources:
- resources: [secrets, configmaps]
  providers:
  - kms:                # 1st encrypts writes
      apiVersion: v2
      name: my-kms
      endpoint: unix:///run/kms.sock
  - identity: {}        # no crypto, migration only
Secret write any client kube-apiservernew DEK per write, AES-GCM KMS v2 plugingRPC, UNIX socket wrap Remote KMSholds the KEK etcdk8s:enc:kms:v2:... ciphertext +wrapped DEK stolen etcd copy: not enough

3.Secret types

All but Opaque and bootstrap start with kubernetes.io/.

OpaqueAny keys, the default
tlstls.crt + tls.key
dockerconfigjsonfor imagePullSecrets, as is dockercfg
service-account-tokenlegacy, not auto-made since v1.24

Also built in : basic-auth, ssh-auth, dockercfg (legacy), bootstrap token. For SA tokens use kubectl create token.

4.Gotchas

  • immutable: true (GA v1.21) cannot be undone.
  • Encryption applies on write: rewrite old ones (kubectl replace, StorageVersionMigration).
  • binaryData keys never become env vars.
  • Static Pods can never use Secrets or ConfigMaps (v1.37).
  • Many small Secrets eat memory. Cap the count (ResourceQuota).

5.Commands

CommandWhat it does
kubectl create configmap NAME --from-file=FILEAlso --from-literal, --from-env-file. Key = file name.
kubectl create secret generic NAME --from-literal=k='v'Opaque Secret. Single-quote special characters.
kubectl get secret NAME -o jsonpath='{.data.KEY}' | base64 --decodeRead one value. Base64 hides nothing.
kubectl create secret ... --dry-run=client -o yamlPrints plain base64. Keep it out of Git, or encrypt it.
kubectl auth can-i list secrets --as=USER -n NSCan USER read every Secret in NS ?
kubectl get secrets -A -o json | kubectl replace -f -Rewrite all Secrets with the current first provider.
In short : Anyone with etcd access, list on secrets or Pod-create rights can read a Secret. Turn on encryption at rest (KMS v2), keep RBAC tight, and rewrite old objects after any key change.

Core Kubernetes · 07

Volumes and Storage

1.Why volumes ?

A crashed or stopped container comes back with a clean filesystem, so the files it wrote are lost. Containers in one Pod also need a place to share files.

  • Every volume survives a container restart.
  • Ephemeral volumes die with the Pod. Persistent ones (a PV used through a PVC) outlive it.
  • Declare in spec.volumes, mount per container with volumeMounts (mountPath, readOnly, subPath). Volumes do not nest.
Pod app sidecar mount emptyDirshared scratch gone when the Pod goes claimName PVC 1 : 1 PVreal storage outlives the Pod

2.Volume types by lifetime

TypeLivesGood forWatch out
emptyDirPodscratch, sharing filesmedium: Memory = tmpfs (RAM). Set sizeLimit.
configMap, secret, downwardAPIPodconfig, secrets, Pod infoRead-only files. subPath mounts update only on a restart.
projectedPodmany sources, one folderSources must be in the Pod's namespace.
ephemeral (generic)Podper-Pod scratch PVCPVC <pod>-<volume> is deleted with the Pod.
csi (inline)Poddriver-specific volumesThe CSIDriver must list the Ephemeral mode.
hostPathnodenode agents, static PodsSecurity risk. Contents differ per node.
persistentVolumeClaimPVdata that has to lastPVC must be in the Pod's namespace.
local (PV only)PVfast disks on one nodeStatic, needs nodeAffinity. Node down = Pod stuck.
Avoid hostPathIt mounts a node path into the Pod and can expose kubelet credentials or the container runtime socket, a way out of the container. Use a local PV for node disks. A hostPath PV without node affinity is for one-node tests.
Good to knowemptyDir is created when the Pod lands on a node and deleted when it leaves. projected merges secret, configMap, downwardAPI, serviceAccountToken, clusterTrustBundle and podCertificate. More in ConfigMaps and Secrets.

3.PV, PVC and StorageClass

  • PV : cluster-scoped, static (admin) or dynamic.
  • PVC : namespaced request: size, modes, class.
  • Binding : 1:1, PV claimRef + PVC volumeName.
  • Pod : uses claimName, PVC in the same namespace.
  • StorageClass : names a provisioner + parameters.
  • Provisioner : makes the PVs, often a CSI driver you install (in-tree cloud types too, v1.25).
  • Kubernetes ships no built-in NFS provisioner.
Access modeMeans
ReadWriteOnce RWOread-write, one node
ReadOnlyMany ROXread-only, many nodes
ReadWriteMany RWXread-write, many nodes (NFS-like)
ReadWriteOncePod RWOPread-write, one Pod

Modes match claims to PVs. RWO keeps an attached or local volume on one node; Pods there share it. RWOP admits one Pod.

4.Reclaim policy

  • Delete : deleting the PVC removes the PV and its storage. Default for dynamic PVs (copied from the class).
  • Retain : PV goes Released, data kept for manual cleanup. Default for hand-made PVs.
  • Recycle : deprecated.
Pending Available Bound new PV claimed DeletedPV + storage Releaseddata kept Failedreclaim broke PVC deleted Retain : clean up,drop claimRef Delete fails A Released PV never binds a new claim by itself.
In short : Every volume outlives a container restart. Ephemeral ones end with the Pod, a PV does not, and a PVC binds 1:1 to a PV that a CSI driver usually creates for it.

1.Dynamic provisioning

With volumeBindingMode: WaitForFirstConsumer :

  1. Admin creates a StorageClass.
  2. User creates a PVC. It is Pending.
  3. User creates a Pod with claimName.
  4. Scheduler picks a node for the Pod.
  5. Driver makes a volume for that node.
  6. Driver creates the PV.
  7. PV and PVC bind. PVC is Bound.
  8. kubelet mounts it into the Pod.
Immediate (default)Steps 5 to 7 follow step 2 at once, blind to where the Pod may run. Static: an admin makes PVs, a matching one binds.
Admin User kube-scheduler StorageClasswaits for a Pod PVCPending PodclaimName CSI driverprovisioner PVclaimRef: PVC kubeleton that node Backendnode's topology 1 2 3 4 class names 5 6 7 8 volume attached to that node

2.StatefulSet claims

  • One PVC per Pod per template: <template>-<sts>-<ordinal>.
  • A moved Pod gets its old PVC back (more).
  • Scale down or delete keeps PVCs, unless persistentVolumeClaimRetentionPolicy says Delete (v1.32).
  • Templates are immutable: resize each PVC.
StatefulSet webtemplate: www creates web-0keeps its PVC web-1gone after scale to 1 mounts www-web-0PVC, own PV www-web-1PVC kept for scale up

3.Commands

CommandWhat it is for
kubectl get pv,pvc -AStatus, claim, class, reclaim policy of all of them
kubectl get scProvisioner, binding mode, expansion, (default)
kubectl get csidriversInstalled CSI drivers (Persistent, Ephemeral)
kubectl describe pvc <name> -n <ns>Events say why it is Pending
kubectl patch pv <pv> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'Keep the PV and data before deleting the claim. Back to Delete, local-path cleans it while Released, not once Available
kubectl patch pvc <pvc> -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'Grow the volume, then follow the PVC events
kubectl patch pv <pv> --type=json -p '[{"op":"remove","path":"/spec/claimRef"}]'Cleaned Released PV back to Available. Remove only /spec/claimRef/uid to keep it for the same claim name
kubectl delete pvc <name> -n <ns>Careful : under Delete the data goes too. In use, it stays Terminating and kubectl waits (--wait=false skips it)

4.Gotchas & good to know

Gotchas
  • WaitForFirstConsumer : Pending until a Pod is scheduled is normal. A nodeName pin keeps it stuck.
  • No storageClassName = default class (even one added later). "" = class-less PVs only.
  • Two default classes: the newest wins. Keep one.
  • Volumes only grow. Never edit a PV's capacity.
Good to know
  • Grow : allowVolumeExpansion: true and a driver that can grow, then raise the PVC request. Failed? Lower it to above the old size (v1.34). FS: XFS, ext3, ext4 (online OK).
  • Snapshots : CSI only, CRDs from your distribution. Restore via PVC dataSource.
  • Databases want an operator, see CloudNativePG.
In short : WaitForFirstConsumer lets the Pod's node pick where the volume lands, StatefulSets keep one PVC per replica, volumes only grow, and Retain guards data you care about.

Core Kubernetes · 08

Namespaces and Ingress

1.What is a Namespace ?

A scope for names. A name is unique inside its namespace, and the next namespace can reuse it.

  • No nesting. Each namespaced object lives in exactly one.
  • Name = RFC 1123 DNS label. The fixed label kubernetes.io/metadata.name is added for you.
  • Deleting one deletes all inside (phase Terminating): Pods first, then the rest.
  • Production : make your own, skip default.
Built inUsed for
defaultYour objects when you name no namespace
kube-systemObjects Kubernetes itself creates
kube-publicReadable by anyone, even anonymous. Mostly cluster use
kube-node-leaseOne Lease per node, the kubelet heartbeat for spotting dead nodes

The kube- prefix is reserved for system namespaces.

2.Names and policy, not traffic

Each of these is namespaced and acts only at home :

  • Role + RoleBinding : RBAC, here only
  • ResourceQuota : total cap, 403 if over
  • LimitRange : per Pod / Container limits
  • NetworkPolicy : traffic (section 4)
Not a firewallBy default any Pod reaches any Pod, in any namespace. Pod a1 calls api.team-b and nothing stops it.
Cluster ns team-a Pod a1 Role + RoleBinding ResourceQuota LimitRange ns team-b Service api forwards Pod b1 team-a's rules stop here api.team-b allowed Cluster-scoped, in no namespace Node, PersistentVolume, StorageClass, IngressClass, ClusterRole, Namespace

DNS crosses the line too. A Service is <svc>.<ns>.svc.cluster.local. A bare api only finds your own namespace's api (the search list starts there), so use api.team-b or the full name.

3.Namespaced or not ?

KindIn a ns ?Short
NamespaceNons
ResourceQuotaYesquota
LimitRangeYeslimits
Role / RoleBindingYesnone
ClusterRoleNonone
NetworkPolicyYesnetpol
IngressYesing
IngressClassNonone

apiVersion : v1 for the first three, rbac.authorization.k8s.io/v1 for RBAC, networking.k8s.io/v1 for the last three.

4.NetworkPolicy : the real wall

  • Namespaced L3/L4 allow-list for TCP, UDP, SCTP.
  • Pods start open. Once a policy selects a Pod for Ingress or Egress, only allowed traffic passes. Policies add up, no deny rules.
  • Both ends must agree : egress at the source and ingress at the target.
  • No policyTypes ? Ingress is implied. Egress only if it has egress rules.
  • Needs an enforcing network plugin (Antrea, Calico, Cilium, Kube-router, kind's kindnet). Without one it is stored and ignored, with no warning.
ns web frontend ns dev other ns prod api default-deny-ingress selects all Pods, no rules allow-from-web ingress from namespace web both select api allowed blocked times out or refused CNI plugin that enforces NetworkPolicy Calico, Cilium, Antrea, Kube-router, kindnet enforces
Gotchas
  • A policy cannot name a namespace. Match the label kubernetes.io/metadata.name=web.
  • In one from entry, namespaceSelector + podSelector mean AND. As two list items (two dashes) they mean OR.
  • Default-deny egress also blocks DNS. Allow the CoreDNS Pods.
  • Test a deny policy before you trust it.
In short : A namespace scopes names and the policy you attach (RBAC, ResourceQuota, LimitRange). It does not block traffic. That takes a NetworkPolicy and a CNI that enforces it.

1.Ingress : rules, not a proxy

An Ingress (networking.k8s.io/v1, namespaced) is just HTTP routing rules. A controller carries the traffic.

  • No controller ? Stored, does nothing.
  • HTTP / HTTPS only (80, 443). Else use a NodePort or LoadBalancer Service.
  • Backends : Services in its own namespace.
  • v1.37 : GA since v1.19, frozen. It stays, but the project recommends Gateway API.
Client shop.example.com HTTPS :443 Load balancer / edge set up by the controller Ingress controller proxy Pods /api / Service api Service web Pods API server config only, no traffic IngressClass spec.controller ingressClassName Ingress shop host + path rules ns shop tls.secretName Secret shop-tls kubernetes.io/tls backends : same ns reads

2.Inside an Ingress

One host, two paths, TLS
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: {name: shop, namespace: shop}
spec:
  ingressClassName: example-class
  tls:                         # hosts = rule hosts
  - {hosts: [shop.example.com], secretName: shop-tls}
  rules:
  - host: shop.example.com
    http:
      paths:
      - path: /api
        pathType: Prefix     # required
        backend:
          service: {name: api, port: {number: 8080}}
      - path: /
        pathType: Prefix
        backend:           # port: number OR name
          service: {name: web, port: {name: http}}

Host first (exact, or *.foo.com = one label), then the longest path, Exact winning ties. No match : spec.defaultBackend, else the controller decides.

TypePathRequestMatch ?
Exact/foo/foo/no
Prefix/foo/bar/foo/bar/bazyes
Prefix/foo/bar/foo/barbazno, whole parts
Prefix/foo/bar//foo/baryes
ImplementationSpecific : up to the IngressClass
  • TLS ends at the LB or controller, plain text after. Port 443 only, hosts share it by SNI.
  • Secret type kubernetes.io/tls (tls.crt, tls.key). A rule with no host gets no TLS. Certs : cert-manager.

3.Picking a controller

IngressClass
  • Cluster-scoped. spec.controller (domain-prefixed, immutable) names the controller.
  • Default : ingressclass.kubernetes.io/is-default-class: "true". Two defaults = class-less Ingresses rejected.
  • kubernetes.io/ingress.class annotation : deprecated.
ingress-nginx is retired
  • Last releases 2026-03-19, archived 2026-03-24. No more fixes, not even security. Installs keep running.
  • Not F5's NGINX Ingress. nginx.ingress.kubernetes.io/* annotations don't carry over.
  • Way out : Gateway API (ingress2gateway 1.0 converts) or another controller.

4.Commands

CommandWhat it tells you
kubectl get ingressclassClasses and their controllers
kubectl get ing -AEmpty ADDRESS = no controller took it (yet)
kubectl describe ingress shop -n shopRules, backends with endpoints, events
kubectl create ingress shop --rule="shop.example.com/api*=api:8080" --dry-run=client -o yamlDraft YAML. Trailing * = Prefix, else Exact
kubectl create secret tls shop-tls -n shop --cert=tls.crt --key=tls.keyThe Secret for spec.tls
kubectl get pods -A --selector app.kubernetes.io/name=ingress-nginxStill running the retired controller ?
In short : An Ingress is HTTP routing config, and a controller does the work. The API is frozen and ingress-nginx is retired, so start new setups on Gateway API.

Core Kubernetes · 09

Rollouts and Probes

1.What starts a rollout ?

A rollout swaps Pods from an old ReplicaSet to a new one. Pods are never edited in place.

  • Trigger : any change under .spec.template (image, env, labels...). Each one makes a new ReplicaSet, a new revision.
  • Changing replicas is scaling, not a rollout.
  • .spec.strategy.type : RollingUpdate (default) or Recreate.
  • Recreate : all old Pods gone first, then the new ones. Downtime by design.

ReplicaSets and scaling : Deployments and Scaling.

New template image, env, labels triggers Deployment controller moves one step at a time up, total 5 at most down, keep 3 or more available New ReplicaSet rev N+1, scales up Old ReplicaSet rev N, scales down creates a Pod New Pod is Ready readinessProbe passes Available after minReadySeconds next step replicas: 4 maxSurge 25% = 1 maxUnavailable 25% = 1 so 3 to 5 Pods during the update

2.Step size

  • maxSurgePods allowed above replicas. 25%, rounds up.
  • maxUnavailablePods allowed to be unavailable. 25%, rounds down.
  • both 0Rejected : the rollout could never move.

A number or a percent. It is not one Pod at a time : 10 replicas start 3 extra Pods and drop 2 at once.

Terminating Pods do not count as available, so the real count can briefly pass max Pods until their grace period ends.

replicassurgeunavailablemin availablemax Pods
11012
31034
41135
1032813
Not a strategy typeBlue/green flips one Service selector from the old Deployment to the new one. Canary runs both behind one shared label, or splits traffic by weight (Gateway API, Knative, Flagger from the Flux project).

3.Readiness sets the pace

A new Pod is available once it is Ready and stays Ready for minReadySeconds (default 0). Beyond maxUnavailable, old Pods only go as new ones become available.

  • No readinessProbe ? Ready the moment it starts, so traffic and the rollout run ahead of the app.
  • A new Pod that never gets Ready stalls the rollout. Old Pods keep serving.
  • Capacity can dip by default. maxUnavailable: 0 keeps full capacity.
Progressingreasonrollout status
TrueNewReplicaSetAvailableexits 0
TrueReplicaSetUpdatedkeeps waiting
FalseProgressDeadlineExceededexits 1
Stalled is only a statusNo progress for progressDeadlineSeconds (default 600, must be greater than minReadySeconds) gives the False row. The controller keeps retrying and never rolls back. That is your job.

4.Revisions and rollback

  • Each ReplicaSet is a revision. revisionHistoryLimit (default 10) is how many old ones stay. 0 means no undo.
  • rollout undo reverts the Pod template only.
  • The reused ReplicaSet gets a new number : rev 2 comes back as rev 4.
  • Resume a paused Deployment before you undo.
rev 1 nginx:1.14.2 rev 2 nginx:1.16.1 rev 3 nginx:1.161 (typo) each set image = a new ReplicaSet ImagePullBackOff, stalls, you act rollout undo --to-revision=2 rev 4 rev 2's ReplicaSet reused undo patches the template, rev 3 scales to 0 Pods.
In short : A template change makes a new ReplicaSet, maxSurge and maxUnavailable size each step, readiness sets the pace, and a stalled rollout waits for you to undo it.

1.Three probes, three jobs

The kubelet runs the probes. Liveness and readiness run side by side and never wait for each other.

Container starts initialDelaySeconds startupProbe if set, runs first Kill container then restartPolicy backoff up to 300s fails restart first success livenessProbe every periodSeconds fails readinessProbe whole life fails passes Ready = False no new Service traffic keeps running Ready = True gets Service traffic counts for rollouts
ProbeOn failureUse it for
startupKill, restartPolicySlow starters. The others wait.
livenessRestart, same Pod, same nodeHangs. Never a dependency check.
readinessNo Service traffic, no restartWarm-up, overload, deps down

One check per probe. No probe counts as Success.

  • execthe command exits 0
  • httpGetstatus 200 to 399
  • tcpSocketthe port accepts a connection
  • grpchealth check says SERVING (GA v1.27)

2.Probes in a Pod template

containers:
- name: nginx
  image: nginx:1.16.1
  ports: [{name: http, containerPort: 80}]
  startupProbe:
    httpGet: {path: /, port: http}
    periodSeconds: 5
    failureThreshold: 12  # up to 60s to start
  readinessProbe:
    httpGet: {path: /, port: http}
    periodSeconds: 5
  livenessProbe:
    tcpSocket: {port: http}

3.Timing

FieldDefaultNote
initialDelaySeconds0From container start. Liveness and readiness also wait for startupProbe
periodSeconds10How often
timeoutSeconds1One try fails after this
successThreshold1Only 1 for liveness, startup
failureThreshold3Fails in a row = failed
Startup budgetfailureThreshold x periodSeconds. Here 12 x 5 = 60s. The docs example uses 30 x 10 = 300s.

4.Readiness gates

  • spec.readinessGates[].conditionType adds a Pod condition that an outside controller sets.
  • Ready = containers ready and every gate True. Missing counts as False.
PodDisruptionBudgetpolicy/v1. Limits voluntary evictions (Eviction API, kubectl drain). Deleting Pods and rolling updates skip it, since maxUnavailable is the rollout's own budget. Healthy = Ready, and maxUnavailable: 0 means drain never ends.

5.Commands

CommandWhat it does
kubectl set image deployment/web nginx=nginx:1.16.1Changes the template image, so a rollout starts
kubectl rollout status deployment/web --timeout=5mWaits. Exit 0 when done, 1 once past the deadline
kubectl annotate deployment/web kubernetes.io/change-cause="to 1.16.1"Fills CHANGE-CAUSE in history (replaces --record)
kubectl rollout history deployment/web --revision=2Lists revisions, or one revision's template
kubectl rollout undo deployment/web --to-revision=2Back one revision, or to N. --dry-run=server previews
kubectl rollout pause / resume deployment/webBatches edits into one revision (Deployments only)
kubectl rollout restart deployment/webNew Pods for all, through a normal rolling update
kubectl describe pod <pod>Probe failures show as Warning Unhealthy events
kubectl create pdb web-pdb --selector=app=web --min-available=50%A budget (or --max-unavailable, never both)
In short : Startup holds the other probes off, liveness restarts a stuck container, readiness pulls a Pod out of traffic, and that same Ready flag paces rollouts and PDBs.

Core Kubernetes · 10

StatefulSets and DaemonSets

1.Why a StatefulSet ?

Deployment Pods are interchangeable. A StatefulSet is a set of Pods with names. It promises four things. The first two survive a reschedule :

  1. a fixed name and DNS name per Pod
  2. its own storage per Pod
  3. ordered start and scaling
  4. ordered rolling updates

Need none of them ? Use a Deployment (Deployments and Scaling). Stable apps/v1 since v1.9.

A Pod in aDeploymentStatefulSet
Name<rs>-<random><sts>-<ordinal>
Replaced bya new Pod, new namesame name, same PVC
DNS namenone per Podone per Pod
Storageshared PVC or noneown PVC per Pod
Orderany0 first, top down
Not an operator : no leader election, replication, failover or backups. That is what Operators add (see CloudNativePG).

2.Identity : name, DNS name, PVC

you create it Service nginx clusterIP: None a DNS name per Pod web-0 web-0.nginx web-1 web-1.nginx web-2 web-2.nginx mounts www-web-0 www-web-1 www-web-2 web-1 moves : same name, same DNS name, same PVC
PieceWhat web-1 gets
Name, hostnameweb-1
DNS nameweb-1.nginx.<ns>.svc.cluster.local
PVCwww-web-1 : template, set, ordinal
Labelsapps.kubernetes.io/pod-index: "1", statefulset.kubernetes.io/pod-name: web-1

The pod-name label lets a Service pick one replica. ordinals.start (GA v1.31) shifts the range : start 1 gives db-1 to db-3.

3.The headless Service

  • You create it and name it in serviceName. Deleting the set leaves it behind.
  • Optional since v1.33, but no DNS names without it. Nobody checks that it exists.
  • DNS lists Ready Pods only. publishNotReadyAddresses: true lets peers meet while booting.
  • Use names, not Pod IPs : IPs change. See Services.

4.Own PVC, in short

  • volumeClaimTemplates : one PVC per template per ordinal. A replaced Pod mounts the same one.
  • PVCs stay on scale down and delete, unless the retention policy (GA v1.32) says Delete.
  • Templates are immutable : grow each PVC by hand (Volumes and Storage).
  • Zonal disks : use WaitForFirstConsumer.

5.Order : OrderedReady vs Parallel

OrderedReady (default)Parallel
Scale up0 up, each waits for lower Pods Running and Readyat once
Scale downfrom the top, each waits until the previous is goneat once
Sick Podscale down waits, unless the sick one goes nextno wait
Change itimmutable : orphan delete, recreate
scale up web-0 Ready ? web-1 Ready ? web-2 scale down web-2 gone ? web-1 gone ? web-0 Parallel : all at once, no waiting

minReadySeconds makes each step wait for Available too. A Failed Pod (eviction, reboot) comes back with the same name.

6.At most one web-1

Never two Pods with one name : a Pod on a lost node stays Terminating, and no twin starts. Ways out, safest first :

  1. Machine really gone : delete the Node object.
  2. Node confirmed shut down : taint it node.kubernetes.io/out-of-service. Pods are force-deleted, volumes detach. Remove the taint after.
  3. Last resort : force delete the Pod. Risks split brain.
Careful
  • Never terminationGracePeriodSeconds: 0.
  • Deleting the set is not ordered : scale to 0 first.
  • With an HPA, leave replicas out of the manifest.
In short : A StatefulSet gives each Pod an ordinal name, a DNS name via the headless Service you create, and its own PVC, then starts Pods from 0 and removes them from the top.

1.Rolling update and partition

A template change makes a new ControllerRevision. RollingUpdate (default) replaces Pods from the highest ordinal down : same name, same PVC, new template.

  • Each step waits for Running and Ready, plus minReadySeconds (Rollouts and Probes).
  • partition (default 0) : only Pods at or above it update. Lower ones stay old, even when deleted.
  • It counts from ordinals.start : with start 5, partition 1 updates ord-6 and up.
3rd 2nd 1st web-0 rev 1 to 2 web-1 rev 1 to 2 web-2 rev 1 to 2 Ready ? Ready ? staged, with partition : Stage partition: 3 Canary partition: 2 Finish partition: 0 nothing moves only web-2 web-1, web-0

2.Three update strategies

RollingUpdate (default)OnDeleteRecreate (alpha, v1.37)
Who replacescontroller, top downyou, by deleting Podscontroller, all at once
Old and newside by sideside by side, no limitnever : downtime
rollout statusworksrefusesrefuses
Switch withdefault"rollingUpdate": nullsame, plus a feature gate

OnDelete : scaled-up Pods get the new template too. Recreate needs StatefulSetRecreateStrategy on kube-apiserver and kube-controller-manager; beta is planned for v1.38, not promised.

3.maxUnavailable (Beta)

  • Several Pods per step. On by default in v1.37 (default 1). In v1.35.4+ and v1.36 the field is dropped silently.
  • A percent rounds down, minimum 1. The docs say up.
  • OrderedReady : several go, but they return one at a time. Parallel : together. Every unavailable Pod counts.
maxUnavailablePods per step
1 (default)1
30%1
60%2
70%2

4 replicas, seen on v1.37.1. 60% again on a kind cluster.

4.The stuck rollout

A template that never gets Ready stops an OrderedReady rollout. Old Pods keep serving.

  • rollout undo alone is not enough : the controller waits on the stuck Pod forever.
  • Delete it : it comes back on the reverted template. Parallel (v1.37) and Recreate do this for you.
  • Upstream : working as intended, a Pod may hold data.
set image : bad tag new revision web-2 not Ready rollout stops rollout undo template reverted web-2 still stuck same bad revision delete pod web-2 your manual step web-2 recreated rollout goes on

5.Revisions and commands

CommandWhat it is for
kubectl get pods -l app=nginx -L controller-revision-hashWhich Pod runs which revision
kubectl patch sts web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'Canary : only web-2 takes the new template
kubectl rollout status sts/web --timeout=10mThe only deadline you get. Checks Ready, not minReadySeconds
kubectl rollout undo sts/web --to-revision=3Pod template only. The old revision returns with a new number
kubectl annotate sts/web kubernetes.io/change-cause='bump nginx'CHANGE-CAUSE in rollout history (--record is deprecated)

10 ControllerRevisions kept. No progressDeadlineSeconds, rollout pause fails. PDBs limit drains, not updates. GitOps ? Revert in Git.

In short : Updates go from the highest ordinal down, partition stages a canary, OnDelete waits for you, and a Pod stuck on a bad template must be deleted by hand after the revert.

1.DaemonSet : one Pod per node

A DaemonSet has no replica count : the list of eligible nodes is the count. A node joins, it gets a Pod. It leaves, its Pod is garbage-collected.

  • For node work : CNI and CSI node plugins, log and metrics agents, kube-proxy.
  • No /scale : no HPA, kubectl scale ds fails.
  • --cascade=orphan keeps its Pods for the next DaemonSet with that selector.
DaemonSet agent one Pod per eligible node node-1 Ready agent node-2 cordoned agent tolerated node-3 new, NotReady no Pod not tolerated cp-1 control plane no Pod not tolerated

2.Which nodes get a Pod ?

  1. nodeName in the template, if set, must match.
  2. nodeSelector and required node affinity must match.
  3. Every NoSchedule and NoExecute taint must be tolerated.

CPU and memory are not checked : a full node leaves the Pod Pending. Free room, or use priorityClassName: system-node-critical. Each Pod is pinned with matchFields metadata.name and the scheduler binds it (Advanced Scheduling).

Added tolerationEffectSo the Pod
not-ready, unreachableNoExecutenever evicted
disk, memory, pid pressureNoScheduleon busy nodes
unschedulableNoScheduleon cordoned nodes
network-unavailableNoSchedulehostNetwork only
Not added
  • not-ready:NoSchedule : a NotReady node gets no new Pod. CNI and kube-proxy tolerate it themselves.
  • The control-plane taint : add it yourself.

3.Updates : maxUnavailable or maxSurge

  • Default : maxUnavailable 1, maxSurge 0. Percents round up.
  • maxSurge needs maxUnavailable: 0. Old goes once new is Available.
  • OnDelete : only deleted Pods and new nodes update.
  • Bad template ? Push a fixed one : broken Pods go first, and use up the budget.
maxUnavailable: 1 old Pod rev 1 no Pod a gap new Pod rev 2 maxSurge: 1 old Pod rev 1 old + new 2 Pods new Pod old removed one node, during the update

4.Deployment vs StatefulSet vs DaemonSet

DeploymentStatefulSetDaemonSet
How manyreplicas, HPAreplicas, HPAone per eligible node
Storageshared PVC or noneown PVC per ordinalhostPath, CSI node mounts
OrdernoneOrderedReady or Parallelnone
StrategiesRollingUpdate, RecreateRollingUpdate, OnDelete, Recreate (alpha)RollingUpdate, OnDelete
Deadline, pauseyesnono
Good forstateless web, APIsdatabases, quorum, brokersCNI, CSI, log and metrics agents

5.Troubleshooting

SymptomCheckLikely cause, fix
Stuck at N-1/Nkubectl get pods -L controller-revision-hashbad template : revert, delete the stuck Pod
No StatefulSet Podskubectl describe sts webname over 52 characters, Pod Security, quota
Pending after a movekubectl describe pod web-2its volume is pinned to another zone
DESIRED too lownode taints and labelsuntolerated taint, or nodeSelector
DaemonSet CURRENT 0kubectl describe ds agentPod Security : use a privileged namespace (Pod Security)
Drain refuseskubectl drain outputadd --ignore-daemonsets : they tolerate cordon
DaemonSet rollout hangsevents, dead nodeshostPort with maxSurge, or a Pod stuck Terminating
In short : A DaemonSet runs one Pod on each node its selectors and taints allow, tolerates node trouble by itself, and updates node by node : off then on, or side by side with maxSurge.

Core Kubernetes · 11

Jobs and CronJobs

1.What is a Job ?

A Job runs Pods until enough of them succeed, then stops. A CronJob makes a new Job on a schedule. Both are stable batch/v1 workloads (see Deployments and Scaling).

  • job-controller in kube-controller-manager makes the Pods, counts the finished ones and writes the result.
  • restartPolicy must be Never or OnFailure. Unset means Always : rejected.
ObjectEndsLost Pod backTrigger
Jobyesyescreate
CronJobyesvia its Jobsschedule
Bare Podyesnocreate
Deploymentnoyes-
ScaledJobyesvia its Jobsevent

Use a Job even for one task : a bare Pod lost with its node is not replaced.

A Job can start the same work twice, even with parallelism 1. Make the work idempotent.

2.The retry loop

A Job is a retry loop with a failure budget. A failed Pod is replaced after a delay, until the failures pass backoffLimit.

  • backoffLimit counts failures you tolerate, not tries : the Job fails when failed > backoffLimit.
  • Default 6 : the 7th failed Pod ends the Job. 0 fails on the first.
  • Delay : 10s, doubling, capped at 10 min (the docs still say six). A success resets it.
  • activeDeadlineSeconds beats the budget. It counts from startTime, Pending and image pulls included.
Failure1234567+
wait10s20s40s80s160s320s600s
Job controller creates Pods create Pod Pod runs restartPolicy Never exit 0 fails succeeded + 1 delay back to 10s failed + 1 status.failed failed > backoffLimit ? yes Job Failed BackoffLimitExceeded no wait 10s to 10m replace the Pod enough ? Complete job done

3.Never vs OnFailure

NeverOnFailure
Retried bycontroller, new Podkubelet, same Pod
Delay cap10 min5 min (CrashLoopBackOff)
Budgetfailed Podsfailed Pods or restarts
At limitfailed Pods stayPod killed, logs may go
Default ofcreate jobcreate cronjob
Debug with NeverFailed Pods and their logs stay around to read. With OnFailure the Job also fails once container restarts in live Pods reach backoffLimit, and that count is known to be inexact (open issue).

4.Completions and parallelism

Kindcompletions / parallelismDone when
Singleunset / unsetone Pod succeeds
FixedN / PN Pods succeeded
IndexedN / Peach index 0..N-1 once
Queueunset / Pa success, none active
  • parallelism can change any time, 0 pauses. completions is frozen (Indexed : only with parallelism, same value).
Work queuePods share the queue and decide when it is empty. After the first success no new Pod starts. completions stays null (the docs say it copies parallelism; it does not).
IndexedPod i gets JOB_COMPLETION_INDEX and hostname <job>-<i>. A headless Service as subdomain lets peers find each other.
In short : A Job is a retry loop with a failure budget : failed Pods return after 10s, 20s ... up to 10 min, by default the 7th failure ends it, and completions and parallelism define done.

1.How a Job ends

Always in two steps : an interim condition first (the Pods still running are deleted), then the final one, with the same reason, once no Pod is left terminating.

Job running FailureTarget interim, Pods deleted Failed terminal SuccessCriteriaMet interim, Pods deleted Complete terminal no Pod left terminating failure checks run first

Pods deleted at the end count as failed : one exit 42 in a parallelism 3 Job showed failed: 3.

What firedReason on both
backoffLimitBackoffLimitExceeded
activeDeadlineSecondsDeadlineExceeded
podFailurePolicy FailJobPodFailurePolicy
maxFailedIndexesMaxFailedIndexesExceeded
a failed indexFailedIndexes
successPolicySuccessPolicy
completions reachedCompletionsReached

kubectl describe job shows no conditions : use get job -o yaml.

2.Pod failure policy

spec:
  podFailurePolicy:
    rules:
    - action: FailJob    # known bug
      onExitCodes:
        operator: In
        values: [42]
    - action: Ignore     # not our fault
      onPodConditions:
      - type: DisruptionTarget
  • FailJobFail the whole Job now.
  • FailIndexFail this index only (needs backoffLimitPerIndex).
  • IgnoreNot counted, but still delays the retry.
  • CountNormal. Also the result when no rule matches.

Rules run in order, first match wins (max 20). Exit 0 never matches. It forces restartPolicy: Never and podReplacementPolicy: Failed. GA v1.31.

DisruptionTarget : preemption, API eviction, taint manager, Pod GC or kubelet termination. Not your bug.

3.Per-index budget, success policy

  • backoffLimitPerIndex : each index gets its own retries. Indexed only, like the rest here (GA v1.33).
  • maxFailedIndexes : past it, the Job fails at once. Unset : every index runs, then any failed one fails the Job (not Complete, as the API comment says).
  • successPolicy : the Job succeeds early once listed indexes (or a count of them) succeed.
  • Failures still come first : the docs' success example can end BackoffLimitExceeded. Raise backoffLimit.
completions 4, backoffLimitPerIndex 1 idx 0 exit 1 exit 1 budget used idx 1 exit 42 FailIndex, no retry idx 2 exit 0 done idx 3 exit 0 done Job Failed (FailedIndexes) failedIndexes "0,1", failed 3

4.Replace, hand off, suspend

  • podReplacementPolicy : TerminatingOrFailed (default) counts a deleting Pod as failed and replaces it after the retry delay (about 10 s). Failed waits until it has failed : one Pod per index at a time. GA v1.34.
  • managedBy : hands the Job to another controller (MultiKueue). If nothing implements it : no Pods, STATUS Running forever. GA v1.35.
  • suspend : running Pods get SIGTERM, counted as neither. Resume resets startTime, so the deadline restarts. Kueue admits Jobs this way.
Edit after create ?
  • Yes : parallelism, backoffLimit, activeDeadlineSeconds, ttlSecondsAfterFinished, podReplacementPolicy, maxFailedIndexes, suspend.
  • No : completionMode, NonIndexed completions, backoffLimitPerIndex, podFailurePolicy, successPolicy, managedBy.
  • Template : scheduling and resources, only while suspended. Never the image : recreate, or use Flux's force annotation.
In short : A Job ends in two steps, failures checked before successes. podFailurePolicy, per-index budgets and successPolicy tune that, and most of the spec is frozen once the Job exists.

1.A CronJob stamps out Jobs

  • At each tick the cronjob-controller creates <cronjob>-<minutes since epoch>. The fixed name blocks a double run.
  • The Job copies jobTemplate, is owned by the CronJob and notes its slot in an annotation.
  • Edits and suspend: true only touch future runs (open issue 128155 : a schedule edit can run a passed minute). Names stop at 52 characters.
CronJob hello schedule + template tick Job hello-29854260 2026-10-06 03:00 UTC Pods history limits keep 3 ok, 1 failed create job --from=cronjob a manual run manual Job not in status.active

2.Schedule syntax

minutehourdaymonthweekday
0-590-231-311-12, jan-dec0-6, sun-sat
302**1-5
  • Above : 02:30, Monday to Friday. Also */15, 1,5, @hourly, @daily, @weekly, @monthly, @yearly.
  • Rejected : 7 as Sunday, a seconds field, @reboot, TZ=.
  • Day and weekday both set means either : 0 0 13 * 5 runs on the 13th and every Friday.
Time zones
  • No timeZone : kube-controller-manager's own zone. On kubeadm that is UTC.
  • Set an IANA name : timeZone: Etc/UTC. Local is rejected.
  • The scheduled-timestamp annotation is in that zone, UTC when unset.
  • DST (library test, New York) : 30 2 * * * skipped on 2026-03-08, 30 1 * * * twice on 2026-11-01. Etc/UTC dodges it.

3.Overlap and missed runs

PolicyLast run still active
Allow (default)a new Job starts anyway
Forbidskipped (JobAlreadyActive), may start late when the blocker ends
Replaceold Jobs deleted, then the new one

A hung run blocks Forbid forever : set activeDeadlineSeconds in the jobTemplate. Manual runs are invisible to Forbid.

Missed runs
  • Catch-up is one Job, for the latest missed slot (downtime, suspend, Forbid).
  • startingDeadlineSeconds : older slots are dropped with no Job and no event (MissSchedule never fires).
  • Over 100 missed : Warning TooManyMissedTimes, and the latest run still starts.
  • Watch lastScheduleTime or kube-state-metrics, not events : they thin out.

4.Cleanup

MechanismCoversDeletes with
ttlSecondsAfterFinishedany finished Job, clock starts at Complete or Failed, 0 = at onceForeground
history limits (3 and 1)finished Jobs the CronJob owns, 0 keeps noneBackground
kubectl delete jobthe Job and its Pods (--cascade=orphan keeps Pods)Background
raw API DELETEthe Job only, its Pods are left behindOrphan

A standalone Job without TTL stays forever. Owners and propagation : Garbage Collection.

5.Debugging, in order

CommandLook for
kubectl describe job piEvents : the reason, FailedCreate (quota, Pod Security, webhook)
kubectl get pods -l batch.kubernetes.io/job-name=pi --field-selector=status.phase=FailedOnly the failed Pods. Then describe pod : exit code, OOMKilled
kubectl logs job/pi --all-pods=trueLogs of every Pod, prefixed
kubectl create job hello-retry --from=cronjob/helloA Failed Job never restarts. Make a new one
kubectl events --for cronjob/helloJobAlreadyActive, UnexpectedJob, TooManyMissedTimes
In short : A CronJob stamps a Job per tick and catches up only the latest missed run. Clean up with TTL or history limits, and debug from the Job's reason down to the exit code.

Core Kubernetes · 12

Nodes

1.What is a Node ?

A Node is a cluster-scoped v1 object that stands for one machine. You rarely write it : the kubelet creates its own Node when it starts, then keeps its status fresh.

  • On the machine : the kubelet, a container runtime and kube-proxy (see Architecture).
  • The name is unique in the cluster. Delete a rebuilt machine's old Node before it joins again.
  • The object is a record, not the machine : deleting it stops nothing.
  • The kubelet signs in as system:node:<name>, group system:nodes. Ready stays False until the runtime answers and the network plugin is up.
machine node-3 kubelet registerNode true by default API server Node node-3 labels, capacity Lease node-3 kube-node-lease status Ready True 1. creates it 2. renews, 10s 3. posts, updates Scheduler can place Pods here Nodes, by watch

Cloud : the cloud-controller-manager adds providerID, zone, region and instance type, and deletes the Node once the instance is gone. Karpenter and Cluster API make nodes on demand.

2.What the object holds

FieldHolds
spec.podCIDRthis node's Pod address range, if the cluster gives one per node
spec.taints, unschedulablewho may land here
status.addressesHostname, InternalIP, ExternalIP, InternalDNS, ExternalDNS
status.conditionsReady and the pressure flags
status.nodeInfokernel, OS image, runtime and kubelet versions
status.imagesup to 50 images on the node

3.Capacity vs allocatable

capacity : all the machine has kubeReserved systemReserved eviction allocatable both empty by default mem 100Mi for Pod requests
  • Stock kubelet : allocatable memory is capacity minus 100Mi, CPU is all of it. Managed services reserve more.
  • The scheduler adds up requests against allocatable, never usage.
  • pods is 110 by default (maxPods). Cloud CNIs often change it.

4.Labels and roles

LabelSet by
kubernetes.io/hostname, /os, /archkubelet
topology.kubernetes.io/zone, /regioncloud, or you
node.kubernetes.io/instance-typecloud
node-role.kubernetes.io/<role>kubeadm, or you

Old guides say node-role.kubernetes.io/master : it is control-plane now.

ROLES is only labels
  • The column lists the <role> part of each node-role label, or <none>. The value is ignored.
  • kubeadm sets control-plane. Any other is yours : kubectl label node n3 node-role.kubernetes.io/worker=
  • A kubelet cannot give itself a role : NodeRestriction forbids it.

5.Reading kubectl describe node

Part, in print orderWhat to read there
Roles, Labels, Taints, Unschedulablewhich Pods may land here, and whether it is cordoned
Lease, Conditionsis it alive : each condition has a reason and a message. A healthy Ready : kubelet is posting ready status
Addresses, Capacity, Allocatable, System Infowhere it is, how big, which kernel, runtime and kubelet
Non-terminated Pods, Allocated resourcesevery Pod here, then the sum of their requests and limits. Not usage : a node can be 95% allocated and idle (for usage, kubectl top node needs metrics-server)
Eventsrecent history, such as RegisteredNode and NodeReady
In short : A Node is the kubelet's own record of its machine. Pods are scheduled against allocatable, which is capacity minus reservations and the eviction threshold.

1.Two heartbeats

HeartbeatHow often
Lease renewTimeevery 10s
Node statuson a change, else every 5 min
  • One Lease per node in kube-node-lease, so a healthy node need not rewrite a large status every 10s.
  • No change for 50s (40s before v1.32) : the node lifecycle controller calls the node late.

2.NotReady means two things

Ready FalseReady Unknown
Written bythe kubeletthe controller
Meanshere, but cannot run Pods (runtime down, no CNI)nobody has heard from the kubelet
Taintnot-readyunreachable
get nodesNotReadyNotReady

Other conditions : MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable. Reason and message : describe node.

3.A node dies : the timeline

0s : last Lease renewal then the kubelet stops : crash, power, network cut 50s grace period : node lifecycle controller about 50s (lab : 52s) : Ready Unknown taints unreachable:NoSchedule and :NoExecute Pods Ready False, yet still listed 1/1 Running each Pod's own 300s toleration : taint-eviction controller about 350s (lab : 353s) : Pods deleted 1/1 Terminating : no kubelet to finish the job a Deployment makes the replacement at once
  • No controller setting moves the Pods. The 300s is tolerationSeconds: 300, added at admission to each Pod without its own. It starts when the taint lands.
  • The containers may still run on the lost machine. Kubelet back : Ready turns True, both taints go by themselves, deleted Pods' containers are killed.
  • Faster failover : give the Pods their own unreachable and not-ready tolerations with a smaller tolerationSeconds. Lab, with 20s : deleted 73s after the last heartbeat.
At 350sDeploymentStatefulSetDaemonSet
its Pod isreplaced at once : a Pod marked for deletion no longer countsstuck Terminating, not replaced : its name is still takennever evicted : it tolerates both taints with no time limit

4.The node protects itself

  • Node-pressure eviction : past a threshold the kubelet sets a condition, the controller adds a NoSchedule taint, and the kubelet evicts Pods. At once, no PodDisruptionBudget asked.
  • Memory : Pods using more than they requested go first, then by priority. No QoS class is safe, only critical Pods are spared.
  • The evicted Pod stays listed : phase Failed, reason Evicted, STATUS often Error.
SignalEvict underCondition
memory.available100MiMemoryPressure
nodefs.available10%DiskPressure
nodefs.inodesFree5%
imagefs.available15%
imagefs.inodesFree5%
pid.availableno defaultPIDPressure

Hard defaults on Linux. Setting one key in evictionHard drops the others. The condition stays True 5 min after the pressure ends.

API eviction (drain)Taint evictionNode-pressureOOM kill
Whoyou, through the API servertaint-eviction controllerkubeletLinux kernel
Budgets (PDB)honouredignoredignorednot involved
Pod afterwardsdeleteddead node : Terminating till it returnsFailed, Evicted, stays listedsame Pod, container restarted

5.Shutdown

  • Graceful node shutdown is on and still does nothing : shutdownGracePeriod and shutdownGracePeriodCriticalPods are both 0. Set them (say 30s and 10s) and the kubelet holds the systemd shutdown to stop ordinary Pods, then critical ones.
Power lost : the out-of-service taint kubectl taint nodes n3 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute gets its Pods deleted, stuck and DaemonSet ones too, and volumes detached at once. Only when the node is really off : if not, old containers run beside the new. Remove the taint yourself.
In short : A node is alive while its Lease is renewed. After 50s of silence it is tainted, and each Pod's own 300s toleration decides when it is deleted. Deployments replace, StatefulSet Pods wait.

1.Cordon, drain, uncordon

  • cordon sets spec.unschedulable. Running Pods stay, DaemonSet Pods still land.
  • drain cordons, then evicts each Pod through the Eviction API, which asks its PodDisruptionBudget. A refusal is retried every 5s, for ever unless --timeout is set.
  • uncordon clears the flag, no Pod moves back. Drain never stops the kubelet or deletes the Node.
1. cordon no new Pods 2. list the Pods static ones skipped 3. Eviction one per Pod refused : nothing is evicted DaemonSet, emptyDir or bare Pod budget allows ? yes Pod deleted own grace period no HTTP 429 try again every 5s

2.What drain refuses

Pod on the nodedrain saysFlag, and what it costs
DaemonSet Podcannot delete DaemonSet-managed Pods--ignore-daemonsets : leaves them running
uses an emptyDircannot delete Pods with local storage--delete-emptydir-data : that data is lost
no controllercannot delete Pods that declare no controller--force : the Pod is gone for good
budget would breakCannot evict pod as it would violate the pod's disruption budget.fix the budget or add replicas (--disable-eviction ignores budgets)

The first three stop the drain before anything is evicted. One replica with minAvailable: 1 blocks it for ever.

3.Labels, nodeSelector, a taint

$ kubectl label node n3 pool=gpu
$ kubectl taint nodes n3 dedicated=gpu:NoSchedule
nodeSelector: {pool: gpu}   # Pod template
tolerations:
- {key: dedicated, value: gpu, effect: NoSchedule}

Label : Pods in. Taint : others out (Advanced Scheduling).

4.Removing a node

  • Drain, kubectl delete node, then kubeadm reset on the machine.
  • delete node removes only the object. Its Pods go about a minute later, no budget asked.
  • A running kubelet does not register again. After a restart it is back in seconds, without hand-added labels, taints or the cordon.

5.Version skew and upgrades

  • A kubelet may be up to three minors older than the API server, never newer. Control plane first.
  • Then node by node : new kubeadm, kubeadm upgrade node, drain, new kubelet and restart, uncordon.
What a kubelet may do Only its own Node and Pods : NodeRestriction and the Node authorizer see to it. But get on nodes/proxy reaches the kubelet API, exec included : treat it as admin (RBAC).

6.Why is it NotReady ?

describe node showsCauseLook at, then fix
Unknown, "Kubelet stopped posting node status."kubelet, machine or network downjournalctl -u kubelet : start it
False, "cni plugin not initialized"no CNI plugin yetls /etc/cni/net.d : install the CNI
False, "container runtime is down"containerd stoppedcrictl info : restart the runtime
False, "PLEG is not healthy"runtime too slow to list containersnode load, disk I/O : restart the runtime

7.Commands

CommandWhat it shows or does
kubectl get lease -n kube-node-leaseOne Lease per node : the heartbeat
kubectl get pods -A -o wide --field-selector spec.nodeName=n3Every Pod on one node
kubectl drain n3 --ignore-daemonsets --delete-emptydir-dataCordon and evict. Then kubectl uncordon n3
kubectl debug node/n3 -it --image=busyboxA Pod with the host root at /host. Delete it after
In short : Cordon blocks new Pods, drain evicts through budgets, and deleting the Node removes only the record. Upgrade the control plane first, then one drained node at a time.

Core Kubernetes · 13

Quick Revision

1.Object map

dashed box = cluster-wide Ingress or Gateway HTTP(S) Service Pod IPs EndpointSlice StorageClass Deployment per template ReplicaSet keeps N Pod scheduled Node selects ConfigMap Secret PVC the claim binds PV the storage uses provisions

apiVersion : v1 for most. apps/v1, discovery.k8s.io/v1, storage.k8s.io/v1, networking.k8s.io/v1. Ingress is GA but frozen, Gateway API is the way forward. Checked for Kubernetes v1.37 (supported : 1.37, 1.36, 1.35).

Ch.ObjectDoes what
02NodeMachine that runs Pods
03PodContainers, one IP, volumes
04DeploymentRolling updates over ReplicaSets
04ReplicaSetKeeps N Pods running
05ServiceStable name + IP for its Pods
05EndpointSliceThe live Pod IPs
06ConfigMapPlain settings, 1 MiB max
06SecretSensitive values, 1 MiB max
07PVC / PVThe claim / the disk
07StorageClassHow new disks get made
08NamespaceName scope, quotas, defaults
08IngressHTTP routes in (frozen)
add-onGateway APINewer routing, as CRDs

2.Triage : STATUS, then where to look

1. get podsSTATUS, RESTARTS
2. describe podEvents, Last State
3. logs --previousthe run that crashed
4. events--for pod/<pod>
5. debugno shell in the image
STATUSWhat it meansRunUsual fixCh.
PendingFailedScheduling : requests fit no node, an untolerated taint, an unbound PVC, a hostPort clashkubectl describe pod <pod>
kubectl get events --field-selector reason=FailedScheduling
Lower requests, add nodes, tolerate, fix the PVC03, 02
ImagePullBackOffBad name or tag, image not pushed, or no imagePullSecretkubectl describe pod <pod>Fix the ref, push it, add a pull Secret03
CrashLoopBackOffKeeps exiting. Waits 10 s, 20 s, 40 s ... up to 300 s between restartskubectl logs <pod> --previousFix the app, config or probe timing03
OOMKilledOver the memory limit, Exit Code 137. RESTARTS climbskubectl describe pod <pod>
kubectl top pod <pod> --containers
Raise the limit, or fix the leak03
Read the exit code
  • 0Exited cleanly
  • 1The app reported an error
  • 137128 + 9, SIGKILL : OOM or killed
  • 143128 + 15, SIGTERM : asked to stop
STATUS is not the phase
  • STATUS shows a container reason when there is one, else status.phase.
  • ImagePullBackOff : phase Pending. CrashLoopBackOff : usually Running.

3.Commands

LookWhat it is for
kubectl get pods -o wideSTATUS, node, Pod IP. 03
kubectl describe pod <pod>Events, Last State. 03
kubectl logs <pod> --previousThe run that crashed. 03
kubectl events --for pod/<pod> --types=WarningIts warnings only, the same ones describe lists. 09
kubectl top pod <pod> --containersLive CPU and memory. Needs Metrics Server
kubectl debug -it <pod> --image=busybox:1.28 --target=<c>No shell in the image ? Add an ephemeral one. 03
ChangeWhat it is for
kubectl apply -f <file>Create or update. 01
kubectl diff -f <file>Preview an apply. 01
kubectl exec -it <pod> -- shShell in a container, via the kubelet. 02
kubectl port-forward svc/<svc> 8080:80localhost:8080 to one Pod behind the Service. 02
kubectl rollout status / undo deploy/<d>Wait for a rollout, or go back one revision. 09
kubectl scale deploy/<d> --replicas=3Set replicas by hand. An HPA overrides it. 04
In short : Each object does one job and points at the next one. When a Pod breaks, read its STATUS first, then work through get, describe, logs --previous and events.

1.Requests and limits

Set per container, for CPU and for memory.

  • requestWhat kube-scheduler reserves. The Pod lands only where the sum of requests fits, even if real use is low.
  • limitA ceiling the kernel enforces with cgroups.
  • Going above the request is fine while the node has room.
  • Limit only ? The API server copies it into the request, so a Pod with only CPU and memory limits is still Guaranteed.
  • LimitRange : namespace defaults, min and max, applied at admission. Break them, get 403 Forbidden.

See : scheduler 02, HPA % of requests 04, LimitRange and ResourceQuota 08.

CPU use above limit Memory use above limit Node pressure + use above request Throttled waits each time slice OOMKilled Exit Code 137 Evicted by the kubelet Never killed for CPU use Restart per restartPolicy Pod Failed all containers stop again ? CPU request = a weight when CPU is contended CrashLoopBackOff 10 s up to 300 s New Pod from its controller usually on another node Limits cause throttling and OOM kills. Requests (then Priority) decide who is evicted.
resources:
  requests: { cpu: "250m", memory: "128Mi" }
  limits:   { cpu: "500m", memory: "256Mi" }  # 0.5 CPU

2.QoS classes

Kubernetes derives status.qosClass from the resources. You never set it, and it is fixed once the Pod is created.

Pod spec all containers, or Pod-level resources Every container : CPU + memory, request = limit, both above 0 ? yes Guaranteed score -997 no Any CPU or memory request or limit set anywhere ? yes Burstable score 3 to 999 no BestEffort score 1000 oom_score_adj : the highest is killed first
  • Pod-level spec.resources (Beta since v1.34, on by default) count too.
ClassRuleUnder node pressure
GuaranteedEvery container sets CPU and memory, request = limitEvicted last, never over its requests
BurstableSome CPU or memory request or limit, not GuaranteedEarly if over its requests, late if under
BestEffortNo CPU or memory request or limit anywhereEarly, no request to stay under
How the kubelet picks
  1. Using more than its requests ? Goes first.
  2. Then lower Priority first.
  3. Then the furthest over its requests.
QoS only predicts this order. It does not count under DiskPressure. More on the kubelet in Architecture.

3.Labels vs annotations

LabelsAnnotationsCh.
PurposeIdentify and group objectsDescribe objects for tools and people04, 09
Selected byServices, workloads, kubectl -lNever05, 04
SelectorsA Service matches on equality only. Deployment, ReplicaSet, Job and DaemonSet also take In, NotIn, Exists, DoesNotExist. A Deployment's selector is immutable and must match its Pod template labels.05, 04
Key[prefix/]name, name 63 chars max. kubernetes.io/ and k8s.io/ are reserved. Same rules for both.04
Value63 chars max, may be emptyAny string, 256 KiB in total per object04
Exampleapp: webkubernetes.io/change-cause: "to 1.16.1"04, 09
RecommendedPrefix app.kubernetes.io/ plus nameinstanceversioncomponentpart-ofmanaged-bynone
In short : Requests place Pods and decide eviction, limits cap them (CPU is throttled, memory is OOMKilled). Labels select, annotations only describe.

Part 2Beyond Core

Core Kubernetes gives you Pods, Services and a control loop. These 29 topics cover the add-ons that plug into that loop through CRDs and controllers, the CI that builds images, and the signals that show what it is doing.

Core Kubernetes
CRDs + Controllers
A real platform
Deliver
Route
Secure
Observe
Certificates and secrets
Cloud infrastructure
Workload identity
How to read
  1. Why it exists
  2. Components
  3. Diagram
  4. Commands
  5. In short
Why Part 2 ?
  • Each add-on solves one gap in core
  • Most are CRDs plus a controller
  • Install order matters (CRDs first)
  • CI and image builds run in Pods too
  • Flux or Argo CD can deliver every add-on
  • Signals: metrics, logs, traces, profiles
“Core runs the Pods. Add-ons run the platform.”

GitOps and delivery

Flux CD v2

1.What is Flux ?

Flux watches a Git repo (or OCI image or bucket) and keeps the cluster matching it, forever, by itself.

You push to Git, Flux applies it. No kubectl apply from laptops.

  • CNCF Graduated (Nov 2022)
  • Latest : Flux v2.9.6
  • Helm, Kustomize and plain YAML all work
  • Runs in the flux-system Namespace
vs kubectl applykubectl is a one-time push from your laptop. Flux is a loop that never stops and fixes drift.

Sources say where (Git, OCI, Helm). Appliers say what (Kustomization, HelmRelease).

Developer git push Git repo cluster (flux-system) poll source-controller kustomize-ctrl helm-ctrl Kubernetes API Pods / Deployments / Services
DirectionArrows run from inside the cluster outward. Nothing pushes in.

2.Why GitOps ?

GitOps = Git is the single source of truth, declarative, pulled automatically, continuously reconciled.

  • Pull model : CI needs no cluster credentials
  • Drift : live edits get reverted by Flux
  • Rollback : just git revert
  • Many clusters : one repo, different paths
Key Points
  • Reconcile runs every spec.interval
  • Also runs on a webhook (Receiver)
  • prune: true : delete in Git, deleted in cluster
  • Per-Kustomization serviceAccountName limits a team
  • kubectl get fluxcd -A shows all
  • Flagger, a sister project, adds canary and blue/green

3.The Controllers (Toolkit)

Small controllers, each owns its CRDs. Composable like Lego.

flux-system Namespace
source-controllerFetches Git, OCI, Helm repo, Bucket. Serves tarballs.
kustomize-controllerBuilds and applies with server-side apply, prune, health checks, SOPS.
helm-controllerHelm install, upgrade, rollback, uninstall from HelmRelease.
notification-controllerEvents out (Slack, Git status). Webhooks in.
image-reflectorScans registries for tags. Picks the latest by rule.
image-automationCommits new image tags back to Git.

Dashed ones are optional.

Each one owns its own CRDs and talks only through the API.
In short : Flux pulls from Git, builds with small controllers, applies, and keeps fixing drift every interval.

1.The Reconcile Loop

Drift gets undone at the next loop. A failure keeps the last good state and retries.

  1. Fetch source
  2. Build (kustomize / helm template)
  3. Diff vs live (dry-run SSA)
  4. Apply, only if drift or new revision
  5. Health check, set status Ready
  6. Wait interval, loop
Fetch Build Diff drift ? Apply Health Ready wait interval,then again

2.Getting Started

  1. flux bootstrap installs controllers in flux-system
  2. Define a source (GitRepository, OCIRepository, HelmRepository)
  3. Define what to apply (Kustomization or HelmRelease)
  4. Push to Git
  5. Flux fetches, applies, runs health checks
  6. Drift and deletions fixed at next interval
  7. notification-controller tells Slack or Git
Tip & Check
  • Use flux create source git ... --export to make YAML you commit.
  • After bootstrap, flux check should show every controller Ready. flux get all -A lists the rest.
flux CLI Git repo push install controllers in flux-system pull
GitRepository + Kustomization
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata: {name: app, namespace: flux-system}
spec:
  interval: 1m
  url: https://github.com/example/app
  ref: {branch: main}
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: {name: app, namespace: flux-system}
spec:
  interval: 10m
  path: ./deploy
  prune: true
  wait: true
  timeout: 3m
  sourceRef: {kind: GitRepository, name: app}

3.CRD Cheat Sheet

KindapiVersionControllerWhat it does
GitRepositorysource.toolkit.fluxcd.io/v1sourceGit branch, tag, semver or commit
OCIRepositorysource.toolkit.fluxcd.io/v1sourceOCI artifact as source
HelmRepositorysource.toolkit.fluxcd.io/v1sourceHelm chart index (http or oci)
Bucketsource.toolkit.fluxcd.io/v1sourceS3-compatible bucket
Kustomizationkustomize.toolkit.fluxcd.io/v1kustomizeApply a path from a source
HelmReleasehelm.toolkit.fluxcd.io/v2helmDeclares a Helm release
Provider / Alertnotification.toolkit.fluxcd.io/v1notificationWhere and which events to send
ImagePolicyimage.toolkit.fluxcd.io/v1image-reflectorChoose latest tag

Check versions with kubectl api-resources | grep toolkit

In short : Each interval Flux fetches, builds, diffs and applies; sources say where, Kustomization and HelmRelease say what.

1.Image Automation

Flux can write new image tags back to Git. Then the normal pull flow deploys them.

NoteOptional controllers. Add them with:
flux bootstrap ... --components-extra
Image Automation Loop Registry ImageRepository ImagePolicy scan picks 5.1.x ImageUpdateAutomation commit Git repo normal pull flow deploys it GitRepository / Kustomization Registry scan ImageRepository ImagePolicy picks 5.1.x ImageUpdateAutomation commit Git repo normal pull flow GitRepository / Kustomization

2.flux Commands

CommandDescriptionExample
flux check --preCheck the cluster meets requirements (K8s v1.34.1+)flux check
flux bootstrap githubInstall controllers, commit to Git, self-manageflux bootstrap github --owner=loren --repo=fleet --path=clusters/prod
flux get all -AStatus of every Flux objectflux get kustomizations
flux reconcile kustomization <name>Sync nowflux reconcile kustomization flux-system --with-source
flux logs --level=error -AController logsflux logs -f --kind=Kustomization --name=apps
flux trace deployment/<name>Who owns this objectflux trace deployment/podinfo -n default
flux suspend / resume kustomizationPause for a live hotfixflux suspend kustomization apps
flux diff kustomization <name>Preview changes vs liveflux diff kustomization apps --path=./apps

3.Flux vs Argo CD vs CI Push

TopicFluxArgo CDCI
ModelPullPullPush
UIOperatorBuilt inNone
Drift fixAutoIf selfHealNo
Creds in CINoNoYes
ConfigCRDsApplicationScripts
Gotchas
  • Two things named Kustomization (Flux vs kustomize.config.k8s.io)
  • v1beta2 / v2beta2 removed in v2.8, run flux migrate
  • Hotfix ? flux suspend first, or the next loop reverts it
  • prune: true deletes what you drop from Git
  • Encrypt secrets (SOPS or External Secrets)
  • dependsOn needs wait to really wait
  • Interval minimum is 60 seconds
  • Stuck Ready=False ? Read the message
  • Repo layout, health, drift, multi-tenancy : Flux deep dive
In short : Image automation commits new tags to Git, the flux CLI checks, syncs, traces and pauses, and Flux is the pull-only, CRD-only cousin of Argo CD.

1.Flux Operator in brief

  • An operator from ControlPlane that installs and upgrades Flux from one FluxInstance named flux, instead of flux bootstrap.
  • distribution.version: "2.9.x" floats patches only. Pin the minor on purpose.
  • A FluxReport holds the observed state. AGPL-3.0, pre-GA (0.x). More : Flux deep dive.

2.The Flux Web UI

In the operator since v0.37.0, on by default, plain HTTP on port 9080 (Service flux-operator, leader pod only).

  • ShowsFlux objects, workloads, logs, events, graph, history
  • DoesReconcile, suspend, resume, download, restart, run job, delete pod
  • NeverEdits manifests : Git is the way in

3.Login modes and roles

ModeRuns asActions
NoneOperator SA (cluster-admin)Off
AnonymousOne shared identityBy RBAC
OAuth2 (OIDC)Each user, via CELBy RBAC
  • flux-web-userget, list, watch on */* : Secrets too
  • flux-web-adminSame, plus reconcile, suspend, resume, download, restart
Both roles read SecretsViewers mask them, but with login on the HelmRelease view shows valuesFrom Secrets in clear. RBAC has no "except" : list core minus secrets plus every other group, and keep get on ResourceSets.

4.How a request is authorized

Browser Gateway TLS, HTTPRoute Web UI operator :9080 HTTPS HTTP 1. 303 to the IdP IdP login code + PKCE 2. code to /oauth2/callback 3. CEL on claims to username, groups 4. as the operator SA + Impersonate-User, -Group API server authn the SA may impersonate ? RBAC as user allow : data deny : 403 5. RBAC decides every read and every button
  1. No session : 401, then /oauth2/authorize answers 303, Location on the IdP (code + PKCE).
  2. Code to /oauth2/callback, ID token verified.
  3. CEL validations pass; claims become user and groups (both empty : refused).
  4. Calls go as the operator SA, with impersonation.
  5. RBAC answers for the user. Buttons first run a SelfSubjectAccessReview. Impersonated also needs get + patch, FineGrained (v0.51.0) only the verb.

A removed group keeps access until the ID token expires. Keycloak : Full group path off, so groups read team, not /team, as in RBAC. See OIDC auth; Argo CD has its own policy.

5.Exposing it safely

  • No auth : port-forward only. Else its own host : HTTPRoute to flux-operator:9080, TLS at the Gateway.
  • FluxInstance cluster.networkPolicy: true (default) admits only flux-system pods. install.yaml has no web policy, so admit the proxy pods to 9080 (NetworkPolicy). The chart's policy admits all namespaces.
  • Config holds clientSecret : a SOPS Secret, hot-reloaded.
Seen in practicePitfall : narrowed roles drift : upgrades re-add the wildcard read, new CRD groups go missing. Do better : a scheduled SubjectAccessReview check that readers list every group, but no Secrets.

6.Commands

CommandDoes
kubectl -n flux-system port-forward svc/flux-operator 9080:9080Open the UI on localhost:9080, nothing exposed
flux-operator create secret web-config ui --base-url=... --issuer-url=... --client-id=... --client-secret-stdin --exportConfig Secret as YAML (OIDC by default). Encrypt it
kubectl auth can-i list secrets --as=jane --as-group=viewersWhat a UI user may do (should be no)
In short : The Web UI lives in Flux Operator on port 9080. Turn on OIDC, let impersonation hand every click to RBAC, and narrow the stock roles : both read Secrets.

GitOps and delivery

Argo CD

1.What is Argo CD ?

A GitOps continuous delivery tool for Kubernetes. The desired state lives in Git (or a Helm or OCI repo). A controller keeps comparing it with the live cluster, and can sync the cluster back to it.

  • Pull model : Argo CD pulls from Git, so CI needs no cluster credentials.
  • Renders Kustomize, Helm, Jsonnet, plain YAML, or a plugin.
  • Extras : web UI, CLI, SSO and its own RBAC on top of Kubernetes.
  • Lives in argocd, drives one or many clusters.

2.Words to know

WordMeaning
Target stateWhat Git says should run
Live stateWhat really runs in the cluster
Sync statusSynced, OutOfSync or Unknown
SyncMake live match target (apply)
RefreshRe-read Git and diff, no apply. Hard refresh also drops the cache
HealthIs it running and serving ? A separate answer : Synced can still be Degraded

Argo project, CNCF graduated 2022-12-06. Latest v3.5.4 (2026-10-06), tested on Kubernetes v1.33 to v1.36. The three newest minors get patches. 3.6 is a release candidate.

3.The parts

A few workloads; Argo's own share one image (quay.io/argoproj/argocd). Real state lives in CRs, ConfigMaps and Secrets.

  • repo-server clones and renders. No Kubernetes rights, no stored credentials.
  • application-controller is a StatefulSet and a classic controller. Scale it to shard clusters.
  • notifications-controller sends alerts. dex is optional with direct OIDC. redis is safe to lose.
YouUI, CLI, CI ApplicationSetcontroller Git repoor Helm, OCI login, sync creates argocd-serverAPI, SSO, RBAC Applicationdesired + status writes login watch pulls,caches 24h dexSSO broker app controllerdiff, sync, health repo-serverrenders YAML asks watch, apply rediscache only Cluster(s)in-cluster + registered remote clusters

4.Detection vs correction

Argo CD always notices drift. Fixing it is opt-in, one switch at a time.

  1. Every 120s (plus up to 60s jitter), or at once on a Git webhook, the controller refreshes each app.
  2. repo-server renders the manifests, the controller diffs them with the live objects, and marks the app Synced or OutOfSync. That is detection.
  3. Correction is a sync, and it is manual by default : press Sync, or run argocd app sync.
syncPolicyNew commit in GitFile removed from GitSomeone runs kubectl edit
none (default)OutOfSync, waits for youOutOfSync, waits for youOutOfSync, waits for you
automatedSynced by itselfKeeps running, stays OutOfSyncStays OutOfSync, not reverted
+ prune: trueSynced by itselfDeleted from the clusterStays OutOfSync, not reverted
+ selfHeal: trueSynced by itselfDeleted only if prune is on tooReverted to Git

OpenGitOps asks for "continuously reconciled". Out of the box Argo CD only observes continuously. Flux fixes drift on every interval.

5.Turning it on

spec:
  syncPolicy:
    automated:
      enabled: true   # 3.1+, unset = on
      prune: true     # default false
      selfHeal: true  # default false

CLI : argocd app set APP --sync-policy automated --auto-prune --self-heal. enabled: false pauses auto-sync but keeps the flags.

Fine print
  • Auto-sync tries once per commit SHA (plus parameters). A SHA that failed is not tried again.
  • Self-heal waits a backoff : 2s, times 3, max 300s. Docs saying "5 seconds" are stale.
  • No retry block : auto-sync retries 5 times (5s doubling, max 3m). A retry block without limit : 0 retries.
  • No rollback while auto-sync is on.
In short : repo-server renders Git, the controller diffs it with the cluster every few minutes and flags OutOfSync, and nothing is fixed until you sync or turn on automated, prune and selfHeal.

1.The three CRDs

All are argoproj.io/v1alpha1 only (no beta, no v1), namespaced, and live in argocd by default. See CRDs.

Kind (short)JobKey fields
Application (app)Source(s) to one destinationproject, source, destination, syncPolicy
ApplicationSet (appset)Template that stamps out many Applicationsgenerators, template, strategy
AppProject (appproj)Guard rails : what may go wheresourceRepos, destinations, roles, syncWindows

2.An Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
  finalizers: [resources-finalizer.argocd.argoproj.io]
spec:
  project: default
  source:
    repoURL: https://github.com/my-org/apps.git
    path: guestbook   # targetRevision unset = HEAD
  destination:
    server: https://kubernetes.default.svc
    namespace: guestbook
  • targetRevision : a branch, a tag, a semver range (matches tags only) or a commit SHA.
  • sources : a list instead of source, e.g. a Helm chart plus values from your repo.
  • destination : server URL or cluster name, never both.
  • finalizer : without it, deleting the Application leaves every resource running. See finalizers.
  • project : required. The default project allows anything. A project that may deploy into argocd is as good as admin.

3.Sync phases and waves

  1. Order : phase, wave (default 0, negatives allowed), kind (Namespaces first, custom resources last), name.
  2. Apply the first wave that is not yet synced and healthy.
  3. Wait for Healthy, pause 2s, next wave. Prunes go in reverse wave order.
PreSynchooks Sync phase wave-1 wave0 wave1 each : apply, wait Healthy, 2s prunes in reverse wave order PostSynchooks any failure SyncFail hooks

4.Hooks

kind: Job   # batch/v1
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
    argocd.argoproj.io/sync-wave: "-1"
spec: ...    # a normal Job
  • Kinds : PreSync, Sync, PostSync, SyncFail, Skip. On app deletion : PreDelete (3.3+), PostDelete.
  • No delete policy means BeforeHookCreation. Hooks skip a selective sync.
  • Helm : Argo CD runs helm template, so helm ls shows nothing. pre-install / pre-upgrade become PreSync, post-install / post-upgrade become PostSync.

5.Health

KindHealthy when
Workloadsupdated replicas = desired (rollouts)
LB Service, IngressLB hostname or IP in status
PVCphase is Bound
Job, CronJobComplete. A failed Job, or a CronJob whose last run failed (3.2+), is Degraded
Custom resources238 bundled Lua checks, or yours
Applicationno check (since 1.8)
  • App health = worst direct child. Best to worst : Healthy, Suspended, Progressing, Missing, Degraded, Unknown.
  • Own check : Lua in argocd-cm, key resource.customizations.health.<group>_<kind>.
  • App of apps : add one for argoproj.io_Application, or waves never wait for a child app.
In short : An Application maps its source(s) to one destination inside a project's guard rails, and a sync runs PreSync, then each wave until Healthy, then PostSync.

1.ApplicationSets

Generators make parameter sets. The template is rendered once per set into an Application.

GeneratorParams fromRe-reads
listfixed valuesnever
clustersclusters, by labelSecret events
gitdirs or files in a repoevery 3 min
matrix, mergechild generators combinedfastest child
pullRequestopen PRsevery 30 min

Also scmProvider, clusterDecisionResource, plugin. 3.6 (RC) adds oci. RollingSync (Beta) syncs the apps step by step and turns their auto-sync off.

git generatordirs apps/a, apps/b clusters generatorlabel env: prod matrixevery pair params : path, server templateone Application per pair create, update, delete a on c1 a on c2 b on c1 b on c2 2 dirs x 2 clusters = 4 Applications
  • Bootstrap : since 3.4 the docs recommend it, with the cluster generator, over app of apps (Cluster API).
  • Admin only : it can create Applications in any project. Put a template variable in project only if admins own every generator source.

2.Install : server-side since 3.3

$ kubectl create namespace argocd
$ kubectl apply -n argocd --server-side --force-conflicts \
    -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.4/manifests/install.yaml
Why server-side ?Client-side apply keeps a copy of the object in an annotation. The ApplicationSet CRD (about 343 KB as JSON) is over the 262,144-byte limit, so plain kubectl apply fails. See SSA.
ManifestFor
install.yamltrying it, not production
ha/install.yamlproduction, 3+ nodes
namespace-installone namespace, no CRDs
core-install.yamlno API server, UI or SSO
  • --force-conflicts overwrites live edits. Customize with Kustomize patches.
  • Self-managed Argo CD needs ServerSideApply=true.

3.Commands

CommandDoes
argocd admin initial-password -n argocdFirst admin password. Change it, then delete its Secret
kubectl port-forward svc/argocd-server -n argocd 8080:443Reach the API unexposed, then argocd login localhost:8080
argocd app get my-app --hard-refreshStatus. Hard refresh also drops the manifest cache
argocd app diff my-appGit vs live. Exit 0 same, 1 different, 2 error
argocd app sync my-app --prune --dry-runPreview a sync that also deletes extras. Drop --dry-run to run it
argocd app history my-app, argocd app rollback my-app IDRoll back to a past sync. Not while auto-sync is on
argocd appset generate appset.yamlShow the Applications an ApplicationSet would make
argocd cluster add CONTEXTRegister a cluster (adds an admin-level argocd-manager SA)

4.Argo CD vs Flux

TopicArgo CDFlux
UIBuilt inIn Flux Operator
UsersOwn SSO + RBACKubernetes RBAC
ApplyClient-side, SSA opt-inServer-side apply
DriftSeen always, fixed if selfHealFixed every interval
Helmhelm template, no releaseReal Helm releases

Both CNCF graduated in 2022. New image tags into Git : Argo CD Image Updater, a separate argoproj-labs controller.

3.0 tightened RBAC defaults. 3.5 renders with Helm 4 only. 3.6 (RC) diffs ServerSideApply apps server-side.

5.Gotchas

  • ignoreDifferences only hides the diff. Sync still writes those fields unless RespectIgnoreDifferences=true.
  • An HPA or KEDA owns replicas ? Leave replicas out of Git.
  • Secrets injected while rendering are cached in plain text in Redis. Keep them in the cluster with External Secrets.
  • Since 3.0 Argo CD tracks resources with an annotation, not a label. Sync every app once after upgrading.
In short : ApplicationSets stamp out Applications per cluster or folder, install and upgrade go server-side since 3.3, and Argo CD is the UI-and-SSO cousin of Flux.

GitOps and delivery

Argo CD Image Updater

1.What is Image Updater ?

A controller from argoproj-labs. It watches container registries and bumps the image tag (or digest) of Argo CD Applications. Argo CD then syncs it. Flux users have image automation instead.

  • Apps built with Helm, Kustomize or a Plugin (v1.3.0+). Plain YAML and Jsonnet apps are skipped.
  • It never touches the Deployment, only the Application source. Argo CD renders the rest.
  • Runs in the cluster that holds the Applications (the hub), not in workload clusters.
v0.x (legacy)v1.x (now)
ConfigAnnotations on each appAn ImageUpdater CR
APIK8s, or Argo CD API + tokenKubernetes API only
Last releasev0.18.0 (2025-12-25)v1.3.0 (2026-08-13)

Bridge : useAnnotations: true on a ref (v1.1.0+) reads the old annotations, minus PR mode, plugin targets and cosign checks.

Alpha, and says soLabs repo, API v1alpha1, and the README says not yet for critical production. One replica, polls every 2m.

2.How one check works

  1. Trigger : the 2m timer, a CR spec change, or a registry webhook. Editing an Application does not.
  2. Match : Applications in the CR's namespace that fit namePattern and labelSelectors.
  3. Live ? : the image, registry included, must be in the app's status.summary.images, else skipped (unless forceUpdate).
  4. Pick : list tags, filter, the strategy chooses.
  5. Write : to the Application spec or to Git, plus an ImagesUpdated Event.
  6. Sync : auto-sync rolls it out. Without it the app just shows OutOfSync.
Registry Docker Hub, GHCR ImageUpdater CR, v1alpha1 list tags, 2m webhook (optional) reads Image Updater controller, 1 replica argocd (default) git or PR Application spec.source params Git repo commit or PR spec changed new commit Argo CD renders and syncs sync Deployment new image

3.The ImageUpdater CR

apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata: {name: nginx-patches, namespace: argocd}
spec:
  applicationRefs:
  - namePattern: "web-*"        # glob
    labelSelectors: {matchLabels: {team: web}}
    images:
    - alias: nginx
      imageName: nginx:1.26.x   # a constraint
      commonUpdateSettings: {updateStrategy: semver}
      manifestTargets:
        helm: {name: image.repository, tag: image.tag}
# no writeBackConfig = method argocd

4.Key fields

  • applicationRefs : which apps. namePattern (glob, required), optional labelSelectors.
  • images : an alias (no "/") and an imageName, the repo plus a version constraint.
  • manifestTargets : one of helm, kustomize, plugin. Helm defaults to image.name / image.tag, most charts want image.repository.
  • commonUpdateSettings : strategy, tag filters, forceUpdate, pullSecret, platforms. At spec, ref or image level, the most specific field wins.
  • writeBackConfig : argocd or git, at spec or ref level, never per image.
  • imagesVerification : cosign public key check (v1.3.0+).

5.Matching rules

  • A CR only sees Applications in its own namespace. spec.namespace was removed in v1.2.0, using it fails.
  • Inside one CR the most specific ref wins : an exact name beats a glob, labels add weight.
  • One CR per Application. Two CRs on one app overwrite each other every cycle.
In short : One namespaced ImageUpdater CR picks Applications and images, the alpha controller polls registries every 2m, and it writes the newest allowed tag back for Argo CD to sync.

1.The tag is a constraint

For semver, the tag in imageName is a version range (Masterminds semver), not the version you run today.

TagMeansPicks
nginx:1.26.3Exact pinNever moves
nginx:1.26~1.26, so >=1.26.0 <1.27.0Newest 1.26.z
nginx:1.26.xAny 1.26 patch1.26.3 (5 of 1209 fit)
nginx:1.xAny 1.y.zNewest 1.y.z
nginx* : no pre-releases (v1.3.0+)Newest, even a new major
app:2.x-02.x with pre-releasesrc and beta too

2.The pin trap

  • Writing today's version pins it forever, with no error.
  • Two doc examples do it : nginx:1.17.10 (PR mode) and backend:1.0.0 (cosign sample).
  • 1.27.5-alpine counts as a pre-release, so a plain range skips it. Tags that are not semver are skipped too.
  • Before v1.3.0 an empty tag could jump to -rc tags.

3.Filters first, then the strategy

All tagsfrom the registry
allowTagskeeps matches
ignoreTagsdrops glob hits
Strategysorts what is left
Last one winsif not already live

4.Tag filters

  • allowTagsOne string, regexp:<expr> or any. Empty allows all, a bare regex matches nothing. Unanchored : add ^ and $.
  • ignoreTagsAn array of globs, a plain string is rejected. ["*"] pauses the image.
  • platformsAn array. Default is the controller's own OS / arch, so an amd64 controller with arm64 nodes needs it (newest-build, digest).
images:
- alias: myimage
  imageName: some/image        # no range here
  commonUpdateSettings:
    updateStrategy: newest-build
    allowTags: "regexp:^[0-9a-f]{7}$"
    ignoreTags: ["latest", "master"]
    platforms: ["linux/amd64", "linux/arm64"]

5.Four update strategies

StrategyOld namePicksTag meansRegistry work
semver (default)-Highest version in the rangeThe rangeTag list, cached
newest-buildlatestNewest image creation dateNothing, use allowTagsA manifest per tag
alphabeticalnameLast tag in byte orderNothing, use allowTagsTag list, cached
digest-Digest of one mutable tag, writes tag@sha256:The tag to watchA manifest every cycle
  • Old names still work, with a warning. A typo like bogus passes the API server and runs semver.
  • calver exists on master only.
  • newest-build uses the build date, not the push date. On a tie lexical order decides, so latest can win.
  • alphabetical : 2026-9-01 sorts above 2026-10-03.

6.Write-back : argocd vs git

argocd (default)gitgit + PR (v1.2.0+)
WritesApplication spec, via the K8s APIA commit on the branchA new branch plus a GitHub PR or GitLab MR
ChangesHelm parameters, Kustomize images, plugin env.argocd-source-<app>.yaml, kustomization or valuesSame files, after review
CredsNone, RBAC on ApplicationsArgo CD repo creds or git:secret:<name>HTTPS token or GitHub App, no SSH
Lasts ?No : a re-created app, app-of-apps or ApplicationSet can revert itYes, it is in GitYes, once merged
In short : The tag is a range, filters run before the strategy, and argocd write-back is quick but fragile while git keeps the bump.

1.Git and pull request write-back

Clonethe app's repo
Write targetone file per app
Commitskipped if no change
Push or PRbranch, PR or MR
Argo CD syncsa tracked branch
spec:
  writeBackConfig:
    method: git:secret:git-creds  # Secret here
    gitConfig:
      branch: main                # base only
      writeBackTarget: helmvalues:/helm/values.yaml
      pullRequest: {github: {}}   # or gitlab
  • (no target)<path>/.argocd-source-<app>.yaml, merged by Argo CD at render time.
  • kustomizationEdits the images in kustomization.yaml. Takes a directory, not a file.
  • helmvaluesEdits a values file. Bare means <path>/values.yaml.

Plain method: git reuses Argo CD's repo creds. A git:secret: Secret must sit in the CR's namespace.

  • Argo CD deploys only the branch the app tracks. Tracking HEAD, a tag or a SHA ? Set gitConfig.branch. A main:<tpl> write branch is force-pushed, PR mode refuses it.

2.Alpha : sharp edges in v1.3.0

  • Registry auth (issue 1797) : if /v2/ answers without auth but repos need it (zot), creds are never sent. Every cycle gets 401. The fix (PR 1801) is on master only.
  • Stored defaults : a ref or image commonUpdateSettings block is saved with updateStrategy: semver. Repeat your strategy in every block.
  • READY lies : Ready=True can sit next to Error=True.
  • git:secret: SSH or password creds skip host-key and TLS checks.

3.What changed when

VersionWhat changed
v1.0.0CR replaces annotations
v1.2.0spec.namespace gone, PR mode, status
v1.3.0Plugin apps, cosign, Secrets in the CR's namespace, PR branch renamed
masterAuth fix, no stored defaults, calver

No Argo CD or Kubernetes version matrix is published. CHANGELOG.md is stale, read the GitHub release notes.

4.Image Updater vs Flux vs Renovate

Image UpdaterFlux image automationRenovate
RunsIn the cluster, beside Argo CDIn the cluster, two controllersIn CI, outside the cluster
ConfigOne CR, v1alpha1Three CRs, v1 since Flux v2.7.0Config file, argocd manager opt-in
Finds imagesApp params, kustomizationSetter comments in any YAMLManifests in Git
WritesApp spec (no Git), commit or PRAlways a Git commitPull requests
Rule of thumbArgo CD shopFlux shop, plain YAMLReview every bump

5.Commands

CommandWhat it does
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/config/install.yamlInstall CRD, RBAC and controller next to Argo CD (or the argo/argocd-image-updater chart)
kubectl apply --dry-run=server -f iu.yamlCheck a CR against the CRD rules, catches a string ignoreTags
kubectl get imageupdater [<name> -o yaml] -n argocdAPPS, IMAGES, LAST CHECKED, READY. The YAML shows the stored updateStrategy and the Error condition
kubectl -n argocd logs deploy/argocd-image-updater-controllerLogs. Set log.level: debug in the ConfigMap, then restart
argocd-image-updater test nginx --semver-constraint 1.27Try a range, writes nothing : nginx:1.27.5. Needs a kubeconfig
kubectl patch imageupdater <name> -n argocd --type merge -p '{"metadata":{"finalizers":null}}'Drop the finalizer of a CR stuck Terminating after the controller went first
In short : Use git write-back for bumps that must last, read the Error condition and not just READY, and mind the v1.3.0 registry-auth bug.

CI and builds

Actions Runner Controller

1.What is ARC ?

  • GitHub's Kubernetes operator for self-hosted Actions runners, and the reference implementation of its scale set APIs. This sheet is ARC 0.15.0 (gha-runner-scale-set charts), pre-1.0, CRDs v1alpha1.
  • Two Helm charts published only as OCI packages, plus one controller image with no latest tag : always pass --version. Runner Pods are ephemeral : one job, then deleted.
  • Scale set mode is the only one GitHub supports. Legacy ARC (actions.summerwind.dev) is community-only, with no controller release since v0.27.6. Moving over is a rebuild, not an upgrade.

2.The pieces

KindOne perDoes
AutoscalingRunnerSetscale set (Helm release)Wanted state : URL, Secret, min and max, Pod template, mode
AutoscalingListenerscale set, controller namespaceThe listener Pod, its ServiceAccount and config Secret. Role and RoleBinding sit in the runner namespace
EphemeralRunnerSetscale set, same nameThe replica count the listener patches
EphemeralRunnerrunnerOne bare Pod, its JIT config, its phase

All four are namespaced, actions.github.com/v1alpha1. Runner Pods are not Kubernetes Jobs (Jobs).

3.From queued job to runner Pod

GitHub Actions service jobs for the scale set 1. long poll (about 50s) 2. assigned jobs = N listener Pod controller namespace 3. patch replicas EphemeralRunnerSet replicas = N + minRunners 4. JIT config EphemeralRunner + Pod bare Pod, deleted after the job 5. register, run ONE job
  1. The listener long-polls over HTTPS : back at once if a message waits, else after about 50s.
  2. The message carries the assigned jobs count.
  3. The listener patches replicas with its own ServiceAccount.
  4. The controller asks for a JIT config per runner. The Pod gets only ACTIONS_RUNNER_INPUT_JITCONFIG, ARC never injects the PAT or App key.
  5. The runner registers with it, runs one job, then the Pod is deleted at once (0.15.0). Runners are created up to 8 at a time.

A job no runner takes is requeued up to 3 times, and GitHub gives up after 24 hours. No webhook, no REST polling budget.

4.The scaling rule

desired = min(minRunners + jobs, maxRunners)

Example, min 2 and max 10 : 5 assigned jobs give 7 runners, 12 give 10. No schedule is built in : patch minRunners from a CronJob (Jobs).

minRunners is idle runners kept warm, not a floor on total Pods. Unset maxRunners means no cap. Both at 0 is a drain : no runners at all.

5.Targeting a scale set

  • runs-on: <scale set name> : the Helm release name or runnerScaleSetName, at most 45 characters.
  • Extra labels (0.14.0+) use the chart key scaleSetLabels, not the docs' runnerScaleSetLabels, and reach GitHub only at creation.
  • self-hosted is never added for you.
jobs:
  build:
    runs-on: example-set
In short : ARC keeps one ephemeral runner Pod per assigned GitHub job, plus the warm ones you ask for. A listener per scale set is told the job count, so target the set by its name.

1.Install

CommandDoes
helm install arc -n arc-systems --create-namespace --version 0.15.0 oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controllerController and CRDs, own namespace
helm install arc-runner-set -n arc-runners --create-namespace --version 0.15.0 -f values.yaml oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-setOne scale set, another namespace. Name = runs-on target
kubectl create secret generic gh-app -n arc-runners --from-literal=github_app_id=... --from-literal=github_app_installation_id=... --from-file=github_app_private_key=key.pemApp Secret for githubConfigSecret

2.Auth to GitHub

CredentialLevelNeeds
GitHub App (recommended)repo or orgrepo : Administration read and write, Metadata read. org : Self-hosted runners read and write
Classic PATrepo, org, enterpriserepo, admin:org, manage_runners:enterprise. The only option for enterprise sets
Fine-grained PATrepo or orgrepo : Administration read and write. org : Administration read, Self-hosted runners read and write
  • Create the Secret yourself in the scale set namespace and reference it by name. IDs are quoted strings, never --set. A copy also lands in the controller namespace (listener config). A node clock ahead of GitHub gives 401.

3.Container modes

containerMode.typePod shapePrivilegedRunner's Kubernetes rights
unsetrunner container onlynonone
dindrunner + dind sidecaryesnone
kubernetesrunner + -workflow Pod, PVCnoPods, exec, logs, Jobs, Secrets
kubernetes-novolume (0.13.0+)as above, no PVCnosame, no batch/jobs
dind mode : one Pod dind sidecar privileged socket runner uses DOCKER_HOST kubernetes mode : two Pods per job runner Pod kube-mode Role hooks create workflow Pod job + services work PVC (RWO, one per runner) novolume : no PVC, files are copied
  • dind is privileged : it needs the privileged level and is node root (Image Builds). A native sidecar.
  • kubernetes needs kubernetesModeWorkVolumeClaim, or the render has spec: null and the API server rejects it.
  • Every job needs a job container (ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER). In kubernetes mode the workflow Pod is pinned to the runner's node.

4.The runner image

  • ghcr.io/actions/actions-runner : Ubuntu 24.04, UID 1001, docker group GID 123, passwordless sudo, Docker CLI and Buildx. The daemon binaries are there, but nothing starts them.
  • ARC turns runner self-update off, so the image tag is the runner version. GitHub stops sending jobs to a runner more than 30 days behind : pin and bump, or float latest with imagePullPolicy: Always (pulls).

5.Rendering for GitOps

  • controllerServiceAccount : set it, or helm template fails ("No gha-rs-controller deployment found").
  • CRDs (0.6 to 1.3 MB each) : client-side apply fails, may not be more than 262144 bytes. Use server-side apply (API machinery).
  • Argo CD : flags.excludeLabelPropagationPrefixes: ["argocd.argoproj.io/instance"], or it fights ARC's children.
  • Delete order : scale sets first, controller last, or finalizers strand objects (finalizers). In Flux, dependsOn the controller.
In short : Install the controller and each scale set as OCI charts in two namespaces, with a GitHub App Secret. dind is privileged. kubernetes mode adds a Pod per job.

1.Security model

  • Never on public repositories : anyone who can open a pull request runs code. Ephemeral runners stop persistence, not a job reading its secrets.
  • dind is node root (Image Builds). In the kubernetes modes the runner's Role can read Secrets in its namespace, the App key included : keep ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER=true, one scale set per namespace.
  • Workflow Pods run as the namespace's default ServiceAccount : set serviceAccountName in a hook template and bind it narrowly (RBAC).
  • Write an egress policy : DNS, 443 out, block 169.254.0.0/16. Workflow Pods carry another label, so they need their own (NetworkPolicy).

2.Upgrades and the version guard

controller 0.15.x reads each set's version label same minor 0.15.y : keep, update other, or no label DELETE the set

A minor upgrade is a reinstall :

  1. Uninstall every scale set, wait for cleanup.
  2. Uninstall the controller.
  3. Delete the four actions.github.com CRDs.
  4. Reinstall both charts at the same version.

Applying both new charts at once lets the old controller delete the new sets.

  • Patch upgrades update in place since 0.15.0. Every minor since 0.11.0 changed all four CRDs, and Helm never upgrades CRDs. Flux does only with upgrade.crds: CreateReplace (CRDs, Flux deep dive).
  • An outdated runner exits with code 7. 0.15.0 parks the whole set Outdated until the runner template changes (a new image does, maxRunners does not).
Seen in practicePitfall : re-rendering with a few --set flags instead of the full values. The chart has no values schema, so a missing work volume claim renders null and helm template raises no error. Do better : keep every value in one file, render both charts from it, diff the full set spec, grep for null.

3.Watching it

  • Since 0.15.0 kubectl get shows only min, max and phase : counts live in metrics, which the chart turns off (no Service either). Set the metrics: block and scrape listeners by Pod label (Prometheus).
  • Gauges : gha_assigned_jobs, gha_desired_runners, gha_busy_runners, gha_controller_failed_ephemeral_runners, and new gha_controller_outdated_ephemeral_runners. Opt-in job_workflow_ref labels add a series per pull request.

4.ARC, KEDA or a fixed pool

ARC scale setsKEDA github-runnerGitea Runner
Learns about jobslistener long poll, told the countREST API every pollingIntervalrunner polls Gitea
Scales0 to N, one Pod per jobScaledJob or Deploymentfixed pool you size
RegistrationJIT per runner, by the controlleryour runner image registerstoken, then .runner
Rate limitsnone on the scaling path5,000/h PAT, 15,000/h Appn/a

KEDA adds self-hosted, linux, x64 labels by default, ARC never adds self-hosted. Drains and OOM kills end jobs : karpenter.sh/do-not-disrupt helps with autoscalers, not kubectl drain (Karpenter, limits).

5.Commands

CommandDoes
kubectl get autoscalingrunnersets -n arc-runnersMin, max and phase per set
kubectl get pods -n arc-systems -l app.kubernetes.io/component=runner-scale-set-listenerOne listener per set
kubectl logs -n arc-systems -l app.kubernetes.io/name=gha-rs-controllerController logs (default names)
kubectl patch autoscalingrunnerset <name> -n arc-runners --type=merge -p '{"spec":{"minRunners":2}}'Warm runners, from a CronJob
In short : ARC keeps one ephemeral Pod per assigned GitHub job. Reinstall for a minor, and treat dind runners as node root and kube-mode runners as Secret readers.

CI and builds

Gitea Actions Runners

1.Gitea Actions in brief

  • Built into Gitea, with GitHub Actions workflow syntax. Files live in .gitea/workflows (else .github/workflows). On since Gitea 1.21. These notes use Gitea 28.1.0 and Gitea Runner 5.0.0.
  • The server queues jobs, runners execute them. No CRDs or controller (contrast ARC).
  • act_runner became Gitea Runner at v1.0.0 (2026-05-05) : image gitea/runner (the old one stopped at 0.6.1).

2.The pull model

  • The runner connects out to Gitea and polls every 2s, backing off to 5s when idle. Gitea never dials it : no Service, no inbound port. Offline after a minute.
  • Plan four connections : runner and job containers to Gitea (never register with localhost), runner and job containers to the internet (actions come from github.com unless DEFAULT_ACTIONS_URL = self). See NetworkPolicy.
Gitea 28 : run queue job: runs-on ubuntu-latest 1. FetchTask poll 2 to 5s 2. task: steps, secrets, job token 4. logs, step state gitea-runner daemon .runner identity, labels, capacity 3. by label schema docker:// job container, per-job network kubernetes:// one Pod per job host runner's own machine 5 5 : every backend fetches checkout, artifacts (job token)

3.Registering a runner

LevelTakes jobs fromToken in the UI
Instanceevery repository/-/admin/actions/runners
Organizationthat org's repositories/org/<org>/settings/actions/runners
Userthat user's repositories/user/settings/actions/runners
Repositorythat repository/<owner>/<repo>/settings/actions/runners
  • The docs print the org path without /org/ : a 404. A token is reusable until reset.
  • register trades it for the runner's own credential, written to .runner. Persist it, never copy it : lose it and the runner registers anew, leaving a stale offline entry.
  • Ephemeral runners (Gitea 1.24+) take one job, credentials revoked on assignment. You start them.

4.Labels pick where a job runs

LabelJob runsNeeds
ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latestin a container from that imagea Docker daemon
ubuntu-latest:kubernetes://...as its own Pod (v4.1.0+)Kubernetes 1.30+, a Role, an image with sh, tar, tail, env
linux-amd64:hoston the runner's machine, no isolationtools installed there; no service containers unless the job sets container:
  • Grammar <name>[:<schema>[:<args>]]. A runner gets a job only if it has every runs-on label. Since 5.0.0 a job without runs-on fails. No match : it waits, then is cancelled after 24h.
  • Labels come only from the runner (config, GITEA_RUNNER_LABELS or --labels), re-declared on every start. Gitea cannot edit them. Never call a host label ubuntu-latest.

5.Commands

CommandDoes
gitea --config /etc/gitea/app.ini actions generate-runner-token --scope myorgPrint a token on the Gitea host (no scope = instance)
kubectl -n ci create secret generic runner-secret --from-literal=token="$TOKEN"Store it with no trailing newline, unlike echo | base64
gitea-runner register --no-interactive --instance URL --token-file /run/token --labels ubuntu-latest:docker://IMAGERegister. --token-file keeps the token out of ps
kubectl -n ci logs runner-0 -c runner | grep -i declareConfirm it registered and declared its labels
In short : A Gitea runner polls Gitea for jobs, matches them by every runs-on label, and runs each one in a container, a Pod or on the host. Register it once and keep its .runner file.

1.Three shapes

The trade : full features and privileged, or unprivileged and fewer features. Upstream recommends the dind image for container jobs : jobs never get the host's socket. Privileged dind is node root (Image Builds, Pod Security).

DinD StatefulSetRootless dindkubernetes:// job Pods
Privileged containeryes, dockerdyes, still privilegedno
Docker actionsyesyesno, the job fails
Servicescontainers on the job networksameextra containers, on localhost
Job limitsthe runner Pod's, sharedsameeach job Pod's own, from pod_template
Extra RBACnonenoneRole : pods, pods/exec, pods/log, secrets

2.The DinD StatefulSet

runner Pod (ordinal 0) init: docker docker:dind, privileged restartPolicy: Always container: runner gitea/runner docker.sock (emptyDir) starts job containers per-job network PVC /var/lib/docker (image cache) PVC /data (.runner) outbound HTTPS only Gitea StatefulSet : concurrency = replicas x runner.capacity
  • A StatefulSet : each Pod keeps its own .runner via volumeClaimTemplates.
  • dind is a native sidecar : up before the runner, down after it (1.29+, GA 1.33, Pods). require_docker: true makes the runner wait.
  • Give dind's data root a volume, or every new Pod re-pulls all job images.
  • terminationGracePeriodSeconds above shutdown_timeout (upstream : 3h). Pin the tags in upstream's statefulset-dind.yaml.

3.kubernetes:// job Pods

  • ShapeNew in v4.1.0. One bare Pod per job, not a Job : a job container plus one per service.
  • NeedsKubernetes 1.30+, a namespaced Role (RBAC). Job Pods get no SA token by default.
  • Templatepod_template merges into every job Pod (name: job, $services). Set resources there. pod_templates.<label> add per-label overrides, such as a nodeSelector.
  • FailsDocker actions, docker:// steps, options, volumes, credentials. Pull secrets go in the template.

4.config.yaml keys that bite

KeyDefaultWhy it matters
runner.capacity1jobs at once per process; each concurrent docker job takes a subnet from the daemon's pool
runner.shutdown_timeout0srunning jobs are cancelled at once on SIGTERM, and Gitea does not re-queue them
container.privilegedfalsetrue makes every job container privileged and turns off option stripping
container.docker_host""jobs get the daemon's socket; "-" keeps it out
metrics.enabledfalse/metrics, /healthz, /readyz on 127.0.0.1:9101, no auth (Prometheus)

config init writes the file, config set edits a key. config generate is a commented reference, not a file to copy.

5.Sizing, and no autoscaler

  • Concurrency = replicas x runner.capacity. Maintainers advise more runners with capacity: 1 over one big one.
  • No upstream controller, CRD or scaler, and no KEDA scaler (contrast ARC). The chart gitea/helm-actions v0.1.2 defaults to runner 2.0.1 : pin your own tag.
  • Job containers on Docker's 1500 MTU hang on big transfers over a smaller pod network : lower dockerd's --mtu (job networks inherit it since 5.0.0).
Seen in practicePitfall : renaming a runner StatefulSet to tidy its name. Its selector and volumeClaimTemplates are immutable, so the new one starts on fresh volumes, every Pod registers anew and the old entries stay offline. Do better : pick a neutral name up front. If you must rename, delete the stale runners.
In short : Run Gitea runners as a StatefulSet so each Pod keeps its registration, and size the pool yourself. Privileged dind runs every workflow; kubernetes:// Pods trade features for no privileges.

1.Trust model

  • Privileged dind is node root for every workflow author, and rootless dind is still privileged (Image Builds). Pod Security Baseline rejects it, a native sidecar included : the runner namespace needs the privileged level.
  • Mitigate with a tainted node pool (scheduling), trusted repositories, egress limits, or kubernetes:// Pods.
  • An instance runner takes every repository's jobs : with stranger PRs, use repository or organization runners.

2.What the runner strips

workflow container.options runner strips --pid, --cap-add, --security-opt, --device ... unless container.privileged: true job container docker.sock (per-job proxy) dind daemon privileged the node docker_host: "-" cuts the socket edge above
  • Since 3.0.0 workflow container.options lose host-escape flags and --privileged is ignored (more in 5.0.0). container.privileged: true turns that off.
  • The runner's own options are trusted (unfiltered since 3.3.1). A workflow's --env-file and --label-file always fail.
Not a sandboxOn dind every job container gets the daemon's socket (a per-job proxy since 3.4.0) and can start privileged containers. No Docker needed ? container.docker_host: "-".
Seen in practicePitfall : setting container.privileged: true so one repository's daemonless builds work (Buildah needs user namespaces the default seccomp profile blocks). Every job container on that runner turns privileged. Do better : run a separate privileged pool with its own label, since the runner's container.options reach every job on it too.

3.Fork PRs, secrets and the job token

  • Gitea decrypts owner and repo secrets and sends them to the runner inside the task. Fork pull requests, except pull_request_target, get only the job token (GITEA_TOKEN, alias GITHUB_TOKEN), clamped read-only.
  • Several approval-gate bypasses for fork PR runs were fixed in 2026, some only in 28.0 : keep Gitea current.
  • The job token cannot publish packages : use a token with write:package (Image Builds). No id-token exists, so no keyless signing from jobs (Image Signing).

4.Differences from GitHub Actions

AreaGitea 28.1, runner 5.0
Contextsgitea.*, with github.* as an alias. No GraphQL.
Actions without a hostcloned from github.com unless DEFAULT_ACTIONS_URL = self
Token scopesno statuses, checks, deployments, id-token; adds code, releases, wiki, projects
runs-onlabels only, no runner groups; required since runner 5.0.0

5.Upgrading across majors

ToCheck
1.0renamed; since 1.0.5 env overrides like GITEA_RUNNER_CAPACITY are ignored : use config.yaml
2.0DOCKER_USERNAME and DOCKER_PASSWORD no longer give pull credentials
3.0one process per .runner, option stripping, cache v2 on
4.0with cache.external_server jobs reach the cache through their runner; valid_volumes globs
5.0git 2.34.1+ on the runner, ${NAME} expands in config, runs-on required
  • An upgrade is an image swap : check the env and config schema at the target tag. Registrations carry over.
  • Gitea 28 deletes completed runs after 400 days (RUN_RETENTION_DAYS) : set 0 first to keep history.
In short : Treat a privileged runner as root on its node : scope it, patch Gitea, and keep strangers' fork PRs away from it. Upgrades are an image swap, but read the breaking notes for each major.

CI and builds

Building Images in the Cluster

1.A RUN step is a container

  • A builder unpacks the base image. Each RUN runs in a new container over the layer stack and is saved as a layer. Then the image is pushed.
  • For that it calls unshare or clone with CLONE_NEWUSER, NEWNS and NEWPID, mounts overlay layers and a fresh /proc, and writes files as many UIDs.
  • No Docker socket on containerd nodes (Images and Pulls).
Two traps
  • No seccomp profile means Unconfined, unless the kubelet sets seccompDefault: true (upstream default false). A builder can break the day a node flips it.
  • In dind, containers get Docker's default profile, so Buildah fails with unshare(CLONE_NEWUSER) : not permitted, unless the job container is privileged or unconfined.

2.The four fences

CI job builder dockerd, buildkitd, buildah, kaniko each RUN needs unshare, mount, chown as root fences seccomp RuntimeDefault : no unshare, no mount AppArmor default : deny mount masked /proc : no fresh /proc Baseline : no privileged, no Unconfined ways past privileged : every fence down Unconfined profiles : rootless BuildKit user namespace Pod : root is not host root userspace snapshots : kaniko

3.The options

OptionPod needsLowest level
Docker-in-Dockerprivileged: truePrivileged
Rootless dindstill privileged: truePrivileged
BuildKit rootlessUnconfined profiles, UID 1000Privileged (explicit Unconfined)
BuildKit usernshostUsers: false plus privileged: truePrivileged
Buildah, userns PodhostUsers: false, UID 1000, no seccomp fieldBaseline, while seccomp is effectively Unconfined
kaniko forksroot, no extra capabilitiesBaseline

Nothing that runs RUN steps with real namespaces is shown to fit Restricted. Levels : Pod Security.

4.Docker-in-Docker

  • The official image says --privileged is required, and the rootless variant needs it too : "rootless" is the daemon's UID, not the Pod's power.
  • A privileged Pod is node root. Any workflow author on that runner can own the node, and the namespace must allow privileged.
  • ARC's dind mode and Gitea's dind sidecar are this : ARC, Gitea runners. Best as a native sidecar.
  • TLS off (DOCKER_TLS_CERTDIR="") and arguments empty or starting with a flag : the stock entrypoint listens on tcp://0.0.0.0:2375, no auth. Keep TLS, or start the arguments with the word dockerd.

5.kaniko, archived

  • Google archived it in June 2025 (last release v1.24.0). Forks : osscontainertools publishes images, Chainguard source tags only.
  • It is the exception : commands run in its own root and are snapshotted in userspace, so it needs root but no new namespaces and fits Baseline. Its README says it does not make untrusted builds safe, and it must run in its own image.
  • No multi-arch index of its own : use manifest-tool.
Seen in practicePitfall : a dind container beside the job with no startup probe, so the first docker call fails and every workflow grows its own until docker info loop. Do better : run dind as a native sidecar with a docker info startup probe, as ARC's dind mode does, so the job starts only once the daemon answers.
In short : A build needs namespaces and mounts. dind and BuildKit get them by being privileged or unconfined, Buildah can get them from a user-namespace Pod, kaniko fakes them in userspace. Pick the smallest hole you can live with.

1.Three shapes, and who gives you which

a. daemon beside the runner runner socket dockerd privileged RUN containers b. builder service CI job buildctl mTLS buildkitd Pods push registry c. daemonless, in the job Pod job Pod runs the builder buildah, kaniko or buildctl-daemonless.sh push registry no daemon to keep running
RunnerDaemon in the job ? Build with
ARC dindyes, privileged sidecar : docker build, Buildx
ARC kubernetesno : daemonless, remote BuildKit, or a hook template that adds dind
Gitea with dindyes, job containers privileged only with container.privileged: true : docker build, Buildah in a privileged job
Gitea kubernetes://no : daemonless or remote BuildKit

2.BuildKit on Kubernetes

  • ExamplesUpstream examples/kubernetes : Pod, Deployment plus Service with mTLS, StatefulSet (warm cache), Job with buildctl-daemonless.sh. Each privileged, rootless (recommended) or userns. All use master tags : pin one.
  • RootlessStill seccomp and AppArmor Unconfined, UID 1000, and --oci-worker-no-process-sandbox : leftover step processes are not killed and steps can signal the daemon.
  • Buildx driverdocker buildx create --driver kubernetes makes the Pods, privileged by default. rootless=true swaps in Unconfined profiles. A persistent-volume-claim option gives a StatefulSet with a PVC (v0.34.0+).
  • Traprequests.ephemeral-storage sets a limit, not a request : scratch use above it evicts the builder.
  • Clientbuildctl --addr kube-pod://<pod> needs pods/exec (RBAC).

3.Buildah in a user-namespace Pod

spec:   # abridged
  hostUsers: false
  containers:
  - name: buildah
    image: quay.io/buildah/stable:v1.43.4-immutable
    securityContext: {runAsUser: 1000}   # else root
  volumes:      # storage.conf points overlay here
  - {name: storage, emptyDir: {sizeLimit: 10Gi}}
  • No privileged, capabilities, /dev/fuse or seccomp field : Baseline on the field rules, but it works only while the node runs Pods Unconfined.
  • Rootless BuildKit cannot do this (rootlesskit needs its own mappings) : BuildKit in a user namespace runs the rootful image with privileged: true.

4.User namespaces and Pod Security

RuleWith hostUsers: false
Restricted runAsNonRoot, runAsUser; Baseline procMountskipped; Unmasked allowed only with it
Policy versiononly v1.35 and later : a namespace pinned to enforce-version: v1.34 gets none
Still checkedprivileged, capabilities, seccomp, AppArmor, host namespaces
privileged: trueallowed, still the Privileged level

The field, node needs and UID mapping : Pod Security.

The Buildah Pod is one published setup (CERN, 2025), not upstream advice.

5.Storage and scratch

  • vfs works anywhere but copies every layer : slow and disk-hungry. Overlay on an overlay root fails (a mount_program is required) : use an emptyDir or PVC, or fuse-overlayfs (needs /dev/fuse).
  • Image-declared VOLUMEs (dind's /var/lib/docker, Buildah's /var/lib/containers) with no mount become a fresh directory in containerd's state on the node for every container start : the data is lost on each restart.
  • Mount an emptyDir with a sizeLimit and set limits.ephemeral-storage, or a big build fills the node or is evicted (Requests and Limits).
In short : A builder runs as a privileged daemon beside the runner, as a shared BuildKit service, or daemonless in the job Pod. Buildah in a user-namespace Pod is the one shape here that runs RUN steps in real namespaces without privileged, and only while seccomp stays unconfined.

1.Multi-arch

source build stage, once per target FROM --platform=$BUILDPLATFORM (native) GOOS=$TARGETOS GOARCH=$TARGETARCH final stage per TARGETPLATFORM COPY --from=build no RUN, so no QEMU index : linux/amd64 + linux/arm64 push to registry other ways : QEMU, one builder per arch
  • dockerd's own builder (docker:dind has no emulators) and Buildah need binfmt handlers : a registration from a privileged container changes every Pod on that node until reboot. Pin the binfmt image : Buildx qemu.image and setup-qemu-action both default to a floating latest.
  • Emulated steps can crash (qemu: uncaught target signal 11) : keep RUN out of the final stage.
  • Native builders : pick nodes by kubernetes.io/arch (Scheduling).
StrategyNeedsCatch
QEMUbundled in BuildKit images, else binfmt handlersmuch slower, node-wide
Native nodesone builder per arch (nodeselector=kubernetes.io/arch=...)needs those nodes
Cross-compile$BUILDPLATFORM stage, ARG TARGETOS TARGETARCHtoolchain must cross-compile
Per-arch, then mergebuildx imagetools create, buildah manifest, manifest-tool (kaniko)extra merge step

2.Reproducible output

  • SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct). BuildKit stamps metadata from it; layer file times need rewrite-timestamp=true, and a failed rewrite only warns : compare two digests.
  • Buildah 1.41+ : --source-date-epoch, --rewrite-timestamp. kaniko : --reproducible.
  • Pin bases by digest and compatibility-version (30). Provenance records the build.

3.Caching

  • A per-job builder starts cold. A registry cache survives : --cache-to type=registry,ref=...,mode=max (min, the default, skips earlier stages).
  • RUN --mount=type=cache is not exported : a warm builder needs a PVC-backed StatefulSet. Buildah needs --layers for --cache-to, kaniko --cache-repo.
  • Never share a BuildKit cache between trusted and untrusted builds (a poisoning bug, fixed in v0.33.1).

4.Pushing

  • A dockerconfigjson Secret as config.json (BuildKit $DOCKER_CONFIG, kaniko /kaniko/.docker/, Buildah --authfile). Record the digest.
  • Pull request builds never push. Only protected branches do, with a narrow write-only token (Gitea's job token cannot publish packages).
  • Anonymous Docker Hub pulls : 100 per 6 hours per IPv4 address or IPv6 /64, shared by one NAT. Log in or mirror.
  • Then sign and gate it, and let GitOps pick up the tag (Flux, Image Updater).
Seen in practicePitfall : an arm64 leg that compiles native code under QEMU : every build takes many minutes and a cache miss repeats them. Do better : cross-compile in a $BUILDPLATFORM stage where the toolchain allows it, move the heavy compile into a rarely rebuilt base image, and keep a mode=max registry cache.

5.Commands

CommandDoes
docker buildx create --bootstrap --name kube --driver kubernetes --driver-opt namespace=buildkit,rootless=trueBuildKit Pods as a Buildx builder
docker buildx build --platform linux/amd64,linux/arm64 --push --cache-to type=registry,ref=<repo>:buildcache,mode=max --cache-from type=registry,ref=<repo>:buildcache -t <repo>:<tag> .Multi-arch with a registry cache
buildah build --platform linux/amd64,linux/arm64 --manifest <list> --source-date-epoch "$(git log -1 --pretty=%ct)" --rewrite-timestamp .Multi-arch, reproducible
buildah manifest push --all --digestfile digest.txt <list> docker://<repo>:<tag>Push all, keep the digest
kubectl label ns <builds> pod-security.kubernetes.io/enforce=privilegedAllow privileged here
In short : Cross-compile instead of emulating, set SOURCE_DATE_EPOCH and the timestamp rewrite, push a mode=max registry cache, and give push credentials only to protected-branch builds.

Certificates and secrets

cert-manager

1.Why cert-manager ?

A Kubernetes add-on that automatically requests, stores and renews TLS certificates. Latest release : v1.21.3.

  • Certs expire (Let's Encrypt certs last 90 days or less)
  • Manual way : make a CSR, get it signed, paste into a Secret, remember to repeat. Nobody remembers.
  • Kubernetes has no "issue and renew" object. Secrets just hold bytes.
  • Adds Certificate and Issuer as CRDs, so a cert is declarative like a Deployment
Manual certs CSR Secret 3am ! forgot to renew, cert expires With cert-manager Certificate TLS Secret renews itself, forever History 2017202020222024 JetstackSandboxIncubatingGraduated

2.Components

Runs as Deployments in the cert-manager Namespace.

  • You declare WHAT (Certificate) and WHO signs (Issuer)
  • Output : Secret of type kubernetes.io/tls with tls.crt, tls.key (and ca.crt if the issuer hands over its CA. ACME does not.)
  • Pods mount it, Ingress and Gateway point at it
Webhook down ?Applying cert-manager objects fails.
-> Wait, or run cmctl check api
Wait, or run cmctl check api
The pieces
  • controllerThe brain. Reconcile loops for Certificate, CertificateRequest, Order, Challenge and Issuers.
  • webhookValidates cert-manager resources. Stamps who asked on each CertificateRequest.
  • cainjectorInjects CA bundles into webhook configs, CRDs and APIServices.
  • acmesolverSmall pod that answers HTTP-01 challenges.
  • startupapicheckInstall Job that checks webhook and API are reachable.
  • cmctlCLI : status, renew, approve, check api.

3.Issuers and CRDs

Issuer is namespaced, ClusterIssuer is cluster-wide. Same spec, different scope.

  • Built in : SelfSigned, CA, Vault, ACME, Venafi (now CyberArk's)
  • External issuers plug in via issuerRef group and kind
  • ACME issuer needs an account key Secret, privateKeySecretRef
Issuer signs Certificate TLS Secret stored in ClusterIssuer = any namespace
Scope rule A Certificate in namespace A cannot use an Issuer in namespace B. Use a ClusterIssuer to share.
KindapiVersionScopeWhat it is
Issuercert-manager.io/v1NamespacedSigner, one namespace
ClusterIssuercert-manager.io/v1ClusterSigner, any namespace
Certificatecert-manager.io/v1NamespacedDesired cert, goes to spec.secretName
CertificateRequestcert-manager.io/v1NamespacedOne CSR + result, auto-created
Orderacme.cert-manager.io/v1NamespacedACME order (ACME only)
Challengeacme.cert-manager.io/v1NamespacedOne domain proof (ACME only)
In short : Declare a Certificate and an Issuer. cert-manager's controller, webhook and cainjector do the rest, and the result is a TLS Secret.

1.ACME Issuance Flow

Certificate to Secret, step by step. The error always lives at the bottom of the chain.

Certificate issuance (ACME)
CertificateYou apply it.
CertificateRequestController makes it, with the CSR.
Issuer (ACME)Picked via issuerRef.
OrderOne per cert.
ChallengeOne per domain.
Let's EncryptSolver proves the domain.
Signed certReturned to the Request.
Secrettls.crt and tls.key stored.

CA, Vault and SelfSigned skip Order and Challenge. The Request goes straight to signing.

2.Solvers, Renewal, Ingress

  • Renews at about 2/3 of lifetime. Default duration is 90 days.
  • Controller watches Certificate + Secret. Expired, missing or mismatched triggers a new Request.
  • Annotate an Ingress with cert-manager.io/cluster-issuer and it makes the Certificate for you (ingress-shim)
  • Gateway API : install the Gateway CRDs first (or restart cert-manager after), then set config.gatewayAPI.enabled=true
HTTP-01DNS-01
ProofToken at /.well-known/acme-challengeTXT record in DNS
WildcardsNoYes
Public HTTP routeNeededNot needed
DNS API credsNoYes (Route53, Cloudflare)
Good forSimple public sitesWildcards, internal clusters
Ingress annotated controller creates (ingress-shim) Certificate fills Secret served as HTTPS

3.Example, Commands and Gotchas

kind: ClusterIssuer
metadata: {name: letsencrypt-staging}
spec:
  acme:
    email: user@example.com
    server: <staging URL>
    privateKeySecretRef: {name: le-staging-key}
    solvers:
    - http01: {ingress: {ingressClassName: nginx}}
kind: Certificate
metadata: {name: example-com, namespace: sandbox}
spec:
  secretName: example-com-tls
  dnsNames: [example.com, www.example.com]
  issuerRef:
    {name: letsencrypt-staging, kind: ClusterIssuer}
Gotchas
  • Test on the STAGING URL (acme-staging-v02.api.letsencrypt.org). Production has rate limits.
  • Stuck cert ? Walk down : Certificate, Request, Order, Challenge
  • "failed calling webhook" right after install ? Wait for the webhook
  • Renewed Secret does not restart Pods. Reload in the app or use a reloader.
  • Gateway listeners need protocol HTTPS or TLS, a hostname, tls.mode Terminate, certificateRefs in the Gateway's namespace
CommandDescription
helm install cert-manager oci://quay.io/jetstack/charts/cert-manager --version v1.21.3 -n cert-manager --create-namespace --set crds.enabled=trueInstall with Helm. crds.enabled=true installs the CRDs.
kubectl get clusterissuer,issuer,certificate -AList issuers and certs. READY=True is the goal.
kubectl describe certificate <name> -n <ns>Conditions and events of a stuck cert
cmctl status certificate <name> -n <ns>Detailed status with related resources
cmctl renew <name> -n <ns>Force a renewal now (also --all)
In short : A stuck cert is solved by walking Certificate, Request, Order, Challenge. Test on staging, and use DNS-01 for wildcards.

Certificates and secrets

External Secrets Operator

1.What is ESO ?

ESO reads secrets from a vault or cloud secret manager (AWS, GCP, Azure, Vault...) and keeps a normal Kubernetes Secret in sync with them.

Git holds only pointers to secrets, never the values. Pods stay unchanged, they still mount a normal Secret.

Latest : v2.12.0 (CRDs use external-secrets.io/v1). CNCF Sandbox project.

In Simple Words
ESO fetches secrets from outside the cluster and turns them into Kubernetes Secrets, then keeps them fresh.

Vault -> ESO -> Secret -> Pod

2.Why it exists ?

  • Secrets in K8s are only base64. Committing them to Git leaks credentials.
  • The real source of truth lives in AWS, GCP, Azure or Vault. Copying values by hand goes stale.
  • Rotating a secret means someone must update every cluster.

Fix : declare WHERE the secret lives in Git, and a controller fetches and refreshes it.

Key Points
  • Declarative, no values in Git
  • Re-fetches every refreshInterval
  • Many backends, one API
  • Output is a plain Secret
  • Templating reshapes values
  • PushSecret goes the other way
  • Generators create values
  • Works with Flux/Argo (just YAML)

3.Parts of ESO

  • ControllerMain operator. Reconciles ExternalSecret and PushSecret, calls providers, writes Secrets.
  • WebhookValidates the ESO CRDs (own deployment).
  • Cert-controllerManages TLS certs for the webhook (own deployment).
  • GeneratorsMake values: Password, UUID, SSHKey, ECR/GCR/ACR tokens and more.
Install Order
  1. Install ESO with Helm
  2. Create a store with auth
  3. Create an ExternalSecret
  4. Controller writes the Secret
  5. Pod mounts the Secret
  6. Refresh loop keeps it current

4.How the Sync Works

watch uses ExternalSecret(Git / kubectl) ESO controller SecretStore Provider(AWS SM, Vault) auth + fetch values K8s Secret create / update plain Secret,Pod unchanged Pod env / mount ExternalSecret(Git / kubectl) watch ESO controller SecretStore fetch values writes Provider(AWS SM, Vault) K8s Secret env / mount Pod plain Secret,Pod unchanged
Reconcile Loop
  1. Watch ExternalSecret
  2. Resolve store
  3. Fetch remote data
  4. Apply template
  5. Write Secret
  6. Wait refreshInterval, repeat from 3

Ready=True on success, SecretSyncedError on failure.

In short : ESO pulls secrets from an external provider and keeps a normal Kubernetes Secret fresh, so Git never holds a value.

1.ESO Kinds + Store Scoping

A SecretStore only works in its own namespace. A ClusterSecretStore is shared, so restrict it with conditions. Tip : one SecretStore per namespace, or one shared ClusterSecretStore.

KindScopePurpose
SecretStoreNamespacedProvider config and auth
ClusterSecretStoreClusterShared provider config
ExternalSecretNamespacedFetch remote data into a Secret
ClusterExternalSecretClusterExternalSecrets in many namespaces
PushSecretNamespacedPush Secret to provider
ClusterPushSecretClusterPushSecret across namespaces
ClusterSecretStore shared ExternalSecretNamespace A ExternalSecretNamespace B SecretStore own store,only in A

2.Example ExternalSecret + Gotchas

kind: ExternalSecret
metadata: {name: db-creds}
spec:
  refreshInterval: 1h
  secretStoreRef: {name: vault, kind: SecretStore}
  target: {name: db-creds, creationPolicy: Owner}
  data:
  - secretKey: password
    remoteRef: {key: prod/db, property: password}
Gotchas
  • Store must be Ready first
  • Env vars need a reloader, files refresh
  • Prefer workload identity (IRSA)
  • Low refreshInterval hits rate limits
  • Owner policy : delete ES, Secret goes too
  • Migrate v1beta1 / v1alpha1 to v1

3.ESO vs Plain Secret vs CSI Driver

Source of truthCreates K8s SecretRotationPod changesHow it works
Plain SecretIn clusterManualManualNoneStatic
ESOExternal providerYes, automaticEvery refreshIntervalNoneController + CRDs
Secrets Store CSIExternal providerOptional (sync)Mount rotationCSI volume in PodNode driver

4.ESO Commands

CommandDescriptionExample
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespaceInstall operator + CRDs--set installCRDs=false--set installCRDs=false
kubectl get externalsecret -AList, with sync statuskubectl get es -n appkubectl get es -n app
kubectl describe externalsecret <name>Conditions + eventskubectl describe es db-credskubectl describe es db-creds
kubectl get secretstore,clustersecretstore -ACheck stores are Readykubectl get csskubectl get css
kubectl -n external-secrets logs deploy/external-secretsController logsadd -f to followadd -f to follow
kubectl annotate es <name> force-sync=$(date +%s) --overwriteForce immediate refreshes = externalsecretes = externalsecret
In short : Pick a namespaced SecretStore or a shared ClusterSecretStore, point an ExternalSecret at it, and check stores are Ready first. Unlike the CSI driver, Pods need no changes.

Data

CloudNativePG Operator

1.What is CNPG ?

An operator that runs highly available PostgreSQL on Kubernetes. You write one Cluster YAML and it builds the Pods, volumes, services, failover and backups.

  • A plain Postgres Deployment has no replication, no failover, no backups.
  • StatefulSets give stable names and disks, but no Postgres knowledge.
  • Operator = the DBA runbook as a reconcile loop.
  • Talks to the Kubernetes API directly. No Patroni, no StatefulSet.
Reconcile loop Cluster YAML API server watch event OperatorDeployment create / patch Pods / PVCsServices / Secrets Instance manager status report, drift repeats the loop
Notes CNCF Incubating (TOC vote 2026-11-02; Sandbox since Jan 2025). Latest stable checked : v1.30.1. Core CRD : postgresql.cnpg.io/v1 kind Cluster.

2.Cluster Layout

One primary plus N hot standbys. Each Pod has its own PVC, no shared disk. Replication is native Postgres WAL streaming.

ServicePoints to
app-db-rwPrimary only
app-db-roReplicas only
app-db-rAny instance
Writers always use -rw.
App Pod app-db-rw app-db-ro writes reads PrimaryPVC 1 ReplicaPVC 2 ReplicaPVC 3 WALstreaming S3 / GCSAzure WAL archive + base backup go to object storage

3.Parts and CRDs

  • OperatorDeployment in cnpg-system. Helm names it cnpg-cloudnative-pg, the plain manifest cnpg-controller-manager. Watches the CRDs.
  • Instance managerPID 1 in every Postgres Pod. Probes, WAL archiving, promotion.
  • PoolerManaged PgBouncer (rw or ro).
  • Barman Cloud pluginShips WAL and backups to object storage. Separate install : cert-manager first, then into the operator's namespace.
  • kubectl cnpgCLI plugin for status, promote, psql.
KindWhat it does
ClusterThe Postgres cluster
BackupOne backup request
ScheduledBackupCron style backups
PoolerPgBouncer pooling
DatabaseDeclarative DB
DatabaseRoleDeclarative role
Publication / SubscriptionLogical replication
ImageCatalog / ClusterImageCatalogImage per Postgres major (namespace, cluster-wide)
FailoverQuorumSync failover quorum (with failoverQuorum)
ObjectStoreThe plugin's bucket config (barmancloud.cnpg.io)
In short : CNPG turns one Cluster YAML into a primary plus streaming replicas, each with its own PVC, behind -rw / -ro / -r Services.

1.Failover

Automatic inside one Kubernetes cluster. Across clusters it is manual : promote the replica cluster yourself.

  • failoverDelay / switchoverDelay tune the timing.
The Leasespec.primaryLease (15/10/2/1 s) : a Lease named after the Cluster, held by the primary's Pod. A promotion gate, not a fence.
Failover Flow
Primary fails
Operator detectstarget pending, old primary unhealthy (out of -rw)
Pick replicamost advanced, once every WAL receiver is down
Promotetakes the Lease, then -rw follows
Old primaryrejoins as replica

2.CNPG vs DIY StatefulSet

TopicDIY STSCNPG
FailoverManual or PatroniBuilt in
ReplicasYou script theminstances: N
Backups / PITRYour cron jobsBackup CRDs
PodsOrdinal orderLowest free serial (1.30) : a recreated instance keeps its name
ConfigEdit, restartRolling update
Passwords / TLSHand madeGenerated
Failover timings
  • A graceful delete of the primary waits up to 180 s for open sessions (the smart shutdown) before the failover starts.
  • After a node loss the phase stays Failing over until the old Pod is evicted, even though a replica is already promoted.

3.Try It

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: app-db
  namespace: apps
spec:
  instances: 3
  storage:
    size: 10Gi
  postgresql:
    parameters:
      max_connections: "200"
  1. Install operator into cnpg-system.
  2. Apply a Cluster with instances: 3.
  3. Operator makes primary Pod and PVC, then clones replicas.
  4. It adds -rw, -ro, -r Services and the app-db-app Secret.
  5. App connects to app-db-rw.apps.svc:5432.
  6. Failover is automatic. WAL ships only once a WAL archiver plugin is set up (next sheet).
Clusterapp-db the operatorbuilds these Pods app-db-1 / 2 / 3+ one PVC each Servicesapp-db-rw / -ro / -r Secretapp-db-app (user + password)
In short : Declare a Cluster and CNPG runs Postgres for you : Pods, PVCs, -rw/-ro/-r Services and failover, with no StatefulSet. Backups need a plugin.

1.Backups Need a Plugin

The in-tree barmanObjectStore is deprecated since v1.26 and goes in v1.32.0 (#11671). Backups and WAL archiving now come from a plugin, most often the Barman Cloud plugin.

  • Install cert-manager first. The plugin runs in the operator's namespace.
  • An ObjectStore (barmancloud.cnpg.io) names the bucket and its keys.
  • The Cluster lists it under plugins:. Nothing is archived until then.
  • Recovery always makes a new Cluster. If it archives too, give it its own serverName : the source's fails with Expected empty archive.
  • Or back up with volume snapshots.
Cluster app-db + plugin sidecar archives bucket WAL + base backups plugins: ObjectStore barmancloud.cnpg.io points at restore new Cluster its own serverName
# in the Cluster's spec
plugins:
- name: barman-cloud.cloudnative-pg.io
  isWALArchiver: true
  parameters:
    barmanObjectName: my-store

Without this entry, applying a Cluster archives nothing, and the Cluster still reads healthy.

Watch out
  • Delete a Cluster and its PVCs go, data included. The safety net is the bucket and the snapshots : Backup objects hold no data, and Poolers keep running, inactive.
  • instances: 1 means no HA, and its primary PDB makes its node undrainable. Use 3 across nodes : the default anti-affinity is only preferred.
  • The default image is the deprecated system flavour. Pin a dated minimal tag plus its digest : a bare digest is refused.
  • Never write via a Pod IP or -r. kubectl get sts shows nothing : look at pods, pvc, clusters.

2.Useful Commands

CommandDescription
helm repo add cnpg https://cloudnative-pg.io/chartsAdd the chart repo (the old github.io address redirects here)
helm upgrade --install cnpg -n cnpg-system --create-namespace cnpg/cloudnative-pgInstall the operator
kubectl get cluster -n appsList CNPG clusters
kubectl cnpg status app-dbHealth, primary, lag
kubectl cnpg promote app-db app-db-2Manual switchover
kubectl cnpg psql app-dbpsql on the primary
kubectl cnpg backup app-db -m plugin --plugin-name barman-cloud.cloudnative-pg.ioOn-demand backup through the plugin. Without -m plugin it asks for barmanObjectStore and fails : the cluster has no backup section
In short : Backups and WAL archiving need a plugin, the Barman Cloud plugin with cert-manager, and a restore is always a new Cluster with its own serverName.

Cloud infrastructure

Crossplane

1.What is Crossplane ?

A control plane framework for platform engineering. You define your own Kubernetes APIs and Crossplane runs the controllers behind them : custom resources without writing an operator.

  • Control plane = a reconcile loop for things outside the cluster : a cloud bucket, a database, a GitHub repo.
  • Declare it once. Hand edits in the cloud get reverted.
  • All of it is API objects, so kubectl, RBAC and GitOps (Argo CD, Flux) just work.
PartWhat it gives you
Managed resourcesOne object per cloud thing
CompositionYour own API from many parts
OperationsOne-off, cron or watch runs
Package managerProviders, functions, configs

Use all four or only some. CNCF graduated 2025-10-28. Latest core v2.4.3 (2026-10-22), a new minor every quarter, the three newest supported.

2.How the parts fit

Helm installs two Deployments in crossplane-system. Packages add more pods.

  • crossplane : core. Installs core CRDs, starts an XR controller per XRD, calls functions.
  • rbac-manager : grants core access to XRs and MRs. Others may need a ClusterRole.
  • Provider pod : one per provider, syncs MRs with a cloud API.
  • Function pod : Deployment + Service, gRPC on 9443.
XRD + Composition platform team XR (kind App) user, in a namespace defines reads watches crossplane pod XR controller Function pods RunFunction, :9443 gRPC apply (SSA) observed Composed resources MRs + any kind Provider pod e.g. provider-aws-s3 watches status cloud API S3 bucket, DB ... the provider reverts drift (AWS default poll 10m)

3.Providers and managed resources

A provider package adds MR CRDs and runs a controller. One managed resource (MR) = one external thing, like one S3 bucket.

  • forProvider : desired settings, enforced. initProvider : used only at create. atProvider : what the cloud reports.
  • managementPolicies : Observe, Create, Update, Delete, LateInitialize. Default ["*"].
  • Keep the cloud resource on delete : leave Delete out. Namespaced MRs have no deletionPolicy.
  • Import : set crossplane.io/external-name and start with ["Observe"].
apiVersion: s3.aws.m.upbound.io/v1beta1
kind: Bucket            # namespaced MR
metadata:
  namespace: default
  generateName: crossplane-bucket-
spec:
  forProvider:
    region: us-east-2
  # providerConfigRef defaults to
  # {kind: ClusterProviderConfig, name: default}
ProviderConfig serves MRs in its own namespace. ClusterProviderConfig serves any namespace. provider-aws-s3 pulls in provider-family-aws, which owns the ProviderConfig.

4.What v2 changed

AreaIn v2 (since Aug 2025)
XRsNamespaced by default. Cluster scope still allowed
ClaimsLegacy. Only with a v1 XRD, scope LegacyCluster
MRsNamespaced, in .m. groups. Cluster MRs are legacy
CompositionsPipeline mode only. May include any Kubernetes kind
OperationsNew, alpha : run a pipeline once, on cron or on watch
Provider CRDsBecome MRDs. With safe-start, an MRAP picks the CRDs
RemovedNative P&T, ControllerConfig, secret stores (use ESO)
Upgrading from v1
  1. One minor at a time, latest patch, up to v1.20.
  2. Run crossplane beta upgrade check (v1.20.9+ CLI only).
  3. Convert mode: Resources, drop --registry, fully qualify packages.
  4. Old claims and legacy MRs keep working.
In short : Crossplane turns Kubernetes into a control plane : providers keep managed resources in sync with the cloud, and functions build your own APIs on top.

1.XRD and Composition

XRD = schema of a new API, turned into a CRD plus an XR controller. Composition = how to build each XR.

apiVersion: apiextensions.crossplane.io/v2
kind: CompositeResourceDefinition
metadata: { name: apps.example.crossplane.io }
spec:
  scope: Namespaced  # default; or Cluster. Immutable
  group: example.crossplane.io
  names: { kind: App, plural: apps }
  versions:          # openAPIV3Schema trimmed
  - { name: v1, served: true, referenceable: true }
---
apiVersion: example.crossplane.io/v1
kind: App            # the XR
metadata: { namespace: default, name: my-app }
spec: { image: nginx }  # + spec.crossplane fields
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata: { name: app-yaml }
spec:
  compositeTypeRef:     # immutable
    apiVersion: example.crossplane.io/v1
    kind: App
  mode: Pipeline        # the only mode in v2
  pipeline:             # 1 to 99 steps
  - step: deployment
    functionRef: { name: function-patch-and-transform }
    input:
      apiVersion: pt.fn.crossplane.io/v1beta1
      kind: Resources   # base Deployment + patches
  - step: mark-ready    # last, bar a sequencer
    functionRef: { name: function-auto-ready }

2.How the pipeline runs

ObserveXR + children, once
1. patch + transformadds the Deployment
2. more stepstemplates, KCL, Python
3. auto-readymarks ready
Applyserver-side apply
  • Each step gets the snapshot, the desired state so far and a context.
  • Pass on all desired state. Anything dropped is deleted.
  • A step may set XR status, never the XR's own spec.
  • Children get an owner reference to the XR (GC), applied with SSA.
  • Namespaced XR : children stay in its namespace. Cluster-scoped kinds fail.
  • Functions do not run when an XR is deleted.

3.Composition functions

function-LatestInput kindGood for
patch-and-transformv0.11.2Resourcesv1-style patches, transforms, readiness checks. No loops or ifs
go-templatingv0.13.2GoTemplateGo templates with Sprig, inline or from files
kclv0.12.3KCLInputKCL programs, inline or oci://
pythonv0.6.0Scriptcompose() or operate(). Also runs in Operations
environment-configsv0.9.2InputLoads EnvironmentConfigs into the pipeline context
auto-readyv0.7.3InputMarks healthy composed resources ready. Put it last

Install as a pkg.crossplane.io/v1 Function. SDKs : Go, Python. function-sequencer (v0.6.1) orders creation, after auto-ready.

4.Readiness and status

  • A composed resource is ready only when a function returns READY_TRUE. Unset = not ready. Crossplane never guesses.
  • auto-ready has checks for core kinds (Deployment, Service, Job ...). Others, MRs too, need Ready=True.
  • No Ready condition at all (a composed Provider, a Cluster API v1beta2 Cluster) = XR stuck. Add an alpha CEL check.
  • XR status : a function returns a partial XR with only status.
External resource in the cloud provider MR status atProvider, Ready observed Function pipeline auto-ready runs last READY_TRUE XR Ready, Synced, conditions legacy only Claim waits until XR Ready XR Ready rule all READY_TRUE : Ready=True, Available otherwise : Creating, Unready ... Warning result : event Fatal : Synced=False TTL (1m) requeues XR
In short : The XRD defines the API, functions build it, and nothing is Ready until a function says so.

1.CRD count and MRD activation

About 3 MiB of API server memory per CRD (docs estimate). provider-upjet-aws v2.9.0 ships 1063 namespaced + 1063 legacy service CRDs.

  • Families : one provider per cloud service.
  • MRDs (alpha API) : each MR CRD becomes an MRD. Only an active one makes a real CRD. One-way.
  • Helm activates all by default. Install with --set provider.defaultActivations={}, then write MRAPs.
  • Copy names from kubectl get mrds : the docs' *.aws.m.crossplane.io examples match nothing.
Provider package e.g. provider-aws-ec2 no safe-start : all Active at install each MR CRD MRD, Inactive safe-start default MRAP names, *.prefix matches yes no Active : real CRD about 3 MiB each Inactive : no CRD 197 cost about 20 MiB v2.4+ : provider pod at 0 replicas until one MRD is active

2.Operations (alpha)

A pipeline that runs to the end, like a Job for API objects. Off until --enable-operations.

  • Operation : once. CronOperation : schedule. WatchOperation : on change.
  • Functions need the operation capability (function-python).
  • Force-applies, no owner refs, can't delete. retryLimit 5 if unset.

3.Packages

Provider, Function, Configuration : pkg.crossplane.io/v1, fully qualified OCI images.

  • Configuration : XRDs, Compositions, dependsOn.
  • Each change = a new revision, one active.
  • DeploymentRuntimeConfig shapes the pod, ImageConfig rewrites registries.
  • Pin by @sha256:. With signature checks on, tags could be swapped before v2.2.3 / v2.3.3.

4.Commands

CommandDoes
helm install crossplane --namespace crossplane-system --create-namespace crossplane-stable/crossplaneInstall core. Repo charts.crossplane.io/stable
curl -sfL "https://cli.crossplane.io/install.sh" | XP_VERSION=v2.5.0 shCLI crossplane (was crank), pinned. Own release train
kubectl get pkg ; kubectl get xrdPackages INSTALLED / HEALTHY, XRDs ESTABLISHED
kubectl get composite -A ; kubectl get managed -AEvery XR, every MR (slow with many CRDs)
kubectl get mrds ; kubectl get mrap -o yamlMRD state, and what each MRAP activated
crossplane resource trace app my-app -n default -o wideXR, children and MRs as a tree with status
crossplane composition render xr.yaml composition.yaml functions.yamlRun the pipeline locally (Docker)
crossplane cluster topCPU and memory of Crossplane pods. Needs metrics-server
crossplane xpkg build --package-root=package/ then xpkg pushBuild and push a package

5.Crossplane vs Terraform vs Cluster API

AspectCrossplaneTerraform / OpenTofu
ShapeAlways-on control planeA CLI you run
StateAn object per resourceOne locked state file
DriftReverted by the loopOnly when someone runs it
AbstractionXRD = a real API + RBACModules
Immutable fieldNever appliedPlans a replacement
Cluster API Clusters only, by design. Crossplane can compose a CAPI Cluster, but v1beta2 reports Available, not Ready : auto-ready needs a CEL check.
Gotchas
  • No READY_TRUE, no Ready XR. Read events too.
  • P&T paths are Optional : typos are silent.
  • Modern XRs publish no connection Secret.
  • A Provider delete is never refused. MRs first.
Good to know
  • Uninstall : XRDs, MRs, Providers, core, CRDs.
  • paused: "true" also blocks deletes.
  • Webhooks off = Usages silently off.
  • Many CRDs slow Argo CD : raise QPS.
In short : Activate only the MRDs you use, pin packages by digest, and debug with resource trace.

Scaling and metrics

KEDA

1.What is KEDA ?

KEDA scales your workloads based on external events (queue length, Kafka lag, cron, Prometheus query) and can shrink them to zero pods when idle.

  • Built-in HPA only scales on CPU / memory.
  • A backlog of 10k messages can sit at low CPU.
  • An HPA reaches zero only with an Object or External metric (beta in v1.37). Idle workers burn money.
  • KEDA ships many ready-made scalers.
  • Batch work wants one Job per event.
  • Pay for workers only while there is work to do.
FeatureHPA aloneKEDA
SignalsCPU, memory, custom metrics you provideDozens of event sources
To zeroBeta in v1.37, Object or External metric onlyYes (min 0)
JobsNoScaledJob
TypeBuilt inAdd-on that feeds an HPA
HistoryMade by Microsoft and Red Hat (2019). CNCF accepted 2020, incubating 2021, graduated 2023-08-22. Latest v2.21.0 (2026-09-23); a 2.21.1 patch is agreed, not out yet.

2.How KEDA scales

KEDA does not replace HPA, it drives it. Two tracks :

  • 0  1 is done by keda-operator.
  • 1  N is done by a normal HPA fed by KEDA metrics.
Activation vs scalingactivationThreshold gates 0  1. threshold sets the target per pod for 1  N. Activation fires only when the value is strictly greater.
Event sourceKafka / SQS/ Prometheus polls30s Scalerin keda-operator value metrics-apiserverexternal metrics active? metrics keda-operator HPA 0 to 1 1 to N Deployment Pods (0..N) two tracks,one Deployment
Scale targetDeployment, StatefulSet or any resource with a /scale subresource.

3.Components

  • keda-operatorWatches ScaledObject / ScaledJob, creates the HPA, does 0  1.
  • metrics-apiserverServes external.metrics.k8s.io so the HPA can read event metrics.
  • admission-webhooksValidate KEDA resources on apply (not in the -core manifest).
  • ScalersBuilt-in plugins that query an event source.
  • HPA (generated)Created per ScaledObject, does 1  N.
  • HTTP Add-onOptional, separate, beta. Scales on HTTP traffic.
Where things liveKEDA itself lives in the keda namespace. A ScaledObject lives beside the workload it scales.
operator, metrics server, webhooks
Key PointsInstall with Helm into the keda namespace. One ScaledObject per workload. Pick the HPA by name if you already have one.
In short : KEDA watches events, the operator handles 0  1, and a generated HPA handles 1  N.

1.ScaledObject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-scaler
spec:
  scaleTargetRef:
    name: worker
  minReplicaCount: 0
  maxReplicaCount: 20
  pollingInterval: 30
  cooldownPeriod: 300
  triggers:
  - type: rabbitmq
    metadata:
      queueName: tasks
      mode: QueueLength
      value: "10"   # ~10 msgs per pod
    authenticationRef:
      name: rabbit-auth

2.Authentication

Credentials live in TriggerAuthentication (namespaced) or ClusterTriggerAuthentication (shared), referencing Secrets or pod identity.

ScaledObject authenticationRef TriggerAuthentication secretTargetRef Secret / pod identity Scaler with creds Event source reads thecredentials
Key fieldsscaleTargetRef picks the workload. triggers[] holds one or more event sources. authenticationRef points at the credentials.
ScaledObject defaultspollingInterval 30s, cooldownPeriod 300s, min 0, max 100. Since 2.21, with min above 0 and no idle count, cached metrics or modifiers, KEDA reuses the HPA's own reads: pollingInterval then paces only status.

3.ScaledObject vs ScaledJob

Queue event
ScaledObjectscales existing Deployment
Queue event
ScaledJobnew Job per event, runs to completion

Use ScaledJob for long-running or one-shot tasks.

KindScopePurpose
ScaledObjectNamespacedScale a workload on triggers
ScaledJobNamespacedSpawn Jobs per event
TriggerAuthenticationNamespacedCredentials for a scaler
ClusterTriggerAuthenticationClusterShared credentials
In short : A ScaledObject points KEDA at a workload and its triggers, a TriggerAuthentication holds the credentials, and a ScaledJob spawns a new Job per event instead.

1.Commands

CommandDescriptionExample
helm repo add kedacore https://kedacore.github.io/chartsAdd the official chart repohelm repo update
helm install keda kedacore/keda -n keda --create-namespaceInstall KEDA via Helmkubectl apply --server-side -f keda-2.21.0.yaml
kubectl get pods -n kedaVerify operator, metrics server, webhookskubectl get pods -n keda -w
kubectl get scaledobject -AList ScaledObjects, READY / ACTIVEkubectl get so -n prod
kubectl describe scaledobject NAMEConditions and events when nothing scaleskubectl describe scaledobject worker-scaler
kubectl get hpaKEDA-made HPA, named keda-hpa-NAMEkubectl get hpa keda-hpa-worker-scaler
kubectl annotate scaledobject NAME autoscaling.keda.sh/paused=truePause at current replicas. KEDA deletes its HPA until the annotation goeskubectl annotate scaledobject worker-scaler autoscaling.keda.sh/paused-replicas=0
kubectl logs -n keda deploy/keda-operatorScaler errors (auth, connection)kubectl logs -n keda deploy/keda-operator --tail=50
kubectl get apiservice v1beta1.external.metrics.k8s.ioIs the external metrics API Available ?kubectl get --raw /apis/external.metrics.k8s.io/v1beta1
helm uninstall keda -n kedaRemove KEDA. Delete its objects first, while it runs, or they hang on finalizer.keda.shkubectl delete scaledobjects,scaledjobs --all -A

2.When nothing scales

get scaledobjectREADY / ACTIVE ?
describeconditions and events
keda-operator logsscaler errors (auth, connection)
get apiserviceexternal metrics API Available ?
get hpakeda-hpa-NAME

3.Gotchas & Good to know

Gotchas
  • No second HPA on a scaled Deployment. The webhook refuses a ScaledObject over an existing HPA; a plain HPA made after it is admitted, then both stall with AmbiguousSelector.
  • Activation is strictly greater : threshold 10, activation 50, metric 40 stays at zero.
  • Scale to zero waits cooldownPeriod (300s) after the last active poll, then goes straight from any count to 0, past the HPA's scale-down window.
  • Scale-down can kill pods mid-task. Handle SIGTERM or use ScaledJob.
Good to know
  • Several triggers can be combined with scaling modifiers.
  • HTTP scaling is the separate HTTP Add-on (beta), or use Prometheus request rate.
  • v2.21.0 breaks three things : a boundServiceAccountToken needs its audience mapped in the operator's settings, not the TriggerAuthentication (CVE-2026-77524); azure-pipelines scaleOnInFlight is now true; Temporal loses buildId, selectAllActive and selectUnversioned.
  • A bad TriggerAuthentication shows only as not READY with the source's own error (Redis : NOAUTH). A missing one is only logged; a missing Secret key is not logged at all.
In short : Install with Helm, check READY / ACTIVE on the ScaledObject, and never give the same Deployment a second HPA.

Scaling and metrics

Karpenter

1.What is Karpenter ?

A node autoscaler. It launches nodes for Pods that cannot be scheduled, then removes or replaces nodes that are empty, underused, outdated or too old.

  • SIG Autoscaling subproject. A core library, one provider per cloud.
  • No node groups : it picks instance types per launch, sized from Pod requests.
  • Never scales Pods. An HPA or KEDA adds replicas, Karpenter adds the nodes.
Versions and Kubernetes 1.37
  • v1.14.1 core and AWS provider (2026-08-21), LTS on AWS until Jul 2027. Azure provider v1.14.3.
  • karpenter.sh/v1 since v1.0. Tutorials with Provisioner, AWSNodeTemplate or v1beta1 are out of date.
  • 1.37 is not declared yet : v1.14 covers 1.26 to 1.36, these notes target 1.37. On 1.37 the AWS provider logs "not compatible" once and runs on, unsupported. Expect support in v1.15.

2.From Pending Pod to node

  1. kube-scheduler gives up : PodScheduled=False, Unschedulable.
  2. Karpenter skips Pods with a nodeName, a nomination, or a DaemonSet owner.
  3. Batch : 1s with no new Pod, 10s max.
  4. Biggest Pods first : existing nodes, planned nodes, then NodePools by weight.
  5. Size = requests + DaemonSet requests. Preferences relax one by one.
  6. Node Ready, kube-scheduler binds.
Pending Pod PodScheduled=False Karpenter watches, batches Unschedulable batch : 1s idle, 10s max Simulate nodes, then NodePools create NodeClaim one new machine launch Cloud API AWS : EC2 CreateFleet boots New Node kubelet joins Ready kube-scheduler binds, not Karpenter binds the Pod

3.NodePool, NodeClass, NodeClaim

NodePool = what nodes may look like. NodeClass = how to build them on this cloud. NodeClaim = one machine.

KindHolds
NodePool
karpenter.sh/v1
Requirements, taints, expiry, disruption, limits, weight
EC2NodeClass
karpenter.k8s.aws/v1
AMI, subnets, security groups, IAM role, kubelet
AKSNodeClass
karpenter.azure.com
v1beta1. Image family, OS disk, maxPods
NodeClaim
karpenter.sh/v1
One machine. Spec is immutable
NodePool what nodes may be NodeClass cloud settings nodeClassRef owns N shapes NodeClaim one machine Instance EC2 / Azure VM launches Node registers Node kubelet joins Claim, instance and Node map one to one. Delete the pool and its claims go too.
NodeClaim lifecycle
Launchedinstance up (5 min)
Registeredkubelet in (15 min)
InitializedReady, taints gone
Readyall three true

Miss a timeout and the claim and instance are deleted. Consolidation waits for Initialized.

4.How Karpenter reads a Pod

Pod fieldWhat Karpenter does
requestsSizes the node (containers, init, sidecars, overhead). Limits are ignored
nodeSelector, required affinityA NodePool must allow it, custom labels included, or the Pod stays Pending
preferred affinity, ScheduleAnywayTreated as required, then relaxed one at a time
tolerationsMust tolerate the NodePool taints. startupTaints need no toleration
PVC, WaitForFirstConsumerLaunches in the zone of the StorageClass or bound PV
DaemonSet PodsNever trigger a node on their own. Counted as overhead

Same rules as in Advanced Scheduling : Karpenter simulates them before it buys a node.

In short : Pending Pods in, NodeClaims out. The NodePool says what, the NodeClass says how.

1.Six ways a node goes away

Replacing nodes is the job. Graceful methods can be paused or slowed, forceful ones only get a drain deadline.

MethodFires whenBudget reasondo-not-disrupt
Consolidation gracefulPods fit elsewhere (delete), or elsewhere plus one cheaper node (replace)Empty, UnderutilizedBlocks it
Drift gracefulNodePool template or NodeClass changed, or a new AMI resolvedDriftedBlocks it, if no TGP
Expiration forcefulOlder than expireAfter (default 720h). No wait for a replacementNone, ignoredDoes not stop it
Interruption forcefulAWS Spot warning, health event, instance stopped or terminatedNoneDoes not stop it
Node repair forcefulAlpha. Ready False or Unknown for 30 min. Skipped if over 20% unhealthyNoneDoes not stop it
Manual deletekubectl delete nodeclaim or node, drained through the finalizerNoneDoes not stop it

2.The graceful loop

Pick a method fixed order Budget left ? per pool, per reason yes Simulate Pods fit elsewhere ? Taint the node disrupted:NoSchedule launch new first Replacement Ready fail : untaint, retry Terminate evict, detach, delete acted : go again now. idle : wait 10s
  • Order : Emptiness, static drift, Drift, multi-node, single-node consolidation. One method per pass.
  • Draining uses the Eviction API, so PDBs hold it back.
Consolidation policy
  • WhenEmpty : only nodes whose Pods cost nothing to disrupt (DaemonSet Pods).
  • WhenEmptyOrUnderutilized (default) : any move that lowers cost.
  • Balanced (v1.14+) : scores savings against disruption cost, moves only at score >= 0.5. Stops near-zero-savings churn.
  • consolidateAfter : quiet time after the last Pod change. Never = off.
DriftTemplate and NodeClass hashes sit on each NodeClaim. A new AMI from the selectors drifts too. Changing weight, limits or disruption never drifts. No off switch : pace it with a Drifted budget.

3.What keeps it safe

apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: default }
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["on-demand"]
      expireAfter: 720h           # forceful
      terminationGracePeriod: 48h # deadline
  disruption:
    consolidateAfter: 1m  # required in here
    budgets:
    - nodes: "20%"        # any reason
    - nodes: "0"          # pause
      schedule: "@daily"  # UTC
      duration: 10m
      reasons: ["Underutilized"]
  • Budgets : max nodes disrupted at once per pool (default 10%). By reason, with a UTC cron window.
  • PDBs : every drain evicts, so a PDB can hold it.
  • do-not-disrupt : on a Pod, "true" or a duration like "30m". On a Node, only "true". Blocks graceful methods. Forceful drains skip those Pods until the TGP deadline.
  • terminationGracePeriod : near its end, leftover Pods are deleted, past PDBs and do-not-disrupt. Each still gets its own grace seconds.
launched expireAfter + TGP normal life forceful drain max node life = expireAfter + TGP
TrapexpireAfter with no TGP, plus a do-not-disrupt Pod or a blocking PDB : the node can sit half drained forever. Set both.
In short : Budgets, PDBs and do-not-disrupt slow graceful methods. The TGP caps every drain.

1.Spot and interruptions

  • Several capacity types allowed ? It prefers reserved, then spot, then on-demand.
  • AWS Spot : price-capacity-optimized, up to 60 instance types per CreateFleet call.
  • No capacity ? That type and zone is skipped for 3 min.
  • 2-minute notice : the replacement starts and the drain runs in parallel.
  • Spot nodes are consolidated by deletion only, unless the alpha SpotToSpotConsolidation gate is on.
  • Do not also run AWS Node Termination Handler.
EventBridge EC2 + Health events SQS queue interruptionQueue Karpenter reads the queue CordonAndDrain Spot, stop, scheduled ForcefulTermination status check failed NoAction rebalance : event only No queue set ? Interruption handling is off.
Azure is differentProvider v1.14.3 has no interruption controller. An evicted Spot VM is deleted, then a new node comes. No drain.

2.Karpenter vs Cluster Autoscaler

AspectCluster AutoscalerKarpenter
ScalesPre-built node groups, an expander picks oneSingle instances, types picked per launch
Scale upUnschedulable Pods, scan every 10sUnschedulable Pods, batch 1s idle, 10s max
Scale downUnneeded for 10 min, under 0.5 utilizationConsolidation policy and consolidateAfter (default 0s)
LifecycleAutoscaling onlyAlso expiry, drift (image upgrades), interruption, repair
TimeoutsProvision 15 minLaunch 5 min, registration 15 min
Pod opt-outsafe-to-evict: "false"karpenter.sh/do-not-disrupt
VersionsUse the release that matches the control-plane minorOne release spans a range of minors

Migrating ? The guide scales Cluster Autoscaler to 0 replicas. The FAQ says the two can run side by side.

3.Providers

ProviderNotes
AWS v1.14.1Reference, LTS. Helm, or managed as EKS Auto Mode (336h expiry, 21-day cap)
Azure v1.14.3Helm, or managed as AKS NAP (GA, last announced on v1.14.0). No Windows
Cluster API v0.2.0Experimental proof of concept, create and delete only. See Cluster API
IBM Cloud v1.0.5SIG owned
GCP v0.7.0CloudPilot AI, pre-1.0

Oracle ships an official OCI provider. The core README lists about a dozen more.

4.Gotchas

  • A disruption: block without consolidateAfter is rejected in v1.14. Many doc snippets fail on it.
  • A new DaemonSet never triggers a node. Give it a high, preempting PriorityClass.
  • Agent taints (Cilium's) belong in startupTaints, or Karpenter keeps launching nodes.
  • Editing expireAfter or TGP drifts the whole pool. So does an @latest AMI alias, on every release.
  • Limits count node capacity, not requests, and can overrun in fast scale-outs.

5.Commands

CommandDoes
helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version 1.14.1 -n kube-systemCRDs as their own release. Helm never upgrades a chart's crds/ folder
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version 1.14.1 -n kube-system --set settings.clusterName=$CLUSTER_NAME --set settings.interruptionQueue=$CLUSTER_NAMEController : 2 replicas in kube-system, a PDB, metrics on :8080. Needs two non-Karpenter nodes in different zones
kubectl get nodepools -o wideNODECLASS, NODES, READY. wide adds WEIGHT, CPU, MEMORY
kubectl get nodeclaims -o wideTYPE, CAPACITY, ZONE, NODE, READY. wide adds IMAGEID, ID, NODEPOOL, NODECLASS, DRIFTED
kubectl describe nodeclaim NAMEWhich condition is False, DisruptionBlocked events
kubectl logs -f -n kube-system -l app.kubernetes.io/name=karpenter -c controllerFollow the controller's decisions
kubectl annotate node NODE karpenter.sh/do-not-disrupt=trueShield one node from graceful methods
In short : AWS Spot needs the SQS queue, CRDs get their own chart, 1.37 is not declared yet.

Scaling and metrics

metrics-server

1.What is Metrics Server ?

Metrics Server asks every kubelet for current CPU and memory use and keeps it in memory. It serves the numbers through the Metrics API. That is what makes kubectl top and the HPA work.

  • Scrapes kubelets about every 15s
  • Keeps only the latest sample (no history)
  • Only CPU and memory, for Nodes and Pods
In Simple Words Metrics Server tells Kubernetes how much CPU and memory things use right now.
Built for autoscaling, not for monitoring

2.Why it exists ?

Kubernetes core does not ship a source of live Pod and Node usage.

Scraping every kubelet yourself means each consumer redoes auth, discovery and aggregation. Metrics Server is the reference implementation of the Metrics API.

Without it
  • kubectl top fails : "Metrics API not available"
  • HPA shows <unknown> targets
  • No CPU / memory for autoscaling
  • VPA recommender has no live data

3.Resource Metrics Pipeline

Metrics Server pulls from the kubelet. Consumers only talk to the API server, never to kubelets.

  • cAdvisorReads cgroup stats per container (inside kubelet).
  • kubeletExposes /metrics/resource.
  • APIServicev1beta1.metrics.k8s.io points the aggregator at the Service.
  • Servicemetrics-server in kube-system, port 443.
cAdvisor stats kubelet scrape ~15s metrics-serverlatest sample in memory proxied via APIService kube-apiserver (aggregator) kubectl top HPA / VPA

4.kubectl top pods : Request Flow

GET metrics.k8s.io/v1beta1 pods
kubectl topUser asks
kube-apiserverMatches group metrics.k8s.io
APIServiceForwards to Service
metrics-serverTLS 443, reads store
PodMetricsListcpu, memory, window

If the APIService is Available=False, kubectl top fails even when the Pod is Running.

5.Metrics Server vs Others

ToolAnswersData kept
metrics-serverCPU / memory now ?Latest, in memory
PrometheusBehaviour over time ?Time-series on disk
kube-state-metricsState of objects ?Object state, no usage
Remember
  • top memory = working set, not free or docker stats
  • --metric-resolution = scrape interval (15s)
  • Not a monitoring source : never scrape it with Prometheus
In short : Metrics Server scrapes kubelets, keeps the latest CPU and memory in memory, and serves it as metrics.k8s.io for kubectl top and the HPA. No history, so use Prometheus for that.

1.HPA Uses It

The HPA asks the Metrics API for Pod usage, compares it with the target, then scales the Deployment.

kube-apiserver proxy metrics-server GETmetrics.k8s.io HPA scale Deployment Pods kubeletusage
Key Idea utilization = usage / request
ratio = currentUtil / targetUtil
desired = ceil(replicas x ratio)
Every 15s. Requests are the 100%, so no requests means no %.
Scale-up is immediate. Scale-down waits 5 min (stabilization window) so replicas do not flap.
hpa.yaml (CPU 50%)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

2.Sizing and Versions

Metrics Server0.9.x0.8.x0.7.x0.6.x
Kubernetes1.34+1.31+1.27+1.25+
Default100m CPU, 200MiB (~100 nodes)
Per node over 100+1m CPU, +2MiB
HA2 replicas (high-availability.yaml)

3.Commands

CommandWhat it tells you
kubectl apply -f .../components.yamlInstalls Deployment, Service, APIService
helm install metrics-server metrics-server/metrics-server -n kube-systemSame install, with Helm
kubectl top nodes / kubectl top pods -A --sort-by=memoryLive CPU and memory per Node or Pod
kubectl get apiservice v1beta1.metrics.k8s.ioAvailable=True ? If not, describe it for the reason
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodesRaw Metrics API, bypasses kubectl top
kubectl -n kube-system logs deploy/metrics-serverScrape errors : TLS, no route to node

4.Gotchas

  • x509 errors on kind or minikube : --kubelet-insecure-tls is for testing only. Real fix is kubelet certs signed by the cluster CA.
  • "No route to host" : set --kubelet-preferred-address-types (e.g. InternalIP).
  • Aggregation layer must be on. Kubelet needs webhook authn / authz, else 401 / 403.
  • kubectl get hpa shows <unknown>/50% : Pods have no requests, or Metrics Server is down.
  • New Pods have no metrics for a scrape or two. Memory is working set, so it differs from free.
  • Metrics API is metrics.k8s.io/v1beta1 : confirm with kubectl api-versions | grep metrics.
In short : The HPA divides usage by requests, so set requests. Check the APIService first when kubectl top fails, and never keep --kubelet-insecure-tls outside a lab.

Scaling and metrics

kube-state-metrics

1.What is kube-state-metrics (KSM) ?

A small service that watches the Kubernetes API and turns the state of objects (Pods, Deployments, Nodes...) into Prometheus metrics on /metrics.

  • Metrics come from object state, not usage
  • Read only : needs just list + watch RBAC
  • Stateless Go binary, keeps objects in memory
  • Metrics on :8080, self metrics + /readyz on :8081
NoteKSM never writes to the cluster. It only reads.
Kubernetes API list + watch kube-state-metrics(in-memory store) gauges /metrics :8080 scrape ~30s Prometheus Alertmanager Grafana

2.Why do we need it ?

  • metrics-server gives only live CPU / memory
  • kubectl shows state only right now, no history
  • We want alerts like CrashLooping Pod or PVC Pending

Fix : KSM exposes object state as time series that Prometheus can scrape, graph and alert on. It shows raw API data, so values match the objects.

Key Points
  • Sits in kube-system or monitoring
  • Runs as Deployment + Service + RBAC
  • Pods, Deployments, Nodes, Jobs, PVCs, HPAs...
  • Secrets : metadata only, never the data
  • Ships no CRDs of its own

3.KSM vs metrics-server

Usage vs state. They complement each other, not replace.

kube-state-metricsmetrics-server
AnswersWhat is the object state?How much CPU / memory now?
SourceKubernetes API objectsKubelet metrics endpoint
OutputPrometheus /metricsmetrics.k8s.io API
Used byAlerts, dashboardskubectl top, HPA, VPA
HistoryVia PrometheusNone, latest only
Usage kubelet cpu, mem metrics-server usage kubectl top/ HPA State API server objects kube-state-metrics kube_* Prometheusalerts

4.Sharding and Scale

One instance holds all watched objects in memory, so memory grows with the cluster.

  • --shard=N --total-shards=M, total 1 means off
  • Auto sharding : a StatefulSet, ordinal = shard index
  • Scrape filter : /metrics?resources=pods
  • 2 replicas without sharding = duplicate metrics
API server slice by UID hash shard 0KSM shard 1KSM shard 2KSM one perslice
In short : KSM reads the API and exposes object state as Prometheus metrics. Pair it with metrics-server for usage, and shard it when the cluster grows.

5.Handy Metrics Cheat Sheet

MetricMeaning
kube_pod_status_phasePod phase (Pending / Running / Failed...)
kube_pod_container_status_restarts_totalRestart counter, CrashLoop detection
kube_pod_container_status_waiting_reasonWhy waiting (e.g. CrashLoopBackOff)
kube_deployment_spec_replicasDesired replicas
kube_deployment_status_replicas_availableAvailable replicas (lower than desired = stuck rollout)
kube_node_status_conditionNode Ready, MemoryPressure...
kube_persistentvolumeclaim_status_phasePVC Bound / Pending
kube_*_labelsInfo metric with K8s labels (allowlisted ones only)

6.Gotchas

  • Labels missing ? Set --metric-labels-allowlist (e.g. pods=[app,team])
  • No CPU / memory here, use metrics-server or cAdvisor
  • Custom Resource State is feature-frozen, needs RBAC on the CRD and the custom resource
  • Big cluster ? Use shards, --resources, namespace filters
  • Helm chart lives in prometheus-community, versioned separately
Custom Resource State
kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind:
        group: myteam.io
        version: v1
        kind: Foo
      metrics:
        - name: active_count
          each:
            type: Gauge
            gauge:
              path: [status, uptime]

Passed with --custom-resource-state-config-file.

7.KSM Commands

CommandDescriptionExample
helm install kube-state-metrics prometheus-community/kube-state-metrics -n monitoring --create-namespaceInstall with the prometheus-community charthelm repo add prometheus-community https://prometheus-community.github.io/helm-charts
kubectl apply -f examples/standardInstall from the repo manifests (uses kube-system)git clone https://github.com/kubernetes/kube-state-metrics
kubectl -n monitoring port-forward svc/kube-state-metrics 8080:8080Reach the metrics endpoint locallycurl -s localhost:8080/metrics | grep kube_deployment_status_replicas
curl 'localhost:8080/metrics?resources=pods'Scrape only some resource typescurl -s 'localhost:8080/metrics?resources=pods,secrets' | head
kubectl -n monitoring logs deploy/kube-state-metricsDebug RBAC forbidden errors on list / watchkubectl auth can-i list pods --as=system:serviceaccount:monitoring:kube-state-metrics
kubectl -n monitoring get pod -l app.kubernetes.io/name=kube-state-metricsCheck the pod, /readyz on :8081 is the probecurl -s localhost:8081/readyz
In short : Watch the kube_* metrics for restarts, replicas and PVC phase, tune labels and shards in the flags, and use the commands above to install and debug KSM.

Workload identity

SPIFFE

1.Why SPIFFE ?

SPIFFE is an open standard, not software. Every workload gets a name (SPIFFE ID) and a short-lived ID card (SVID). Services prove who they are without static secrets or IP addresses.

  • Problem : how does service A know the caller is really service B ?
  • IPs change and Pods are short-lived, so network identity breaks
  • Static secrets (API keys, long-lived certs) leak and are rarely rotated
  • Fix : one name format, one document format, one local API
  • Zero trust : authenticate by cryptographic identity, not by network position
Before Service A API key in Secret leaked, copied, never rotated With SPIFFE Service A SVID (short TTL) rotated automatically identity = what you can prove, not where you sit (IP, network)

2.The Pieces

SPIFFE is the spec. SPIRE is the main implementation of it. Do not mix them up. Both are CNCF graduated.

SPIFFEthe spec, no tagged release
SPIREimplementation, v1.15.5
  • CSI driver : spiffe-csi v0.3.0 (the spire 0.30.4 chart still installs 0.2.7)
  • Libraries : go-spiffe, java-spiffe, spiffe-helper
Key Terms
  • SPIFFE IDThe name. spiffe://<domain>/<path>, path meaning is up to the admin.
  • Trust domainIdentity boundary and root of trust. Lowercase, max 255 bytes, self-registered.
  • SVIDThe document carrying the ID : X.509, JWT or WIT.
  • Trust bundleJWKS of a domain's public keys.
  • Workload APILocal gRPC API, Unix socket, no auth handshake.

3.Anatomy of an Identity

Scheme + trust domain + path = SPIFFE ID. No query, no fragment. Path /ns/<ns>/sa/<sa> is a Kubernetes convention, not required.

spiffe:// prod.example.org /ns/payments/sa/api scheme trust domain path whole ID X.509-SVIDsingle URI SAN JWT-SVIDsub claim verify with bundle
X.509-SVIDJWT-SVIDWIT-SVID
FormatCertificateSigned tokenJWS compact, WIMSE
ID inURI SANsubsub
ReplayLow, proves keyBearer, until expLow, key proof
Best formTLSL7 hops, proxiesWIMSE use cases
Stable ?YesYesIncubating
CheckChain to bundleAudience + expPossession proof
Name vs DocumentThe SPIFFE ID is only a name. The SVID is the signed document that carries it, and the trust bundle is what a peer checks it against.
ID = who, SVID = proof, bundle = whom to trust
In short : SPIFFE gives every workload a name (SPIFFE ID) and a short-lived, auto-rotated ID card (SVID). SPIRE is the implementation.

1.Getting an Identity

The Pod never holds a secret to ask for its identity. The platform vouches for it.

Workload API flow
PodCalls FetchX509SVID, sends no credentials.
Workload API socketUnix domain socket.
Agent (e.g. SPIRE)Identifies the caller out of band (PID, cgroup, kubelet info).
Control plane / CAMatches registered identities, signs SVID.

Agent streams SVID + bundle back to the Pod and pushes again on rotation.

2.mTLS and Federation

Two workloads present X.509-SVIDs and check each other with the right trust bundle. Different trust domains swap bundles at a bundle endpoint.

  • https_web : Web PKI
  • https_spiffe : authenticated with an SVID
  • Refresh on spiffe_refresh_hint, the producer's interval in seconds (5 min if it is absent, and SPIRE's default)
  • Short TTL means few revocation lists
CarefulNever merge bundles from different trust domains.
Trust domain A own keys, own root Workload A verify B with bundle B Bundle Aendpoint Trust domain B own keys, own root Workload B verify A with bundle A Bundle Bendpoint mTLS X.509-SVIDs swap bundles

3.In a Pod

The spiffe-csi driver (csi.spiffe.io) mounts the socket read-only, so the Pod needs no hostPath. Libraries find the socket through SPIFFE_ENDPOINT_SOCKET.

volumeMounts:
- name: spiffe-workload-api
  mountPath: /spiffe-workload-api
  readOnly: true
env:
- name: SPIFFE_ENDPOINT_SOCKET
  value: unix:///spiffe-workload-api/spire-agent.sock
volumes:
- name: spiffe-workload-api
  csi:
    driver: "csi.spiffe.io"
    readOnly: true
Node spire-agent socket dir spiffe-csi shared hostPath (agent + CSI) mounts Pod /spiffe-workload-api spire-agent.sock SPIFFE_ENDPOINT_SOCKET read-only socket, no Secret mounted
In short : The Pod asks the local Workload API for its SVID with no credentials, peers verify X.509-SVIDs with trust bundles, and spiffe-csi mounts the socket so the Pod needs no hostPath.

1.Who is Who

SPIFFE is the spec. Other tools implement it, reuse its IDs, or get replaced by it.

ThingIssues SVIDs ?Attests ?
SPIREYesYes, node + workload
Istio / ConsulYes, own CAPartial (SA / ACL token)
cert-managerSPIFFE-style certsNo, policy only
Pod CertificatesIf its signer puts the ID inAPI server checks node, Pod, SA
SA tokenNo, K8s API JWTNo
Static secretNoNo, holder wins
SPIFFEID, SVID, API SPIREimplements Istio / Consuluse SPIFFE IDs cert-managerno attestation Static secretwhat it replaces SPIFFE-style replaces

2.Commands

CommandDescriptionExample
kubectl apply -f csidriver.yamlRegister the spiffe-csi driverkubectl get csidriver csi.spiffe.io
kubectl exec <pod> -- ls /spiffe-workload-apiCheck the socket is mountedshows spire-agent.sock
kubectl exec <pod> -- env | grep SPIFFELibraries find the socket hereSPIFFE_ENDPOINT_SOCKET=unix:///...
spire-agent api fetch x509 -socketPath <sock>Fetch an X.509-SVID like a workload (SPIRE CLI)socket path from above
openssl x509 -in svid.pem -noout -textInspect the URI SANURI:spiffe://prod.example.org/ns/payments/sa/api
kubectl logs -n <ns> ds/spiffe-csi-driverDebug mount failures (name depends on install)add -c spiffe-csi-driver
Debug order
CSIDriverregistered ?
Socketmounted ?
Envvar set ?
FetchSVID back ?
URI SANright ID ?
stop at the
first ? that fails

3.Gotchas

The model
  • SPIFFE is a spec, you install an implementation (SPIRE)
  • SPIFFE has no CRDs. ClusterSPIFFEID comes from spire-controller-manager (check apiVersion first)
  • SPIFFE ID is identity, not authorization. You still need policy
  • Trust domains are lowercase and self-registered, so use one you own
  • Write the trust domain in lowercase (the spec says MUST). Scheme and trust domain compare case-insensitively, the path case-sensitively
In the cluster
  • Share the socket dir by hostPath between agent and CSI driver, or a driver restart breaks mounts
  • The CSI driver itself needs a privileged namespace (hostPath). Pods that mount its volume still pass restricted
  • Broken CSI driver means Pods fail to terminate fully
  • JWT-SVIDs are bearer tokens : check audience, keep TTL short
  • Clock skew or failed rotation shows up as sudden TLS errors
In short : Debug from the socket outward (driver, mount, env, fetch, URI SAN). SPIFFE is identity, not authorization, and needs an implementation such as SPIRE.

Workload identity

SPIRE

1.Why SPIRE ?

Pods talk to each other, but how does Pod A prove it is Pod A ? IPs change, so an IP is a bad identity.

  • Static secrets (API keys, long-lived certs) get leaked, copied and never rotated
  • Service Account tokens prove identity only to the k8s API, not across clusters or clouds
  • SPIRE checks facts (node, namespace, SA, image) then issues a short-lived SVID
  • Result : zero-trust mTLS with no secret to store

SPIFFE = the standard. SPIRE = the software. CNCF Graduated (TOC vote 2022-08-23, announced 2022-09-20).

Static secret Long-lived key leaks, copied,never rotated With SPIRE Workload SVID (short) auto-rotated, used for mTLS SVID lifetime (SPIRE defaults) X.509 SVID : 1h, agent renews at half-life JWT SVID : 5 min, minted per audience hardened Helm chart : X.509 4h, JWT 1h

2.Components

The hardened chart's recommended layout puts the server in spire-server and the agents in spire-system. One server, one agent per node.

  • Node attestor proves the node (options in 3.)
  • Workload attestor inspects the caller (k8s via kubelet, unix via uid/gid)
  • SVID = X.509 cert (mTLS) or JWT (bearer calls)
  • Trust domain = the security boundary, its trust bundle holds the CA certs
Pieces
  • SPIRE ServerSigning authority. Holds entries, issues SVIDs. Usually a StatefulSet.
  • SPIRE AgentDaemonSet. Attests to the server, then attests local workloads.
  • Workload APIUnix socket where a workload asks "who am I ?".
  • controller-managerTurns CRDs into registration entries.
  • SPIFFE CSI driverMounts the agent socket into Pods.

3.Attestation and SVID Flow

Two steps : the agent proves the node, then identifies the process.

Server Agent Pod 1 node attest 2 give me SVID kubelet 3 which pod ? 4 request SVID 5 signed SVID 6 SVID + bundle dashed = done ahead of time, cached 7 renewed at half-life, before expiry
Node attestorProves withCatch
k8s_psatProjected SA tokenDefault on k8s
aws_iidEC2 identity docAWS only
gcp_iitGCE identity tokenGCP only
azure_msiManaged identityAzure only
join_tokenOne-time tokenManual, tests
x509popExisting node certNeeds your PKI

one per agent, the server must run the same plugin

In short : SPIRE attests the node, then the workload, and hands out short-lived SVIDs for mTLS. No static secret to leak.

1.Registration Entries

Entry = SPIFFE ID + parent ID + selectors. The parent ID points at the agent allowed to hand it out.

ID looks like spiffe://trust-domain/path

Kind (cluster scope)Use
ClusterSPIFFEIDWhich pods get which ID
ClusterFederatedTrustDomainFederate with another trust domain
ClusterStaticEntryHand-written entry for non-pod things

all spire.spiffe.io/v1alpha1

Server Agent (node entry) webk8s:ns:defaultk8s:sa:web dbk8s:ns:datak8s:sa:db web + db parent ID = the agent's ID spiffe://td/spire/agent/k8s_psat/<cluster>/<node>

2.k8s Automation

spire-controller-manager is a reconcile loop, like a Deployment controller, but the output is SPIRE entries.

Controller manager flow
ClusterSPIFFEIDYou apply the CRD.
controllerWatches Pods and the CRD.
Server APIEntries created, updated, deleted.
Pod deletedIts entry is removed.
kind: ClusterSPIFFEID
metadata: {name: web}
spec:
  className: spire-mgmt-spire
  spiffeIDTemplate: "spiffe://{{ .TrustDomain }}\
    /ns/{{ .PodMeta.Namespace }}\
    /sa/{{ .PodSpec.ServiceAccountName }}"
  podSelector:
    matchLabels: {app: web}
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: default
Reading it apiVersion: spire.spiffe.io/v1alpha1. Pods labelled app: web in default get an ID built from their Namespace and ServiceAccount. className names the chart's controller manager (release namespace, then release): without it, the CR is ignored. A workload fetches it over the Workload API socket, no credentials needed.
Resulting IDServiceAccount web in default gets
spiffe://<trust-domain>/ns/default/sa/web
no Pod IP
In short : Entries map selectors to SPIFFE IDs under a parent agent. ClusterSPIFFEID CRDs let the controller manager write them for you.

1.Install and Inspect

Two Helm charts from the hardened repo, CRDs first, then server + agent + controller manager, both released into spire-mgmt.

spire-crdsCRDs chart, release in spire-mgmt
spireServer, agent, controller manager
get podsIn spire-server and spire-system
entry showEntries exist
CommandDescription
helm upgrade --install --create-namespace -n spire-mgmt spire-crds spire-crds --repo https://spiffe.github.io/helm-charts-hardened/Install the CRDs first (release namespace: spire-mgmt)
helm upgrade --install -n spire-mgmt spire spire --repo https://spiffe.github.io/helm-charts-hardened/ -f values.yamlServer, agent, controller manager. The values set the trust domain, the cluster name and global.spire.recommendations.enabled
kubectl get pods -n spire-serverServer and controller manager (agents and CSI driver: -n spire-system)
kubectl exec -n spire-server spire-server-0 -c spire-server -- spire-server entry showList registration entries
spire-server entry create -spiffeID <id> -parentID <id> -selector <sel>Create an entry by hand, e.g. -selector k8s:ns:default
spire-server entry delete -entryID <id>Delete an entry
kubectl get clusterspiffeidList ClusterSPIFFEID objects
kubectl logs -n spire-system -l app.kubernetes.io/name=agentAgent logs when attestation fails

2.No SVID ? Debug Order

agent logsAttest errors ?
entry showEntry exists ?
selectorsMatch ns / SA ?
parent IDNode's agent ?
first "no"
= the bug

3.Gotchas

Setup
  • Install the CRDs before the main chart (separate spire-crds chart)
  • Server needs persistent storage (SQLite by default) or entries and CA state are lost, prod usually wants external SQL
  • Charts are beta and the controller API is v1alpha1, use released versions not git main
  • Recheck versions before quoting (latest seen : v1.15.5, chart spire 0.30.4)
Runtime
  • A typo in a selector like k8s:ns means silently no SVID
  • Parent ID must match the agent that really runs the workload
  • SVIDs are short-lived, apps must reload certs via the Workload API or a library
  • Not a secret store, that is Vault or External Secrets
In short : Install spire-crds, then spire. Check entries with entry show, and when no SVID shows up, look at selectors, parent IDs and the agent logs.

Networking and traffic

MetalLB

1.Why MetalLB ?

On bare metal a LoadBalancer Service stays EXTERNAL-IP <pending> forever: no cloud controller hands out IPs.

  • NodePort is ugly (high ports, clients must know node IPs)
  • Cloud LBs (ELB, GCP LB) are not available on-prem, in homelabs or at the edge
  • MetalLB = IP allocation from your pools + announcement to the network (L2 or BGP)

Fills in type: LoadBalancer, the Service type after NodePort.

Without MetalLB Service LB EXTERNAL-IP<pending> With MetalLB Service LB 192.168.10.5 announced to LAN or routers

2.Components

Installs into metallb-system, idle until you create a pool and an advertisement.

  • Two jobs : assign IPs, then advertise them
  • Works with IPv4, IPv6 and dual stack
  • Config is Custom Resources only
  • controllerDeployment. Watches Services, picks an IP from an IPAddressPool, writes it to the Service status.
  • speakerDaemonSet. Announces the IP via ARP/NDP or BGP. Uses memberlist (7946 TCP/UDP) to spot dead nodes.
  • frr-k8sDefault BGP backend. Wrapper around FRR, adds BFD and IPv6.

3.L2 Mode Flow

Controller allocates, speaker announces. Only the leader node answers ARP/NDP.

Allocation and announcement (L2)
kubectl applyService type=LoadBalancer
API serverSends a watch event.
controllerPicks a free IP from the pool, sets status.
speaker (leader)Answers ARP/NDP for the IP.
Then traffic :
ClientsHit the IP.
LeaderGets all traffic for that IP.
One leader per IP, chosen via memberlist. If it dies a new one wins, brief gap.

4.BGP Mode

Every node peers with your routers and announces the Service IP. Traffic is spread across nodes (ECMP).

  • Needs a BGP-capable router
  • Backends : FRR-K8s (default), native, legacy FRR (deprecated)
Node 1 Node 2 Node 3 BGP session Router Client flows spread (ECMP)
AspectL2 (ARP/NDP)BGP
NeedsSame LANBGP router
Load balancingNo, one node per IPYes (ECMP)
FailoverNew leader, brief gapRoute withdrawn, BFD speeds it up
SetupVery lowHigher (ASNs, peers)
UseHomelab, small LANDatacenter, larger scale
L2 = failover only, one node takes it all.
BGP = real load spreading across nodes.
In short : Controller allocates an IP from your pool, speaker announces it via L2 (one leader, failover) or BGP (ECMP, needs routers).

1.Config Objects (CRDs)

Before v0.13.0 config was a ConfigMap. Now it is CRDs in metallb-system. Convert old config with quay.io/metallb/configmaptocrs.

KindapiVersionPurpose
IPAddressPoolv1beta1IP ranges to allocate
L2Advertisementv1beta1Announce via ARP/NDP
BGPPeerv1beta2Router to peer with
BGPAdvertisementv1beta1Announce via BGP
BFDProfilev1beta1BFD timers
L2Advertisement BGPAdvertisement ipAddressPools selects the pool IPAddressPool Service picks a pool peers (opt.) BFDProfile BGPPeer bfdProfile (opt.)
kind: IPAddressPool
metadata: {name: first-pool, namespace: metallb-system}
spec:
  addresses: [192.168.10.0/24]
---
kind: L2Advertisement
metadata: {name: example, namespace: metallb-system}
spec:
  ipAddressPools: [first-pool]
Pool and advert
  • Both use apiVersion: metallb.io/v1beta1
  • A Service can pick a pool with the annotation metallb.io/address-pool
  • A pool with no advertisement is unreachable
  • An L2Advertisement with no ipAddressPools announces every pool

2.Commands

CommandDescription
kubectl apply -f metallb-frr-k8s.yamlInstall, FRR-K8s backend (recommended). metallb-native.yaml is L2 only
helm install metallb metallb/metallb -n metallb-system --create-namespaceInstall with Helm, after helm repo add
kubectl get pods -n metallb-systemCheck controller and speaker
kubectl get ipaddresspools -n metallb-systemList the pools
kubectl logs -n metallb-system ds/speakerDebug announcements and leader election
kubectl describe svc <name>Events show IP and announcing node

3.Gotchas

  • kube-proxy in IPVS mode needs ipvs.strictARP: true or L2 misbehaves (ipvs is deprecated since v1.35)
  • Install alone does nothing: no pool, no advertisement, no IP
  • Pool IPs must be unused on your LAN (outside the DHCP range)
  • Port 7946 (TCP/UDP) must be open between nodes for memberlist
  • L2 is failover, not load balancing. Legacy FRR mode is deprecated
In short : A pool plus an advertisement is all MetalLB needs. Check pods, pools and speaker logs when an IP stays pending.

Networking and traffic

Gateway API CRDs

1.What is the CRD Bundle ?

One YAML file per release of kubernetes-sigs/gateway-api. You apply it once and the cluster learns the Gateway API kinds.

  • Defines the kinds, their schema and CEL validation
  • Cluster-scoped : one copy, shared by every controller
  • Versions apart from Kubernetes core, latest v1.6.3
  • Each CRD is annotated with its bundle-version and channel
Key ideaThey are only the API. A controller (Envoy Gateway, kgateway, Istio, Cilium...) makes them real.
gateway-api repo kubectl apply CRD bundleschema, CEL, v1.6.3 watched by Controllervendor chart, own release programs Proxy (data plane) one percluster one ormany
Why Gateway API ? Ingress vs Gateway API, the role model, request flow and YAML examples are in the Gateway API section. This topic is only the CRDs.

2.Standard vs Experimental

Two release channels, same version number. Experimental is a superset of Standard.

  • Standard : only v1 kinds and fields
  • Experimental : adds alpha fields, plus kinds in group gateway.networking.x-k8s.io
Note : do not mix channels or versions across controllers. The CRDs are cluster-scoped, so every controller sees the same copy.
StandardExperimental
StableNo breaking changes in a versionMay break any release
Filestandard-installexperimental-install
UseProductionTrying features
RollbackSafe (pin version)Can fail if objects use experimental fields

3.Versions and Graduation

KindRolePoints to
GatewayClassNames the controllercontroller
GatewayListeners, the LBGatewayClass
HTTPRouteL7 host, path, headerGateway
GRPCRouteL7 gRPC methodGateway
BackendTLSPolicyTLS to backendService
TLSRouteRoute by SNIGateway
ReferenceGrantAllows cross-ns refsits own ns
ListenerSetExtra listenersGateway
TCPRoute / UDPRouteL4 port routingGateway

Only GatewayClass is cluster-scoped, every other kind lives in a namespace.

v1.0 GatewayClass, GatewayHTTPRoute v1.1 GRPCRoute v1.4 BackendTLSPolicy v1.5 TLSRoute, ReferenceGrantListenerSet v1.6 TCPRoute, UDPRoute kinds graduate into Standard as v1
Upgrade note : a graduated kind keeps its old version (v1alpha2) for a few releases. Move objects to v1 before it is removed.
In short : The CRD bundle is only the API : one cluster-wide copy per channel and version, kinds graduate to v1 over releases, and a separate controller does the work.

1.Install : CRDs vs Controller

One kubectl apply registers the CRDs. The controller, installed apart, does the real work.

  • Always apply with --server-side
  • Install the CRDs first, then the controller
  • Without a controller objects are stored, status stays empty
  • Controller writes Accepted and Programmed to status
kubectl apply registers API serverCRDs registered Gateway, Routes stored watches Controller writes statusconditions
Note : check with kubectl get gatewayclass, ACCEPTED = a controller owns it.

2.Upgrade and Uninstall

An upgrade is the same apply with a newer file. The risk is what the new version drops.

  • A removed version (v1alpha2) must not be in storedVersions
  • New required fields can reject old objects
  • The controller must support the new bundle version
Upgrade Flow
Checkannotation
Read notesremovals
Applyserver side
Verifystatus
Uninstall : kubectl delete -f with the same file. Deleting a CRD deletes every object of that kind, so every Gateway and Route goes with it.

3.Commands

CommandDescriptionExample
kubectl apply --server-side -f standard-install.yamlInstall or upgradepin the v1.6.3 URL
kubectl get crd | grep gateway.networkingList the CRDsk8s.io, x-k8s.io
kubectl get crd gateways.gateway.networking.k8s.io -o yaml | grep bundleBundle versionv1.6.3, standard
kubectl get crd tcproutes.gateway.networking.k8s.io -o jsonpath='{.status.storedVersions}'Versions in etcdlists v1alpha2 = migrate first
kubectl apply --server-side --force-conflicts -f <file>Take over fieldsafter Helm owned them
kubectl delete -f <same file>Uninstall the CRDsdrops all Routes too

4.Helm-Managed CRDs

Many controller charts ship the Gateway API CRDs. Helm treats them differently by folder.

Helm actioncrds/ foldertemplates/
installCreatedCreated
upgradeSkipped (stale)Updated
uninstallKeptDeleted, with all objects
Rule : one owner for the CRDs per cluster : the platform, one chart, or kubectl. Never two.
Gotchas
  • metadata.annotations: Too long = client-side apply, add --server-side
  • Two charts, two bundle versions : every upgrade flips the CRDs
  • One chart's experimental CRDs switch the channel for every controller
  • Old admission webhook left over : remove it, CEL replaced it
  • templates/ CRDs : add resource-policy: keep
  • v1.6.0 : ReferenceGrant spec now required
In short : Install the CRDs once with server-side apply, pick one channel and one owner, check storedVersions before upgrades, and let a controller do the rest.

Networking and traffic

Gateway API

Gateway API : the role-based, portable successor to Ingress

1.Why Gateway API ?

Typed routing objects, split by who owns what. This topic is how routing behaves. The CRDs, channels and install are in the Gateway API CRDs section.

  • Match header, query, method, not just host + path
  • Redirect, rewrite, mirror : typed filters
  • Canary is a weight on backendRefs
  • Timeouts per rule, TLS Terminate or Passthrough
  • ingress-nginx was archived 2026-03-24, no more fixes

Ingress is still supported but frozen. Gateway API is the upstream successor (SIG Network).

Ingress one objecteveryone edits annotationsper-vendor dialect Gateway API GatewayClass : infra provider Gateway : cluster operator Routes : app dev
Why roles ?Splitting by owner means a dev can ship a Route without touching the shared Gateway.

2.Resource Model

Each role edits its own object. parentRefs / gatewayClassName point up to the parent; backendRefs point down to the backend.

FieldOnPoints to
controllerNameGatewayClassController (up)
gatewayClassNameGatewayGatewayClass (up)
parentRefsRouteGateway (up)
backendRefsRoute ruleService (down)
certificateRefsListenerSecret (TLS)
hostnamesRouteListener hostname

GatewayClass is cluster scoped, everything else is namespaced. A Route can list several parentRefs (two Gateways, or one listener by sectionName). Route and Listener hostnames must overlap.

GatewayClasscluster scoped Controller controllerName Gatewayns infra gatewayClassName HTTPRoutens app-a HTTPRoutens app-b parentRefs Service app-a Service app-b backendRefs
Just a spec CRDs are not a controller. You also need one (Envoy Gateway, kgateway, Istio, Cilium, NGINX Gateway Fabric...).

3.Request Flow

The Gateway status gives the address. The listener handles TLS, then a Route rule picks the backend.

ModeWho decryptsRoute
TerminateGateway, cert in certificateRefsHTTPRoute, GRPCRoute
PassthroughBackend, Gateway reads SNI onlyTLSRoute
Client to Pods
Client
DNS / LB IPaddresses
Listener :443TLS
Pods
Serviceweighted refs
HTTPRoutehost, path, header
Cert note : a Secret in another namespace needs a ReferenceGrant. Gateway status.addresses = the LB IP clients hit.
In short : Gateway API is a spec plus CRDs. Infra owns GatewayClass, ops owns Gateway, devs own Routes, and a controller does the work.

1.Matching and Filters

A rule = matches + filters + backendRefs, checked in that order.

FieldExampleNote
matches
pathPathPrefix /apiExact, Prefix, Regex
headersenv = canaryExact (Core), regex (impl.)
queryParamsv = 2Exact (Extended)
methodPOSTExtended
filters
RequestHeaderModifierset, add, removeCore filter
RequestRedirectto https, 301Core, no backend
URLRewrite/v1/x to /xprefix or full path
RequestMirrorcopy to shadowreply dropped
rule
timeoutsrequest: 10sbackendRequest per try
Request matchesAND inside, OR across filtersrun in listed order backendRefs + weight
  • Tie break : Exact path, longest prefix, method, most headers
  • No rule matches : 404

2.Gateway + Canary Route

kind: Gateway
metadata: {name: my-gw, namespace: infra}
spec:
  gatewayClassName: my-class
  listeners:
  - {name: http, port: 80, protocol: HTTP,
     allowedRoutes: {namespaces: {from: All}}}
---
kind: HTTPRoute
metadata: {name: web, namespace: app}
spec:
  parentRefs: [{name: my-gw, namespace: infra}]
  rules:
  - backendRefs:
    - {name: web-svc, weight: 90}
    - {name: web-canary, weight: 10}
  1. Operator creates the Gateway
  2. Controller sets status.addresses
  3. Dev creates the HTTPRoute in ns app
  4. allowedRoutes must permit ns app
  5. Controller sets Accepted / ResolvedRefs
weight 90 / 10web-svc / web-canary
Canary = weights. Devs ship it without touching the Gateway.
Attach rules A Route finds its Gateway with parentRefs. The Gateway must allow it with allowedRoutes (default : same ns).

3.Status and ReferenceGrant

OnConditionReasons when False
GatewayAcceptedListenersNotValid
GatewayProgrammedPending, AddressNotAssigned
ListenerResolvedRefsInvalidCertificateRef
RouteAcceptedNotAllowedByListeners
RouteResolvedRefsBackendNotFound, RefNotPermitted
ns app HTTPRoute ns db Service refs ReferenceGrantfrom: app Routesto: Service allows Grant lives in the TARGET namespace. Missing : RefNotPermitted
CommandShows
kubectl describe gateway my-gw -n infraListener conditions, attachedRoutes
kubectl describe httproute web -n appstatus.parents per Gateway
kubectl get referencegrant -AWho may point across ns
ingress2gateway printConvert Ingress to Gateway YAML
In short : A rule matches, filters, then picks weighted backends. Cross-namespace refs need a ReferenceGrant; when it breaks, read Gateway AND Route conditions.

Networking and traffic

kgateway

1.What is kgateway ?

kgateway is a control plane that watches Gateway API resources and configures Envoy proxies to route traffic into your cluster.

Think "Ingress, but modern, with Envoy doing the work".

  • Formerly Gloo Gateway (OSS), made by Solo.io
  • CNCF Sandbox project (accepted 2025-03-04)
  • Control plane is Go, talks to Envoy over xDS
  • Default GatewayClass : kgateway
  • Latest stable seen : v2.4.7
Request flow Client LoadBalancer Service Envoy proxy Pod(one per Gateway) Service / Endpoints App Pods

2.kgateway on Gateway API

Why Gateway API, and who owns what : the Gateway API section. Here is what kgateway does with each object.

GatewayCreating one makes the controller deploy an Envoy Deployment plus a LoadBalancer Service. Tune the proxy with GatewayParameters.
Objectkgateway does
GatewayClassClass kgateway, made by the chart
ListenerEnvoy listener : port, protocol, TLS
HTTPRouteEnvoy routes, pushed over xDS
TrafficPolicyExtras, attached by targetRefs
ProtocolsHTTP, gRPC, TCP/TLS

3.Control Plane vs Data Plane

kubectlor GitOps
API serverstores objects
controllerwatch, translate
EnvoyxDS config

status flows back to the Gateway (Accepted, Programmed)

  • ControllerPod in kgateway-system.
  • EnvoyAuto-created per Gateway. Applies routes and policies.
In short : The kgateway controller turns Gateway API objects into Envoy config, and Envoy carries the traffic.

4.kgateway CRDs

All in group gateway.kgateway.dev, version v1alpha1.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: httpbin}
spec:
  parentRefs: [{name: http}]
  hostnames: ["www.example.com"]
  rules:
  - backendRefs:
    - {name: httpbin, port: 8000}
KindUse it for
GatewayParametersCustomize generated proxy
TrafficPolicyAuth, rate limit, CORS, retry, JWT
BackendLambda, static, GCP upstreams
BackendConfigPolicyUpstream TLS, health checks
HTTPListenerPolicyListener settings, access logs
DirectResponseFixed reply, no backend
GatewayExtensionReusable ext auth config

5.Handy Commands

CommandDescriptionExample
helm upgrade -i kgateway-crdsInstall kgateway CRDs chartGateway API CRDs firstGateway API CRDs first
helm upgrade -i kgatewayInstall the control planeget pods -n kgateway-systemget pods -n kgateway-system
kubectl get gatewayclassConfirm class kgateway existsget gatewayclass kgatewayget gatewayclass kgateway
kubectl get gateway,httproute -ACheck Gateways and routesget gateway http -n kgateway-systemget gateway http -n kgateway-system
kubectl get svc,deploy -n <ns>Find Envoy Deploy + Svcget svc -n kgateway-systemget svc -n kgateway-system
kubectl describe httproute <name>Read route statusdescribe httproute httpbindescribe httproute httpbin
kubectl get trafficpolicy -AList policiesget trafficpolicy -Aget trafficpolicy -A
Charts : <chart> is oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds (and .../kgateway), with -n kgateway-system --version v2.4.7. Add --create-namespace on the first install.

6.Gotchas

  • Spell the class kgateway exactly or nothing programs.
  • Install Gateway API CRDs first.
  • Do not use ports 19000, 8082, 9091 for listeners.
  • Own CRDs are v1alpha1, so APIs can change.
  • Cross namespace backendRefs need a ReferenceGrant.
  • Route stuck ? Read HTTPRoute status first.
Good to know
  • Policies attach by targetRefs or label targetSelectors.
  • TLS certs are Secrets (cert-manager can issue them).
  • agentgateway is a separate Rust data plane for AI / MCP traffic.
  • Quickstart chart version can lag the newest release.
In short : Install the CRDs and chart, check the GatewayClass and route status, and keep an eye on those v1alpha1 policies.

Serverless

Knative Operator

Knative Operator : install Knative with two small YAML files

1.What is the Knative Operator ?

A controller that installs, upgrades and configures Knative Serving and Eventing for you.

  • You write a KnativeServing or KnativeEventing CR
  • The operator reconciles it into a full Knative
  • Runs as two Deployments, knative-operator and operator-webhook, in ns knative-operator (helm --create-namespace)
  • Part of the Knative project (CNCF graduated, TOC vote 2025-09-11)

Tip : a hand edit to a ConfigMap key the CR sets is reverted; a key the CR does not set stays. Edit the CR instead.

You (kubectl/Flux) apply KnativeServing CR watches knative-operator uses bundled Knativemanifests creates knative-serving nsDeploys, ConfigMaps, RBAC statusReady

2.Why do we need it ?

Manual install hurts
  • Many big manifests, right order
  • Upgrades by hand, drift creeps in
  • ConfigMaps get edited ad hoc
  • HA and limits patch many Deployments

Fix : declare Knative in one CR. GitOps friendly.

Remember

Operator = one controller that owns every Knative Deployment, ConfigMap and RBAC rule it installs.

TopicOperator (CR)Manual
Installoperator + 1 CRCRDs, core, ingress
Upgradechange spec.versionre-apply, fix drift
Configspec.configedit ConfigMaps
HA, limitsspec.workloadspatch each Deployment
Driftoperator reverts its keysstays drifted
Extra partyes, the operatorno

Operator itself ?

Upgrade it with its Helm chart first. It knows its own Knative minor and the three before it.

3.The Two CRDs

Both use operator.knative.dev/v1beta1. CR is named like its namespace.

KindNamespaceControls
KnativeServingservingServices, Revisions, autoscaler
KnativeEventingeventingBrokers, Triggers, Channels
Remember

Serving answers HTTP requests. Eventing moves events to your code. Install either one, or both.

knative-operator ns knative-serving ns knative-eventing ns controller,webhook,activator,autoscaler deliversevents Your Service / Broker
In short : The operator owns every Knative Deployment, ConfigMap and RBAC rule; you only write a KnativeServing or KnativeEventing CR.

1.Key spec fields

  • versionpins Knative, change it to upgrade
  • config.<map>keys go into Knative ConfigMaps
  • ingressistio, kourier, contour or gateway-api
  • high-availabilityreplicas for control plane
  • workloads / servicesreplicas, resources, nodeSelector, env
  • registryimage rewrite, imagePullSecrets (air-gapped)

All of it lives under spec.

KnativeServing CR
apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  version: "1.24"
  ingress:
    kourier:
      enabled: true
  config:
    network:
      ingress-class:
        kourier.ingress.networking.knative.dev
  high-availability:
    replicas: 3

2.Networking layer

Layerspec keyNote
Istioistio.enabledinstall Istio first
Kourierkourier.enabledlight, set class
Gateway APIgateway-apiCRDs + Gateway
ContourcontourEnvoy based

Tip : turn on one networking layer at a time; the operator does not enforce it.

3.Upgrades

Upgrade Flow
version "1.23"running
edit to "1.24"git commit / apply
rolloutby operator, then Ready=True

One minor at a time: a skip sets VersionMigrationEligible False. One minor down passes.

You edit one line, the operator does the rollout.

4.Handy Commands

CommandWhat it does
helm install knative-operatorInstall the operator (add --create-namespace)
kubectl get knativeserving -o yamlstatus.version and conditions (the list shows only TARGET CLUSTER)
kubectl logs -f deploy/knative-operatorFirst stop if not Ready
kubectl describe knativeservingSee status.conditions
Gotchas
  • Create the namespace first
  • Never hand-edit Knative ConfigMaps
  • No networking layer means no routes
  • Delete CRs before removing the operator
  • Operator is one more thing to upgrade
  • Keep the CR in ns knative-serving
  • Ready Unknown, IngressNotConfigured ? no ingress yet
  • spec.version must be one it ships
Quick Q and A

Upgrade ? bump spec.version

HA ? spec.high-availability.replicas

Serving ? request-driven apps, scale to zero

Eventing ? Brokers and Triggers

In short : Knative Operator turns a KnativeServing or KnativeEventing CR into a running Knative, and keeps it that way.

Serverless

Knative

1.What is Knative ?

Knative adds serverless behavior to Kubernetes. It is a CNCF graduated project (TOC vote 2025-09-11, announced 2025-10-08). Latest is v1.24.0.

  • Serving : runs HTTP apps that autoscale, even down to zero pods.
  • Eventing : routes CloudEvents between producers and consumers.
  • Two independent parts, install either one alone.
  • Plain Deployment + Service + Ingress + HPA means many YAMLs and no scale to zero.
Try-out sizeOne node, 6 CPUs, 6 GB RAM, 30 GB disk.
Service (ksvc) Configuration Route makes Revision v1 Revision v2 10% 90% Deployment K8s Service+ Ingress

Editing the template makes a new immutable Revision. Rollback and traffic split are cheap.

2.Serving : How a Request Flows

The activator holds requests when a Revision has zero pods, wakes it up, then forwards. It also stays in the path while spare room is short: with the default target-burst-capacity of 211, until three idle Pods are ready.

  • controllerReconciles Service, Revision, Route.
  • activatorBuffers requests at zero pods and on bursts.
  • autoscalerReads metrics, sets replicas (KPA).
  • queue-proxySidecar in every pod. Counts concurrency.
  • webhookValidates and defaults resources.
Client Networking layerKourier / Istio Activator pods up Pod queue-proxy User container Autoscaler metrics setsreplicas

3.Autoscaling

Default autoscaler is KPA, driven by concurrency or RPS. Scale to zero is on by default.

  • Annotations autoscaling.knative.dev/min-scale and max-scale.
  • Cold start : first request waits for a pod. Use min-scale 1 for latency.
  • HPA class: serving-hpa.yaml by YAML; the Operator always installs it. CPU needs metrics-server.
0 pods request 1 pod load N pods idle, scale to zero
TopicDeployment + HPAKnative Service
Scale to zeroNoYes (default)
Scale signalCPU / memoryConcurrency or RPS
Traffic splitIngress or mesh tricksBuilt in per Revision
RollbackReplicaSets, manualImmutable Revisions
Objects to writeDeployment, Service, Ingress, HPAOne Service
Cold startNoneYes
In short : Knative Serving runs an HTTP app as one Service with immutable Revisions and built-in traffic split, and the KPA autoscaler scales it from zero to N.

1.Eventing : Broker and Trigger

Producers do not know consumers. Triggers decouple them.

  • SourcePingSource, ApiServerSource, KafkaSource. Turns things into CloudEvents.
  • BrokerEvent mailbox in one namespace.
  • TriggerFilter + subscriber.
  • ChannelLower-level pipe with Subscription.
SourcePingSource Broker Trigger Atype=x Trigger Btype=y App 1 App 2 after retries Dead letter sink

2.CRD Cheat Sheet

KindapiVersionWhat it is
Service (ksvc)serving.knative.dev/v1Whole app, Config + Route
Configurationserving.knative.dev/v1Pod template, makes Revisions
Revisionserving.knative.dev/v1Immutable snapshot
Routeserving.knative.dev/v1URL to Revisions, percent split
Brokereventing.knative.dev/v1Event ingress
Triggereventing.knative.dev/v1Filter + subscriber
InMemoryChannelmessaging.knative.dev/v1Dev-grade channel
KnativeServingoperator.knative.dev/v1beta1Operator CR for Serving
KnativeEventingoperator.knative.dev/v1beta1Operator CR for Eventing
operator.knative.dev serving.knative.dev eventing.knative.dev Service, Revision,Route, Config Broker, Trigger,Sources messaging : channels Your cluster knative-serving activator, ctrl knative-eventing ctrl, broker two namespaces, install either one

3.Install Steps

Setup Flow
1. CRDs + coreserving-crds, serving-core
2. NetworkingKourier, Contour, Istio, Gateway API
3. DNSMagic DNS (a LoadBalancer IP) or Host header
4. Apply ksvcRevision, Route, Deployment
5. EventsBroker + Trigger
In short : Eventing routes CloudEvents from Sources through a Broker to Triggers that filter and deliver them, and Serving and Eventing install as two independent parts in their own namespaces.

1.kubectl Commands (Knative)

CommandDescription
kubectl apply -f serving-crds.yamlInstall Serving CRDs, then serving-core
kubectl apply -f serving-default-domain.yamlMagic DNS via sslip.io (needs a LoadBalancer IP; YAML installs only)
kubectl apply -f operator.yamlInstall the Knative Operator
kubectl get ksvc -AList services, URL, ready state
kubectl get revisions,routes,configurationsObjects a ksvc created
kubectl get pods -n knative-servingControl plane health
kubectl describe ksvc <name>Conditions show why not Ready
curl -H "Host: <url>" http://<INGRESS_IP>Call a service with no DNS
kubectl get brokers,triggers,sourcesInspect eventing objects

2.Minimal Knative Service

hello.yaml
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: hello
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/max-scale: "5"
    spec:
      containers:
        - image: "ghcr.io/knative/helloworld-go\
            @sha256:05c2c89a7a143b85682a69d9aa\
            9f1d0f2a0ec857ba83bd13b8317f2e5ddb13de"
One Service makes the Configuration, Route and first Revision. Annotations here are copied to every Revision. The image is pinned by digest: helloworld-go has only the tag latest.

3.Gotchas and Fit with Core K8s

Gotchas
  • No networking layer: the ksvc still gets a URL, but stays not Ready (IngressNotConfigured).
  • Revisions are immutable. Edit the Service, not the Revision.
  • Gateway API networking is beta in install docs.
  • Native brokers (Kafka, RabbitMQ) beat channel-based ones.
Fits with core K8s
  • A ksvc makes a Deployment, a Service and an Ingress-like route.
  • Control plane lives in knative-serving and knative-eventing.
  • Config is in ConfigMaps like config-autoscaler, config-network, config-domain.
In short : Knative Serving scales HTTP apps from zero to N with Revisions and traffic split, while Eventing routes CloudEvents with Brokers and Triggers.

Observability

OpenTelemetry Operator

1.What is the OTel Operator ?

A Kubernetes operator that deploys and manages OpenTelemetry Collectors and injects auto-instrumentation into your pods, all from YAML.

  • Hand-writing Collector Deployments, ConfigMaps and Services drifts.
  • Tracing normally means changing every app, per language.
  • Many Collector replicas double-scrape Prometheus targets.

Operator = controller + CRDs + admission webhooks. The webhooks need TLS certs, so cert-manager is the documented prerequisite.

You (kubectl apply) Operator Controller Webhooks creates Collectors mutates App pods(injected) cert-manager (TLS) install first (webhook cert)

2.The Moving Parts

Collector and Instrumentation CRs do most of the work. The operator keeps them in sync.

Reconcile loop Collector CR Operator reconciles ConfigMap + Workload + Service drift

Drift detected, operator updates it back.

  • ManagerWatches CRs, creates workloads, ConfigMaps, Services.
  • WebhooksValidate CRs and mutate pods for injection.
  • Collector CRMode, image, receivers, processors, exporters, pipelines.
  • Instrumentation CRExporter endpoint, propagators, sampler, language images.
  • Target AllocatorSplits Prometheus targets across replicas.
  • OpAMPBridge CRConnects Collectors to an OpAMP server for remote config and health.

3.Collector Modes

The CR picks how the Collector runs. The operator creates a Service <name>-collector.

Tip : modes map to Deployment, DaemonSet and StatefulSet.
ModeShapeWhen to use
deploymentStateless gatewayCentral OTLP endpoint
daemonsetOne per nodeNode metrics, logs
statefulsetStable pod namesTarget Allocator
sidecarInside app podPer-pod isolation
In short : The OpenTelemetry Operator turns Collectors and auto-instrumentation into CRDs. The Collector CR picks a mode and the operator builds the workload, ConfigMap and Service.

1.Auto-Instrumentation

No image change needed. A pod annotation opts the workload in, and a webhook mutates the pod.

Injection flow
Deploymentinject-java: true
API servercalls the webhook
Operator webhookreads Instrumentation
Mutated Podinit container + OTEL_* env
Appsends OTLP
Collector Svc:4317 / :4318
Annotation (pod template)Value
instrumentation.opentelemetry.io/
inject-java
"true"
...-nodejs / python / dotnet / go"true"
...-apache-httpd / nginx / sdksame pattern
Other values"name", "ns/name"
or "false"
Note : the init container copies the language agent into the app container. The app image never changes.
Env vars injected
  • OTEL_SERVICE_NAMEworkload
  • OTEL_EXPORTER_OTLP_ENDPOINTCR exporter
  • OTEL_PROPAGATORSfrom the CR
  • OTEL_TRACES_SAMPLERCR sampler
  • OTEL_RESOURCE_ATTRIBUTESpod, node, ns
Mutated Pod init container language agent copies app container + OTEL_* env

2.Target Allocator

Shards Prometheus scrape targets across Collector replicas so nothing is scraped twice. It can discover ServiceMonitor and PodMonitor CRs.

  • Enabled with spec.targetAllocator.
  • Usually statefulset mode (stable pod names); daemonset needs the per-node strategy.
  • It rewrites scrape_configs to http_sd_configs.
  • prometheusCR.enabled: true picks up the monitor CRs.
Service / PodMonitor Target Allocator http_sd: a share each Collector 0 Collector 1 splits targets, no double scrape scrape only own targets
allocationStrategy :
consistent-hashing (default)
least-weighted (fewest targets)
per-node (daemonset mode)
Tip : keep a prometheus receiver in the Collector config. The allocator only hands out targets, the Collectors still scrape.
In short : Annotate the pod template and the webhook injects the agent and OTEL_* env from the Instrumentation CR. The Target Allocator splits Prometheus targets across Collectors.

1.Handy Commands

CommandWhat it does
kubectl apply -f opentelemetry-operator.yamlInstall via manifest from GitHub releases
helm install opentelemetry-operator open-telemetry/opentelemetry-operatorInstall via Helm (repo open-telemetry)
kubectl get opentelemetrycollectors -AList Collector CRs
kubectl get instrumentations -AList Instrumentation CRs
kubectl describe pod <app-pod>Check init container + OTEL_* env
Install order
cert-manager
Operator
Collector CR
Instr. CR
annotate pod template
restart pods

2.Quick YAML

# Collector (v1beta1)
kind: OpenTelemetryCollector
spec:
  mode: deployment
  config:
    receivers: { otlp: ... }
    exporters: { debug: {} }
    service: { pipelines: ... }
# Instrumentation (v1alpha1)
kind: Instrumentation
spec:
  exporter:
    endpoint: http://demo-collector:4318
  propagators: [tracecontext, baggage]
  sampler: { type: parentbased_traceidratio }
  java:
    image: <registry>/autoinstrumentation-java
  • Collector CR: mode plus a normal Collector config.
  • Instrumentation CR: where to send, how to propagate, how to sample.
  • Endpoint is the <name>-collector Service.
  • Apply both, then annotate the pod template.
Gotchas
  • Annotation goes on the pod template, not the Deployment metadata.
  • Create the Instrumentation CR first. Restart existing pods.
  • Webhook cert not ready blocks CR and pod creation.
  • Versions are 0.x. Read release notes before upgrading.
Where the data lands
  • Metrics : remote write or scrape into Prometheus.
  • Logs : OTLP to Loki at /otlp.
  • Traces : OTLP to Tempo on 4317 (gRPC) or 4318 (HTTP).
  • Profiles : Pyroscope (OTLP profiles are alpha).
In short : Install cert-manager, then the operator. Apply the Collector and Instrumentation CRs, then annotate the pod template.

Observability

Prometheus and ServiceMonitors

1.What is Prometheus ?

A pull-based time-series database. It scrapes /metrics from discovered targets into a local TSDB, answers PromQL and sends alerts to Alertmanager.

Defaults by layer
  • Plainscrape 1m, timeout 10s, rules 1m, retention 15d
  • Prometheus CRscrape 30s, rules 30s, retention 24h if unset

First query you run : up == 0.

2.Data model

  • Series = name (__name__) + labels.
  • New label value = new series.
  • Targets add job, instance. Scrapes add up, scrape_*.
  • Base units, plural suffix, counters end in _total.
TypeBehavesSeriesQuery with
Counteronly goes up, resets to 0 on restart_totalrate(), increase()
Gaugegoes up and downthe valuethe value, avg_over_time()
Histogramcumulative buckets, aggregatable_bucket{le}, _sum, _counthistogram_quantile() over rate
Summaryquantiles computed in the client{quantile}read it, cannot be aggregated

Native histograms : one composite sample, far fewer series. Stable since 3.8.0, still opt-in with scrape_native_histograms: true.

3.The scrape loop

relabel_configs pick targets and build __address__. metric_relabel_configs run on every sample (not up). A breached sample_limit fails the whole scrape. Vanished series get a staleness marker; lookback is 5m.

node-exporter :9100 kube-state-metrics :8080 kubelet / cAdvisor :10250 finds targets GET /metrics discovery __meta_* labels relabel keep / drop scrape every interval metric relabel every sample TSDB head + WAL, 2h blocks rules alerts Alertmanager PromQL, Grafana

4.PromQL essentials

  • x instant, x[5m] range vector. Rate first, then aggregate : sum(rate(x[5m])), never rate a sum.
  • rate needs two samples : range of 4x the scrape interval or more (a convention). At 1m, [1m] is usually empty.
  • histogram_quantile must keep le in by. absent(x) alerts on vanished metrics.
Starter queryShows
up == 0targets that failed their last scrape
sum by (job) (rate(http_requests_total[5m]))request rate per job
histogram_quantile(0.9, sum by (job, le) (rate(http_request_duration_seconds_bucket[10m])))p90 latency, classic histogram
absent(up{job="api"})1 when the api job has no series at all

5.Where Kubernetes metrics come from

SourcePort, pathMeasures
node-exporter (DaemonSet):9100/metricsnode CPU, memory, disk, network
kube-state-metrics:8080/metricsobject state from the API
kubelet (no Service : the operator makes one):10250/metricsthe kubelet's own metrics
cAdvisor (in the kubelet):10250/metrics/cadvisorper-container usage
kube-apiserver:6443/metricsrequest rates and latencies
metrics-serverreads kubelet /metrics/resourcelatest CPU, memory : not a source
In short : Prometheus scrapes discovered targets into a local TSDB of name-plus-labels series. Query counters with rate first, then aggregate.

1.Why an operator ?

  • Config becomes CRDs (Operators).
  • Instance CRDs become StatefulSets.
  • Scrape CRDs : one gzipped config in Secret prometheus-<name>.
  • config-reloader : /-/reload, no restart.

Install : bundle.yaml (kubectl create, into default), kube-prometheus-stack (Helm), kube-prometheus (jsonnet).
Also : Probe, ThanosRuler, AlertmanagerConfig (v1alpha1).

2.The CRDs

KindVersionShortRole
Prometheusv1prominstance
PrometheusAgentv1alpha1promagentscrape, remote write only
Alertmanagerv1aminstance
ServiceMonitorv1smonscrape Services
PodMonitorv1pmonscrape Pods
ScrapeConfigv1alpha1scfgraw jobs, outside targets
PrometheusRulev1promrulerules

3.ServiceMonitor anatomy

  • selector picks Services by label; namespaceSelector : own namespace by default.
  • endpoints[].port = Service port name, not a number or Pod port name. Or targetPort (container port name or number).
  • path /metrics, interval from the CR. relabelings before the scrape, metricRelabelings after.
  • job = Service name (default). Job : serviceMonitor/<ns>/<name>/<i>.
  • PodMonitor selects Pods directly : DaemonSets, sidecars, no Service (named ports : Services).
kind: ServiceMonitor
metadata: {labels: {release: kps}}
spec:
  selector: {matchLabels: {app: web-app}}
  endpoints:
  - {port: web, interval: 30s}

4.How a ServiceMonitor gets picked

  • Two levels : serviceMonitorSelector matches labels, and serviceMonitorNamespaceSelector the namespace.
  • kube-prometheus-stack wants release: <release>. No label : silently ignored. Or set serviceMonitorSelectorNilUsesHelmValues: false (also pods, probes, scrape configs, rules).
Prometheus CR serviceMonitorSelector + NamespaceSelector ServiceMonitor watches selector Prometheus /-/reload operator Service writes EndpointSlices Secret prometheus-<name> config-reloader Pods port web
ValueResource selectorNamespace selector
{}all objectsall namespaces
omitted (null)nothingown namespace only
matchLabelslabelled objectslabelled namespaces

5."My ServiceMonitor is not picked up"

  1. Labels match ? Namespace matched ?
  2. Rejected ? InvalidConfiguration Warning Event.
  3. In the generated config ?
  1. Discovered ? None : no Service. Dropped : port.
  2. RBAC : list/watch Services, Pods, Endpoint(Slice)s.
  3. Scraped ? up == 0 and its error.
CommandDoes
kubectl get prometheus,servicemonitors,podmonitors,prometheusrules -Alist objects
kubectl -n monitoring get secret prometheus-k8s -ojson | jq -r '.data["prometheus.yaml.gz"]' | base64 -d | gunzip | grep <name>in the config ?
kubectl get events -n <ns> --field-selector involvedObject.name=<name>rejected ?
kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090UI : Targets, SD pages
promtool check config prometheus.ymlvalidate a config
kubectl apply --server-side --force-conflicts -f monitoring.coreos.com_*.yamlCRDs, server-side
In short : A ServiceMonitor picks Services by label and scrapes their Pods on a named port. The Prometheus CR must select it and its namespace : {} = all, omitted = none.

1.Recording rules and PrometheusRule

  • Precomputes an expression every group interval, stored as a new series.
  • Name : level:metric:operations, e.g. job:http_requests:rate5m.
  • PrometheusRule objects, picked by ruleSelector + ruleNamespaceSelector. Nil ruleSelector = no rules.
  • Rendered into ConfigMaps prometheus-<name>-rulefiles-<n>.

Test with promtool test rules. Alerting rules (for, keep_firing_for) : Alertmanager Routing.

2.Storage : head, WAL, blocks, retention

  • Writes : in-memory head + WAL. Every 2h the head becomes a block.
  • Compaction merges blocks (to 10% of retention or 31d). Retention deletes whole blocks, up to 2h late.
  • Local disk, no NFS. Size retention at most 80 to 85% of the volume.
  • Disk = retention s x samples/s x 1 to 2 bytes.
scrape head (memory) every 2h block WAL compaction bigger blocks retention deleted
LayerDefault retentionWatch out
Plain Prometheus15dtime or size, first to trigger wins
Prometheus CR24h when nothing is setno volumeClaimTemplate = emptyDir
kube-prometheus-stack10don emptyDir unless a volumeClaimTemplate is set

Config-file retention since 3.8.0 (reloadable, flags deprecated). The operator uses it only for 3.11.0+ (3.8 to 3.10 : unit bug).

3.Cardinality

  • Series count, not sample count, drives memory.
  • Most metrics under 10 label combinations; check any over 100. No user IDs or emails as labels.
  • Find : /api/v1/status/tsdb, promtool tsdb analyze.
  • Limit : sample_limit, metric relabel drop or labeldrop.

4.High availability and scale

OptionWhat it does
replicas: 2twin Pods, prometheus_replica; dedupe later
shards: Nhash of __address__; no resharding
Agentscrape + remote write; no queries, rules

Remote write : a queue per endpoint reads the WAL (slow receiver, scraping goes on). About 25% more memory; down over about 2h = data lost. RW 2.0 still experimental. Tempo remote-writes span metrics here; KEDA runs PromQL.

5.Three ways to run it

WayCRDsDiscoveryRules
plain prometheus chartnoprometheus.io/* annotations, by relabelingConfigMap
kube-prometheus-stack (Helm)yesServiceMonitor and friends (13 out of the box)35 PrometheusRule objects
kube-prometheus (jsonnet)yessamekubernetes-mixin
  • The operator ignores prometheus.io/scrape : leaving the plain chart drops those targets. Add a ScrapeConfig.
  • Helm never upgrades CRDs. Flux defaults : install.crds: Create, upgrade.crds: Skip. Set both CreateReplace.
CommandDoes
promtool check rules rules.ymllint rules
promtool test rules tests.ymlunit-test rules
promtool tsdb analyze <dir>cardinality
curl localhost:9090/api/v1/status/tsdbtop series
curl localhost:9090/api/v1/status/buildinfotrue version
Seen in practice

No operator : a community chart rendered once, applied by Flux. Annotation targets; most app metrics by remote write from the OTel gateway. Rules in a ConfigMap, promtool-tested.

Pitfall : K3s repeats apiserver and etcd series in the kubelet job, so the head grows until an OOMKill. Raising limits for duplicated series is the wrong fix. Do better : drop them at scrape and alert on head series.

In short : Record expensive queries, watch retention and series count, scale with shards or an agent plus remote write. The operator only uses what its selectors match.

Observability

Alertmanager Routing

1.Two halves

  • Prometheus evaluates alerting rules and sends the firing alerts. Loki's ruler can send too (Kubernetes Events as a Signal).
  • Alertmanager deduplicates, groups, inhibits, silences and routes them to receivers (Slack, PagerDuty, webhook, ...).
RememberAlertmanager never runs PromQL. If an alert is wrong, fix the rule in Prometheus.

2.Alerting rules

FieldDefaultMeaning
exprrequiredPromQL; one alert per output series
for0stime in pending before firing
keep_firing_for0skeep firing after it clears (anti-flap); 2.42.0+
labelsnoneidentity and routing keys
annotationsnonetext; may change freely
  • No $value in labels : each evaluation is a new alert, so for never completes.
  • for on increase(x[15m]) > 0 is not "bad for that long" : one failure holds it true.
- alert: PodCrashLooping
  expr: increase(
    kube_pod_container_status_restarts_total[15m])
    > 3
  for: 5m
  keep_firing_for: 10m
  labels: { severity: warning }
  annotations:
    summary: "{{ $labels.namespace }}/\
      {{ $labels.pod }} restarting"

On Kubernetes it lives in a PrometheusRule, selected by ruleSelector (Prometheus and ServiceMonitors). Restarts : kube-state-metrics.

3.The send loop

  • Resent at most every 1m (--rules.alert.resend-delay) with EndsAt = now + 4 x max(eval interval, resend-delay).
  • Pending is never sent; resolved is resent for 15m. Send to every replica, never via a load balancer.
TrapNo more sends, so alerts resolve as EndsAt passes : a dead Prometheus looks like "all clear". A Watchdog catches it.

4.Inside Alertmanager

dispatcher, once per alert alert in route walk tree group group_by wait group_wait each flush gossipsettle inhibitedyes: drop timeintervals silencedyes: drop peerwait dedup(nflog) sendretry receiver
  • The dispatcher runs once per alert. A new group waits group_wait.
  • Each flush runs the notifier stages in order. Inhibited and silenced alerts are dropped there (red).
  • A sent group goes into the nflog, so other replicas dedup against it.

5.The three timers

TimerDefaultWhat it does
group_wait30sdelay the first message of a new group so related alerts join
group_interval5mpace for updates; also the send timeout, max(group_interval, 10s) plus the peer wait
repeat_interval4hresend an unchanged group; rounded up to a multiple of group_interval

amtool check-config says SUCCESS for repeat_interval < group_interval; only the server logs a warning at load. resolve_timeout does nothing for Prometheus alerts (they always carry EndsAt).

In short : Prometheus fires and keeps re-sending alerts. Alertmanager groups them by labels, waits group_wait, updates every group_interval, repeats every repeat_interval.

1.The routing tree

  • Every alert enters the root route : no matchers allowed, a receiver required.
  • It walks the children depth-first and stops at the first match, unless that child sets continue: true. No child matches : the current node handles it.
  • Unset settings inherit from the parent. group_by: ['...'] means every label (no grouping).
  • First match wins : a catch-all above specific routes eats everything.
alert severity=critical, namespace=shop root route receiver: default group_by: [namespace, alertname] 1st 2nd 3rd alertname = Watchdog alertname = InfoInhibitor severity = critical heartbeat "null" pager no match : skip, match : stop

2.Matchers

  • Operators =, !=, =~, !~. matchers: replaces deprecated match and match_re (still parsed).
  • RE2 regexes are anchored : =~ "prod" misses production.
  • In YAML, quote a matcher holding { } [ ] , # | > :
  • Since 0.27 a UTF-8 parser runs in fallback mode; strict mode is opt-in (utf8-strict-mode).

3.Inhibition and silences

  • Inhibition mutes target alerts while a source alert fires and the equal labels match. A label missing on both sides counts as equal.
  • The source must arrive first, so a very short group_wait can leak the target.
  • A silence is matchers plus a time window, made with amtool or the UI. Silences and the notification log are saved to disk every 15m and kept 120h.
Way to muteTriggerLifetimeUse for
Inhibit ruleanother alert firingconfigcritical mutes warning, node down mutes its Pods
Silencematchers, by handstart and end time, savedmaintenance, known issues
mute_time_intervalsthe clockconfigquiet hours on a route
null receiverroute matchconfigdrop on purpose (InfoInhibitor)

4.Receivers

  • A receiver holds integrations; one with none is a null receiver.
  • No RESOLVED in Slack or email unless send_resolved: true.
  • *_file fields (api_url_file, routing_key_file) read a mounted Secret (ConfigMaps and Secrets).
  • Go text templates (.CommonLabels, .Alerts.Firing) : check-config parses but never runs them, so render them.
Integrationsend_resolved default
Slackfalse
emailfalse
webhooktrue
PagerDutytrue

Rocket.Chat is false in code, though the docs say true.

CommandWhat it does
amtool check-config alertmanager.ymlparse the config, print SUCCESS
amtool config routes test --config.file=am.yml --verify.receivers=pager severity=criticalreceivers a label set reaches
amtool alert add TestAlert severity=warning --annotation=summary=testinject a test alert
amtool silence add alertname=X -d 2h -c "maintenance"silence for 2h (default 1h)
amtool silence expire $(amtool silence query -q alertname=X)end those silences now
amtool template render --template.glob='t/*.tmpl' --template.text='{{ template "slack.default.title" . }}'render against example data
In short : Labels decide everything : the first matching route picks the receiver, inhibit rules and silences mute by matchers, and amtool tests a tree before any rule fires.

1.Operator objects

  • Alertmanager CR (CRDs) : StatefulSet alertmanager-<name>, Service alertmanager-operated (web 9093, tcp-mesh and udp-mesh 9094). One replica : gossip off.
  • Config : Secret alertmanager-<name> or AlertmanagerConfigs, merged into alertmanager-<name>-generated (Operators).
  • AlertmanagerConfig is v1alpha1 (API Machinery), validated by a webhook (Admission). Namespace selector nil = own namespace.
Operator traps
  • Operator v0.94.1 defaults to Alertmanager v0.34.0, the inhibition un-mute bug. Set spec.version.
  • limits.maxPerSilenceBytes passes a flag Alertmanager lacks : crash-loop.

2.Team routes : AlertmanagerConfig

  • The operator appends namespace="<ns>" to each top route and both inhibit sides (removes nothing from routes), forces continue: true, renames receivers, prepends the routes.
  • Alerts with no namespace (node, cluster-wide) never reach a team route.
AlertmanagerConfig ns team-a severity=critical to team-a-slack operator append namespace, force continue, rename generated route severity="critical", namespace="team-a" continue: true, receiver team-a/team-a/team-a-slack placed before the main routes
StrategyOnNamespace (default)OnNamespaceExceptForAlertmanagerNamespaceNone
Effectnamespace matcher addedsame, except objects in Alertmanager's own namespace see all alertsnothing injected

3.High availability

  • Replicas gossip silences and the nflog on 9094, TCP and UDP (TCP only with --cluster.tls-config).
  • Each waits position x 15s and skips if a peer already sent. Broken gossip fails open : duplicates.
  • Allow 9093 from Prometheus, 9094 TCP and UDP between replicas (Network Policy in Depth).
Prometheus am-0wait 0s am-1wait 15s am-2wait 30s gossip :9094 Slack nflog : already sent nflog : already sent

4.Watch the watcher

  • Watchdog always fires (vector(1)). Route it to a heartbeat service : when pings stop, the pipeline is dead.
  • Both distributions ship it unwired. Set group_interval on its route too, or repeat_interval: 1m becomes 5m.
  • Self-monitoring : alertmanager_notifications_failed_total{reason}, alertmanager_config_last_reload_successful, alertmanager_notifications_suppressed_total; on Prometheus, prometheus_notifications_alertmanagers_discovered.
Settingkube-prometheus
v0.19.0
kube-prometheus-stack
92.1.1
replicas31 (emptyDir)
root group_by[namespace]['namespace']
repeat_interval12h12h
receiverDefault'null'
Watchdogempty receiverrouted to null
CommandDoes
kubectl -n monitoring get secret alertmanager-main-generated -o jsonpath='{.data.alertmanager\.yaml\.gz}' | base64 -d | gunzipmerged config
amtool cluster showgossip status
curl -XPOST localhost:9093/-/reloadreload
Seen in practiceA plain Deployment, a PVC for silences and nflog, Slack receivers by severity, URLs from a SOPS Secret (Secrets in Git with SOPS) via Flux CD v2. Pitfalls : no Watchdog, nothing scrapes Alertmanager, rules set runbook but the template prints runbook_url. Do better : a Watchdog to a dead man's switch, scrape it, one label name everywhere.
In short : The operator runs Alertmanager as a StatefulSet and merges team AlertmanagerConfigs scoped to their namespace. Run replicas, send to all of them, wire a Watchdog.

Observability

Loki and Log Pipelines with Alloy

1.Labels, streams, chunks

  • kubectl logs shows one container (Pods). Loki keeps every Pod's lines, found by label.
  • A stream = one tenant + one unique label set. Its lines are compressed into chunks; a small TSDB index maps label sets to chunks. Both live in object storage.
  • A query picks streams by label, then greps, parses and aggregates the lines.
  • Cardinality matters : 15 label names per series by default; under 100,000 active streams, even for a huge tenant.
Good labels cluster, namespace, app, container. Few values, and they do not change per request.
Bad labels pod name, request ID, trace ID. Every new value is a new stream. Put them in structured metadata or in the line.

2.Write path and read path

  • The distributor hashes tenant + labels onto the ring. Quorum floor(RF / 2) + 1 : RF 3 needs 2 writes.
  • Out-of-order lines only within 1h of the stream's newest line (older docs say 2h), else too_far_behind.
  • The compactor keeps one index per tenant per day and applies retention.
Alloy / OTel client push distributor validate, rate limit, hash ingesters replicas, WAL, chunks flush Grafana logcli query frontend split, cache querier runs LogQL last 3h older object storage chunks + TSDB index

3.Labels vs structured metadata vs line

Structured metadata (Loki 3.0+) : kept with each line, not indexed, still queryable. Trace IDs go here (Tempo).

Index labelsStructured metadataLine
indexedyes (TSDB)nono
defines a streamyesnono
good forbounded valuespod, trace ID, user IDthe rest
query{app="x"}| trace_id="abc" after the selector|= "abc", parsers
needsany schematsdb + v13any

4.Deployment modes

ModeTargetsScaleStatus
Monolithic-target=allup to about 20 GB/dayHelm chart default since 12.0.0
Simple scalablewrite, read, backendabout 1 TB/daydeprecated : Loki 4.0 will refuse to start
Microservicesone per componentvery largesupported

The chart moved to grafana-community on 2026-03-16. Chart 17.0.0 refuses minio.enabled=true; the MinIO subchart goes on 2026-10-31.

5.Schema and retention

  • New installs : tsdb, v13, period: 24h. Structured metadata is on, so Loki will not start unless the active period is v13 + tsdb.
  • A schema change cannot be rolled back : add a new period starting at a future date.
  • Retention is off by default. The compactor runs it : retention_enabled, delete_request_store (required), retention_period.
  • Markers sit on local disk : a PVC (Storage) or 3.7's deletion_marker_object_store_prefix.
Bucket lifecycle rule : over retention + 2h, tenant chunk prefix only; an empty prefix also expires index/.
In short : Loki indexes labels, not text. A stream is one label set, its lines are compressed into chunks in object storage, and anything high-cardinality belongs in structured metadata or the line.

1.What is Alloy ?

  • Grafana's collector : an OTel Collector distribution with native Prometheus, Loki and Pyroscope pipelines.
  • It replaces two retired agents : Grafana Agent (EOL 2025-11-01) and Promtail (EOL 2026-03-02, not part of Loki 3.7).
  • alloy convert --source-format=promtail migrates. Positions and metric names change : dashboards break.

2.Alloy syntax in five lines

  • Block : <component.name> "<label>" { attr = value }.
  • Export : discovery.kubernetes.pods.targets. References build a graph, so file order does not matter.
  • // comments, backtick raw strings for regexes. alloy fmt, alloy validate.
  • --stability.level gates experimental components.
  • UI on 127.0.0.1:12345 (the chart sets 0.0.0.0). Reload : /-/reload or SIGHUP.

3.A Kubernetes log pipeline

loki.source.file "pods" {
  targets    = discovery.relabel.pods.output
  forward_to = [loki.process.pods.receiver]
  file_match {
    enabled = true // else the glob is literal
  }
}
loki.process "pods" {
  stage.cri {}
  forward_to = [loki.write.default.receiver]
}
discovery.kubernetes Pods on this node targets discovery.relabel labels + __path__ output loki.source.file file_match { enabled = true } forward_to loki.process stage.cri, labels receiver loki.write batch, retry POST Loki /loki/api/v1/push
Three traps
  • A glob __path__ (/var/log/pods/*$1/*.log) needs file_match. alloy validate stays quiet.
  • Set alloy.mounts.varlog: true. Persist storagePath : /tmp/alloy loses positions.
  • No clustering on a log-tailing DaemonSet.
loki.write Batches 1MiB or 1s; on 5xx or 429 backs off 500ms to 5m, drops after 10 retries; other 4xx drop at once. Watch loki_write_dropped_entries_total. WAL experimental, off.

4.File tailing vs API tailing

loki.source.fileloki.source.kubernetes
reads/var/log/pods on the nodekubelet log endpoint, through the API server
needshostPath, usually root (Pod Security)read on pods/log (RBAC), no privileges
runs asDaemonSet (DaemonSets)Deployment, clustered
restart safetypositions file of offsetspositions file of last timestamps
costnode disk, inotifykubelet CPU and network
node logsyesno

5.loki.process stages that matter

StageDoes
stage.cristrip CRI prefix, join partials
stage.json / logfmtextract fields
stage.labelsbounded field to a label
stage.structured_metadatahigh-cardinality field, unindexed
stage.drop / multilinedrop noise, join stack traces

6.Other ways in

  • Events : loki.source.kubernetes_events, run once or clustered, never per node (Events).
  • OTLP : otelcol.receiver.otlp to otelcol.exporter.otlphttp, endpoint http://loki:3100/otlp. It appends /v1/logs, so the full path fails. Collector setup : OTel Operator.
In short : Alloy wires components into a graph by reference : discover Pods, relabel them, tail files or the API, turn fields into labels or structured metadata, and push the lines to Loki.

1.Query shape

  • A log query is a stream selector, then a pipeline : line filters |=, !=, |~, !~ (this regex is not anchored, unlike label matchers), a parser (json, logfmt, pattern, regexp), label filters, line_format.
  • A parse failure adds __error__ (filter __error__=""). One matcher must not match empty : {app=~".*"} is rejected.
{namespace="shop"} index |= "error" | json | status >= 500 line_format on each line : only the first box uses the index

2.Metric queries

Like PromQL : wrap a log query in rate, count_over_time, bytes_rate; unwrap a field for quantile_over_time; sum by.

QueryGives
{namespace="shop", container="api"} |= "error" != "timeout"error lines, minus timeouts
{job="ingress"} | json | status >= 5005xx lines, parsed
sum by (container) (rate({namespace="shop"} |= "error" [5m]))errors per second, per container
{job="x"} | trace_id="0242ac120002"one trace's lines (metadata)
quantile_over_time(0.99, {app="api"} | logfmt | unwrap duration_seconds(latency) [5m]) by (route)p99 latency per route

3.OTLP and its labels

  • OTLP logs go to /otlp/v1/logs. Loki 3.7.8 indexes 18 resource attributes (docs say 17, the code wins), dots as underscores : service_name, k8s_pod_name ... The rest is structured metadata.
  • Grafana now advises demoting k8s.pod.name and service.instance.id : replace the global list, or per tenant set ignore_defaults: true and list what to index.
  • A bare per-tenant structured_metadata rule loses : defaults are prepended, first match wins.

4.Limits and discard reasons

ReasonHTTPKnob (default)
rate_limited429ingestion_rate_mb 4, burst 6
per_stream_rate_limit4293MB
stream_limit429max_global_streams_per_user 5000
line_too_long400max_line_size 256KB
too_far_behind400the 1h window
max_label_names_per_series40015

Counted in loki_discarded_samples_total{reason}. "too many outstanding requests" is the query queue : frontend max_outstanding_per_tenant 2048, scheduler max_outstanding_requests_per_tenant 32000.

5.Running it

CommandDoes
logcli query --since 1h --limit 100 '{namespace="shop"} |= "error"'LogQL from a shell
logcli series '{}' --since=1h --analyze-labelsfind high-cardinality labels
loki -config.file=loki.yaml -verify-config=truecheck a config and exit
curl http://loki:3100/readyreadiness, says ready
alloy validate config.alloyschema errors, not runtime ones
kubectl debug -it loki-0 --image=busybox --target=lokia shell : no shell in the image since 3.6.0
Seen in practice Monolithic Loki, a rendered chart under Flux, chunks in a self-hosted S3 store with SPIRE-issued tokens, no static keys (SPIRE). Logs come from an OTel Collector DaemonSet tailing /var/log/pods over OTLP (OTel Operator), not Alloy. Pitfall : the OTLP default labels, k8s_pod_name included, against current advice. Do better : demote pod names to structured metadata, keep a few bounded labels.
In short : Pick streams by label, then filter and parse the lines. Wrap a log query in rate or count_over_time to get numbers, and keep OTLP labels few.

Observability

Kubernetes Events as a Signal

1.The Event object

  • A namespaced API object that says why another object changed.
  • The API reference : "informative, best-effort, supplemental data". Reasons and messages "may evolve with time".
  • One stored object, two API views : core v1 and events.k8s.io/v1 (GA 1.19). v1beta1 is gone since 1.25.
  • Short name ev. Events about Nodes or PVs live in default.
core/v1events.k8s.io/v1Meaning
involved­Objectregardingthe object it is about
messagenotetext, max 1024 bytes if eventTime is set
reasonreasonCamelCase cause, max 128
typetypeNormal or Warning
reporting­Componentreporting­Controllerwho reported it (source is deprecated­Source)
countseries.counthow many repeats
last­Timestampseries.last­ObservedTimelast seen

2.Who writes Events

  • Legacy recorder (core/v1) : kubelet and kube-controller-manager. A repeat bumps count.
  • Newer recorder (events.k8s.io/v1) : kube-scheduler and kube-proxy. Repeats become a series.
  • Flux 2.9.x writes core/v1. Controllers record Events while they reconcile (who is who).
Not an audit logAny client with create on events can write one (RBAC). For "who did what", use the API server audit log.

3.Folded, throttled, dropped

kubelet / scheduler recorder (client-go) same event : bump count ~10 similar / 10 min : combined spam bucket : 25, then 1 per 5 min queue of 1000 : drop when full lost lost kube-apiserver validate EventRateLimit : alpha, off etcd lease 1h expired watchers kubectl events, exporters list / watch from etcd, no cache
WhatRecorder numbers
Legacy combinefrom the 10th similar message within a sliding 600 s, prefix (combined from similar events):
Legacy spamburst 25, then 1 per 300 s, per source, object and type
New series2nd repeat starts it; later counts written on a 30 min heartbeat or after 6 quiet min
Both1000-event queue per sink, 12 write tries
Server sideThe admission plugin EventRateLimit can limit Events per server, namespace, user or source and object. It is still alpha (since 1.13) and off by default.

4.Retention

  • --event-ttl defaults to 1h. It is an etcd lease set on every create and update, so an Event vanishes an hour after its last write.
  • "No events" on an old Pod means expired, not healthy.
  • No watch cache for Events : lists and watches hit etcd. Big clusters give Events their own etcd (--etcd-servers-overrides=/events#...).
Best effortNever build billing or audit on Events. A missing Event proves nothing.
In short : An Event is a best-effort note about why an object changed. Recorders fold and drop repeats before the API server, and the API server forgets each Event an hour after its last write.

1.Three ways to look

  • kubectl events (stable 1.28) : sorted by time, --for kind/name, --types=Warning, --watch.
  • kubectl get events is not sorted : add --sort-by=.lastTimestamp.
  • kubectl describe (1.36+) shows Events for one exactly named object only. Add --show-events for several.

2.Field selectors

  • A closed list : involvedObject.kind, .name, .namespace (and four more), reason, reportingComponent, source, type, metadata.namespace, metadata.name.
  • Operators =, ==, != only. A comma means AND.
  • Anything else, message=... included, returns "field label not supported".

3.Reasons worth knowing

ReasonFromTypeNearest metric
FailedSchedulingschedulerWarningkube_pod_status_unschedulable
Pulling / PulledkubeletNormalnone
Failed (start or pull)kubeletWarning..._waiting_reason
BackOffkubeletWarning (restart), Normal (image pull)..._waiting_reason (CrashLoopBackOff or ImagePullBackOff)
Unhealthykubelet proberWarningnone (readiness : kube_pod_status_ready)
FailedMountkubeletWarningnone
EvictedkubeletWarningkube_pod_status_reason{reason="Evicted"}
SystemOOMkubelet, on the NodeWarningnone
OOMKillingnode-problem-detector onlyWarning..._last_terminated_reason{reason="OOMKilled"}

Metrics come from kube-state-metrics. More on the reasons : scheduling, image pulls, probes, volume failures.

4.Reading a failure story

scheduler kubelet FailedScheduling Warning Scheduled Pulling Pulled Created Started Unhealthy x3 Killing BackOff x12 a healthy start stops at Started yellow = Warning

5.Traps

  • BackOff and Failed each mean two things : read the message. A type=Warning filter misses image-pull BackOff.
  • Scheduler Events read through core/v1 show count: null. Read series.count, which can lag up to 30 min.
  • Messages change between releases : 1.35 dropped the container name from Created / Started, 1.37 added preemption text to FailedScheduling. Match the reason, not the message.
CommandWhat it does
kubectl events -A --types=WarningAll warnings, sorted by time
kubectl events -n shop --for deployment/web --watchOne object, streaming
kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestampWarnings, sorted
kubectl get events -n shop --field-selector involvedObject.kind=Pod,involvedObject.name=web-0One Pod
kubectl get events -A --field-selector reason=FailedSchedulingOne reason
kubectl describe pods -l app=web --show-eventsSeveral Pods (1.36+)
In short : Use kubectl events for a sorted view and field selectors to narrow it. Read the reason, not the message, and remember that some reasons mean two things and one of them is Normal.

1.Why export ?

The 1h TTL means no history. kube-state-metrics deliberately exports no Event metrics (high cardinality, "transition signals"). If you want Events tomorrow, ship them somewhere today.

2.Three exporters

kubernetes-event-exporter v1.7Alloy loki.source.kubernetes_eventsOTel k8s_events / k8s_objects
Maintenanceno release since 2024-02, forks, no clear successoractive, v1.20.1active, alpha / beta
Count updatesignored : first occurrence onlyre-sent with the new countnew and updated
Late eventsdropped after 5 s by defaultkeptkept
Restartstartup replay discardedpositions filestorage extension
One instance byleader electionsingle Deployment, or clustering (1.16+)k8s_leader_elector
Sinksabout 20 receiversLoki via loki.writeany OTel exporter

The two Collector receivers go in a deployment-mode Collector (OpenTelemetry Operator). k8s_objects was k8sobjects before v0.151.0.

3.One collector, never a DaemonSet

  • Each replica of a per-node agent that watches Events ships every Event.
  • Use one Deployment, leader election, or Alloy clustering.
  • Clustered on all namespaces, one Alloy peer collects : failover, not scale-out.
  • Alloy syntax : Loki and Alloy.
Alloy : all Events to Loki
loki.source.kubernetes_events "cluster" {
  job_name   = "integrations/kubernetes/eventhandler"
  log_format = "logfmt"
  clustering { enabled = true }
  forward_to = [loki.write.default.receiver]
}

4.Alerting : metric first, Event second

  • Prefer the state metric (Prometheus) : kube_pod_status_unschedulable == 1.
  • Events for causes with no status field : FailedMount, probe flaps, FailedScheduling detail, Flux errors.
  • Loki ruler (LogQL, Prometheus rule format) with a count threshold : Events are lossy.
  • No ruler.alertmanager_url, no alerts (Alertmanager).
Events API exporter / Alloy / OTel Loki Loki ruler needs alertmanager_url Alertmanager Slack Flux controller Event POST notification-controller drops repeats for 5 min Alert / Provider Slack
# ruler rule shape only, untested
count_over_time({job=...} | logfmt
  | type="Warning"
  | reason="FailedMount" [10m]) > 0
WantUse
CrashLoopBackOff..._waiting_reason
OOMKilled..._last_terminated_reason
why a volume failed to mountFailedMount Event

5.Flux events

  • Each Flux controller writes a Kubernetes Event and POSTs it to notification-controller.
  • That drops repeats for 5 min, then sends via Alert, Provider : notification.toolkit.fluxcd.io/v1beta3.
  • Normal is info, Warning is error. Read : flux events --for Kustomization/<name>.
Seen in practicePitfall : one exporter prints Events as JSON to its Pod log for a collector to ship. Its 5 s age limit plus throttled lookups can quietly discard more Events than it sends, and nothing alerts on them. Do better : alert on events_discarded, raise the age limit and QPS.
In short : To keep Events, export them with one collector into a log store. Alert on the state metric first, and on exported Events with a count threshold when the reason is the only signal.

Observability

Tracing with Tempo

1.Traces, spans, context

A span is one timed piece of work. A trace is the tree of spans that share one 16-byte trace ID.

  • Span : 8-byte span ID, parent span ID (none for the root), name, start and end.
  • Kind (server, client, producer, consumer, internal), status (unset, ok, error), attributes, events, links.
  • Resource says who made it : service.name, k8s.pod.name.
gateway api db traceparent traceparent one trace ID, three spans GET /order (root, no parent) api /order SELECT parent ID time start
The traceparent header

00 - 4bf92f3577b34da6a3ce929d0e0e4736 - 00f067aa0ba902b7 - 01

version - trace ID (16 bytes) - parent ID, the caller's span (8 bytes) - flags

Lowest flag bit = sampled. tracestate carries vendor keys. SDKs default to propagators tracecontext,baggage.

2.The pipeline

  1. The app makes spans : SDK, or an agent injected from an Instrumentation CR (OTel Operator).
  2. Each outgoing call carries traceparent. The next service adds a child span.
  3. Head sampling happens here, in the SDK.

Alloy syntax lives in Loki and Alloy. Platform parts emit spans too : kgateway, Knative, CoreDNS (Cluster DNS).

  1. The SDK batches and exports OTLP to a node agent (DaemonSet Collector or Alloy) that adds k8sattributes metadata.
  2. A gateway tier can tail-sample, then exports OTLP to Tempo's distributor on 4317.
  3. Tempo stores the spans. Grafana queries it on 3200.

3.Ports and protocols

PortWhatTrap
4317OTLP gRPCReceivers bind localhost since Tempo 2.7 : set 0.0.0.0:4317
4318OTLP HTTP /v1/tracesInjected Java, Node.js, Python agents (operator v0.160.0) send http/protobuf here
3200HTTP API (queries, /ready)Was 80 before 2.8
9095Internal gRPCserver.grpc_listen_port
Seen in practice

Pitfall : an Instrumentation CR points agents at :4317. The Java agent defaults to http/protobuf and the operator passes the endpoint, not a protocol. HTTP at a gRPC port : no crash, no failed probe, no telemetry, for months. Do better : 4318 for HTTP, and alert when spans stop.

4.Sampling

HeadTail
DecidesAt the root, from the trace ID (parentbased_traceidratio)After seeing most spans : keep errors and slow traces
CostCheap, but blind to which traces will failEvery span of a trace must reach one instance
WhereSDK or Instrumentation CRCollector tail_sampling processor (beta)
  • Tail defaults : decision_wait: 30s, num_traces: 50000. Policies like status_code, latency, probabilistic.
  • Scale with two tiers : a loadbalancing exporter with routing_key: traceID in front.
  • Sampling breaks links : an exemplar or log line can point at a dropped trace.
  • It shrinks span metrics : with 10% sampling, expect roughly 10% of the request rate.
In short : Every hop passes the trace ID in traceparent, apps export spans over OTLP to a collector, and the collector samples and forwards them to Tempo. Mind the ports : 4317 is gRPC, 4318 is HTTP.

1.Components

Tempo stores only traces, indexes nothing up front, and keeps Parquet blocks in object storage : cheap.

ComponentJobModes
DistributorReceives OTLP, Jaeger, Zipkin. Applies limits, shards by trace IDboth
Kafka-compatible logDurable write-ahead logmicro­services
Block-builderKafka to Parquet blocks in object storagemicro­services
Live-storeRecent traces, queryable within secondsboth
Query frontendShards a query into jobsboth
QuerierRuns jobs on live-stores and object storageboth
Backend scheduler, workerCompaction and retention. Only one scheduler should runboth
Metrics-generatorSpan metrics and service graphsoptional

2.Write path and read path

  • Kafka acks before the distributor answers : it is the durability point, so RF is 1 (2.x clusters used 3).
  • Three consumers read Kafka. Queriers read live-stores for recent data and object storage for the rest.
  • Monolithic (-target=all, the default) needs no Kafka and runs everything in one process. The generator remote-writes to Prometheus.
write path Collector / Alloy distributor Kafka metrics-generator block-builder live-store remote write Parquet Prometheus object storage read path Grafana query frontend querier recent older blocks

3.Storage and retention

  • 3.1 writes vParquet5, and will not start if told to write vParquet3.
  • local storage on a PVC (Volumes) is for tests and monolithic mode.
  • Retention key : backend_worker.compaction.block_retention, default 336h (14 days).
  • The chart's tempo.retention: 24h reaches only the scheduler key, which deletes nothing : a default chart 3.1.0 install keeps 14 days. Set tempo.backendWorker.compaction.block_retention too.

4.From 2.x to 3.x

2.x3.x
ingesterlive-store plus block-builder
compactorbackend scheduler plus worker
RF3 in clustersRF1, Kafka for durability
local-blocks processorlive-store serves recent data
flat overridesrefused unless overrides.enable_legacy_overrides: true

No downgrade from 3.x. TraceQL metrics history starts at the upgrade.

5.Installing on Kubernetes

  • grafana-community/tempo : a monolithic StatefulSet, 3.x since chart 3.0.0, persistence off by default.
  • tempo-distributed needs Kafka. The Tempo Operator v0.22.0 still targets 2.x.
  • Tempo has no auth : put a proxy in front if anything outside the cluster can reach it.
CommandWhat it does
helm install tempo grafana-community/tempo --version 3.1.0 -n observability -f values.yamlMonolithic Tempo
kubectl -n observability port-forward svc/tempo 3200Reach the API
curl -s localhost:3200/readyReadiness (probes)
curl -s -G localhost:3200/api/search --data-urlencode 'q={status=error}'TraceQL search
tempo-cli migrate config --mode=monolithic old.yaml > new.yaml2.x to 3.x config
In short : Tempo 3.x writes through Kafka (or in-process when monolithic), keeps recent traces in live-stores and the rest as Parquet in object storage. Set retention on the backend worker.

1.TraceQL

  • { conditions } selects spans, | chains a pipeline. A trace comes back if a spanset survives.
  • Intrinsics use a colon : span:duration, trace:duration, span:status.
  • Attributes use a dot : span.http.status_code, resource.service.name. Bare duration means span:duration.
  • Regex is fully anchored. Structure : >> descendant, > child, ~ sibling.
QueryFinds
{ resource.service.name = "frontend" && status = error }Failed spans in frontend
{ resource.service.name = "frontend" } >> { status = error }Errors anywhere below frontend
{ trace:duration > 5s }Slow traces
{ span.http.status_code >= 500 }Server errors
{ resource.service.name = "api" } | rate() by (span.http.status_code)Request rate by status (metrics)
{ span:name = "GET /:endpoint" } | quantile_over_time(span:duration, .99)p99 latency (metrics)
({ status = error } | rate()) / ({ } | rate())Error ratio (3.1 arithmetic)

2.TraceQL metrics

  • Spans to series : rate(), quantile_over_time(), by().
  • GA in 3.0. Alerting on them is experimental. 3.1 adds arithmetic.

3.Metrics from traces

The metrics-generator turns spans into series and remote-writes them to Prometheus (needs --web.enable-remote-write-receiver). Its processors do nothing until listed in overrides.defaults.metrics_generator.processors. Spans that ended over 30s ago are not counted.

SeriesLabelsAnswers
traces_spanmetrics_calls_totalservice, span_name, span_kind, status_codeRequest and error rate
traces_spanmetrics_latencysameLatency histogram
traces_service_graph_request_totalclient, server, connection_typeCalls between services
traces_service_graph_request_failed_totalsameFailed calls

Cardinality : span_name is usually the biggest driver; raw URLs or user IDs as dimensions explode it. The docs list ..._unpaired_spans_total and ..._dropped_spans_total service-graph series that Tempo never emits : do not alert on them.

4.Correlations in Grafana

  • Each jump is set on its own data source. serviceMap draws the service graph from Prometheus series. Trace to profiles (tracesToProfiles) is in Pyroscope.
  • Trace to logs : default time shift is 0, start with -2s / 2s. With Loki, the link shows only when a configured tag is on the span.
  • Exemplars need three switches : send_exemplars: true on the generator, Prometheus --enable-feature=exemplar-storage (else silently dropped), Grafana exemplarTraceIdDestinations.
  • Trace IDs go in Loki structured metadata, not labels (Loki and Alloy).
Metrics (Prometheus) Traces (Tempo) Logs (Loki) Profiles (Pyroscope) exemplar traceID tracesToLogsV2 tracesToMetrics derived field tracesToProfiles each arrow is set on one data source
Seen in practice

A monolithic Tempo from a rendered chart (Flux) on a local volume, a two-tier OTel Collector, no tail sampling, the generator feeding a RED dashboard. Pitfall : exemplars are sent, but Prometheus runs without exemplar-storage and Grafana has no exemplarTraceIdDestinations, so metric to trace is quietly broken. Do better : set all three switches and click one exemplar through to its trace.

In short : Find traces with TraceQL, turn spans into RED metrics with the metrics-generator, and wire each jump in Grafana on its own data source. Exemplars need all three switches.

Observability

Continuous Profiling with Pyroscope

1.The fourth signal

  • Continuous profiling samples running code all the time, in production, and stores the stacks. Later you ask : which function used the CPU or memory at 03:12 ?
  • A one-off pprof grab only answers "now, on this one instance".
  • A hot Pod in kubectl top (Metrics Server) is the cue to open its profile.

Overhead : the docs say about 2 to 5%. That is the vendor's own estimate, with no published method.

QuestionSignal
Is it slow or failing ?metrics, Prometheus
What happened ?logs, Loki
Which request, which hop ?traces, Tempo
Which line of code ?profiles, Pyroscope

2.Profiles and flame graphs

  • A pprof profile is gzip-compressed protobuf : a list of samples. Each is a stack of locations plus one value per sample type (CPU nanoseconds, bytes allocated).
  • Pyroscope names a type <name>:<sample_type>:<sample_unit>:<period_type>:<period_unit>. CPU time : process_cpu:cpu:nanoseconds:cpu:nanoseconds.
Reading a flame graph root (all samples) handler gc json encode db query depth = stack
  • Width is the share of the total. Depth is the call stack. The x axis is not time.
  • Self = time in the function itself. Total = with its callees.
  • Alloc rising and never falling suggests a leak.

3.Profile types

TypeAnswersGo endpoint
CPUwhere CPU time goes/debug/pprof/profile?seconds=N
wallwhere time goes, waiting includednone (Java, .NET, Node.js)
allocwho allocates, and how much/debug/pprof/allocs
inusewhat is live right now/debug/pprof/heap
goroutinehow many goroutines, and where/debug/pprof/goroutine
mutex, blocklock and channel contentionoff until runtime.Set­Mutex­Profile­Fraction / Set­Block­Profile­Rate

4.Four ways in

Pod /debug/pprof node kernel app + SDK OTel Collectorprofiling receiver Alloy pyroscope.scrape Alloy pyroscope.ebpf(DaemonSet) pyroscope.receive_http pullevery 15s samples push pyroscope.write Pyroscope :4040 push, direct OTLP, alpha
  • Code change : pull needs pprof exposed, SDK push adds a library, eBPF and OTLP need none.
  • Privileges : eBPF needs root and host PID, OTLP needs root for eBPF. Pull and SDK push need none.
  • Maturity : pull, eBPF and SDK push are GA. OTLP is Alpha.
PulleBPFSDK pushOTLP
Profile typesCPU, memory, goroutine, mutex, blockCPU onlyrichest per languageCPU
Custom labelsPod labelsPod labelsyes, dynamicresource attributes
In short : Profiles are sampled stacks with values, drawn as flame graphs where width is the share, not time. Pull, eBPF, SDK push and OTLP are four ways to get stacks to Pyroscope with a good service_name.

1.v2 storage

distributor segment-writer metastore (Raft)the only stateful part register compaction-worker jobs query-frontend which blocks ? query-backend plan object storage (segments, then blocks) segment every 500ms merge read directly

Pyroscope 2.0 (2026-04-20) made v2 storage the default : no ingesters and no local disks on the write path. Durability comes from object storage.

Ports4040 HTTP API, UI, ingest and OTLP (gRPC and HTTP; the gRPC client must speak cleartext HTTP/2, and it is not 4317 as in Tempo). 9095 internal gRPC. 7946 memberlist. 9099 metastore Raft.

2.Components

ComponentStateJob
distributorstatelessvalidates, routes by tenant, then by service_name
segment-writerstatelessbuffers, writes a segment every 500ms, client waits until durable and indexed. A StatefulSet, no volumes
metastorestatefulRaft index in 6-hour windows. Run 3 (1 failure) or 5 (2 failures)
compaction-workerstatelessmerges segments into blocks
query-frontendstatelessplans the read from metastore metadata
query-backendstatelessreads object storage directly, no cache layer

3.Single binary vs microservices

  • Single binary (-target=all, the default) runs every role, with a one-member metastore. It may use the local filesystem backend on a PVC.
  • Microservices need object storage; filesystem is single node only.
v2 or dual ? The chart renders plain v2 since chart 2.0.2. The bare binary still defaults to v1-v2-dual : writes go to v2, old v1 data stays readable. v1 (ingesters) is deprecated, and v1 knobs like pyroscopedb.max-block-duration do nothing on v2.

4.Defaults that bite

SettingDefaultDo
pyroscope.persistence.enabledfalse : profiles lost on restartenable it
extraArgs.log.leveldebugset info
retention vs querier.max-query-lookback31 days kept, 7 queryable, older clippedraise the lookback
querier.max-query-length1done query covers a day at most
ingestion limit4 MB/s, burst 2 MB, then HTTP 429use Alloy 1.19+ (retries 429)
CommandWhat it does
helm -n observability install pyroscope grafana/pyroscope --set pyroscope.persistence.enabled=truesingle binary with a PVC
kubectl -n observability port-forward svc/pyroscope 4040open the UI
curl -s localhost:4040/readyreadiness
profilecli query series --query='{service_name="checkout"}'list series
profilecli upload --extra-labels=service_name=payments ./cpu.pprofupload a pprof file
In short : Pyroscope v2 writes segments straight to object storage and keeps only its Raft metastore stateful. Turn on persistence, quiet the logs, and remember profiles are kept 31 days but only 7 are queryable.

1.Annotation pull

  • Annotate the Pod template. .port or .port_name is required : the Helm doc's named-port fallback is not in the template.
  • These keys are the chart's embedded Alloy. Plain Alloy reads pyroscope.io/service_name.
  • service_name is the main label (UI, distributor); a missing one is unspecified.
# checkout Deployment, pod template
annotations:
  profiles.grafana.com/cpu.scrape: "true"
  profiles.grafana.com/cpu.port_name: http-metrics
  profiles.grafana.com/memory.scrape: "true"
  profiles.grafana.com/memory.port_name: http-metrics
  profiles.grafana.com/service_name: "checkout"
# containers[].ports: name http-metrics, 8080
# the app imports net/http/pprof

2.eBPF DaemonSet

  • pyroscope.ebpf samples every CPU at 19 Hz (since Alloy 1.11.0; 97 before) and sends every 15s.
  • Root in the host PID namespace (privileged and hostPID, or a capability list), so its namespace needs the privileged level (Pod Security).
  • One Pod per node (DaemonSet), local Pods by spec.nodeName, tolerations for control-plane nodes.
  • CPU only (off-CPU sampling exists, off by default).
  • Stripped binaries show module names only; very shallow stacks mean no frame pointers.
  • Never set a CPU limit on a profiler : throttling drops samples when the node is busiest.
  • Alloy syntax is taught in Loki and Alloy.

3.From a slow span to its code

  • Wrap the tracer provider (otelpyroscope.NewTracerProvider(tp) in Go). Samples in the local root span get span_id, trace_id; the span gets pyroscope.profile.id.
  • CPU (Go, Ruby, .NET, Python); CPU or wall (Java). Spans under about 10ms may have no samples.
  • The Tempo data source's tracesToProfiles (datasourceUid, profileTypeId, tags) adds Profiles for this span (span start minus 60s to end plus 60s).
  • Map service.name (from the OTel Operator or SDK) to service_name. No id or tags, no link (others : Tempo).
trace in Tempo span (pyroscope.profile.id) Profiles for this span Pyroscope, filtered by span_id flame graph of that span

4.OTel profiles and the gotchas

OTel Profiles : public Alpha since 2026-03-26, proto still v1development; Pyroscope calls OTLP ingest "suitable for development and testing". Relabel the eBPF profiler's process.executable.name to service_name at ingest.

SymptomCauseFix
no profiles from an annotated Podno .port or .port_nameadd one
a new "service" every rolloutservice_name built from the Pod nameuse namespace and container
no memory profileseBPF is CPU onlyuse pull or an SDK
empty mutex and blockoff in the Go runtimecall Set­Mutex­Profile­Fraction / Set­Block­Profile­Rate
nothing older than 7 days7-day query lookback (data kept 31)raise max-query-lookback
Seen in practice Single-binary Pyroscope on a local-path volume (one node), the chart's Alloy scraping only itself, and a privileged eBPF Alloy DaemonSet profiling every Pod's CPU, all rendered charts under Flux CD v2. Pitfalls : CPU limits throttle both Alloys; an eBPF service_name built from Pod names churns per rollout; traces-to-profiles left unwired. Do better : no CPU limit on profilers, a stable service_name per workload, span profiles.
In short : Annotate Pods for pull, run eBPF as a privileged DaemonSet with no CPU limit for CPU everywhere, and use an SDK when you need more types or span profiles. Map service.name to service_name so a slow span leads to its flame graph.

Part 3Advanced Concepts

Parts 1 and 2 cover what to run. Part 3 looks inside: the loops behind every object, the API you can extend, the checks on every request, identity, sizing and scaling, how images, packets and volumes reach a Pod, and how clusters run from Git.

Objects you use
Control loops
APIs you add
Authorize
Admit
Store
Schedule
Why Part 3 ?
  • Every object has a loop behind it
  • The API can learn new kinds
  • Every write is checked before it is saved
  • Every image can be traced and verified
  • Policy decides who acts and where Pods run
  • Autoscalers size Pods and replicas
  • GPUs, volumes and clusters are objects too
  • A Git repo can drive the whole cluster
“Objects say what you want. Controllers make it true.”

Extending the API

Controllers and Reconciliation

1.What is a controller ?

A loop that keeps one kind of object true. You write the desired state in spec. The controller moves the world toward it and reports in status.

  • Read the desired state (spec) from the API server.
  • Observe the current state.
  • Act to close the gap, then report back in status.
  • Most built-in controllers act only through the API server. The Job controller never runs a container. It asks for Pods, and kubelets run them.

2.Level, not edge

  • Reconcile from what the object looks like now, never from the event that woke you.
  • spec.replicas goes 2  5  3 ? The system does not have to pass through 5.
  • Your controller may have been down while things changed. Edge triggers and watches only make the loop faster.
for { desired := get(); current := observe(); makeChanges(desired, current) }

3.The control loop

  • One pass handles one key, namespace/name. It is idempotent : create if missing, update if different, do nothing if converged. Safe when it runs twice, runs late or reads a stale cache.
  • Open world : users, the HPA and the garbage collector change things too.
  • A child's watch event queues its owner's key, via ownerReferences (Owners and GC).
  • Outside systems (cluster autoscaler, cloud routes) : desired state from the API, status back to it.
You / GitOps apply spec (desired) API server : generation +1 read the current value, not the change status observedGeneration + conditions written via /status reconcile(key) one key, one pass observed cache + world create / update / delete only if different children or an external system watch event

Each pass compares whole states. A missed event does no harm.

4.Spec in, status out

# kubectl get deploy web -o yaml (trimmed)
metadata:
  generation: 7        # +1 on a spec change
  resourceVersion: "184213"   # opaque
spec:
  replicas: 3          # desired, written by you
status:                # observed, via /status
  observedGeneration: 7   # == generation
  availableReplicas: 3
  conditions:
  - type: Available
    status: "True"
    reason: MinimumReplicasAvailable
    lastTransitionTime: "2026-10-06T14:02:11Z"

5.Status rules

  • generation : set by the API server. Deployments bump it on spec or annotation changes, CRDs with a status subresource on any change except metadata and status.
  • observedGeneration : the generation the controller acted on. Lower means the status is stale.
  • conditions : keyed by type. status True, False or Unknown, a CamelCase reason (required), lastTransitionTime = last flip. Missing means Unknown.
  • Write status only through /status. PUT and POST on the main resource ignore it.
  • 409 Conflict : someone wrote first. Re-read and retry (return the error to requeue).

6.Commands

CommandShows
kubectl get deploy -o custom-columns=NAME:.metadata.name,GEN:.metadata.generation,OBSERVED:.status.observedGenerationOBSERVED < GEN : not caught up yet
kubectl get deploy web -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'Each condition with its reason
kubectl wait --for=condition=Available deploy/web --timeout=120sBlocks until the condition is True
kubectl rollout status deploy/webobservedGeneration first, then replicas
kubectl get pods -w --output-watch-eventsThe event stream an informer gets
kubectl events --for deployment/webEvents recorded while reconciling
In short : A controller is a level-based loop : it reads spec, compares it with what exists, changes only the difference, and reports observedGeneration and conditions in status.

1.Inside a client-go controller

kube-apiserver shared informer LIST (or stream), then WATCH Reflector deltas RealFIFO store Indexer local cache notify Event handlers Add(key) Workqueue namespace/name keys Get() Workers x N one key reconcile(key) your code Lister Get() read only, no API call requeue on error writes, incl. /status

client-go gives you the informer. You write the handlers, the queue wiring and reconcile.

  • Informer = Reflector + RealFIFO (DeltaFIFO before 1.33) + Indexer, one per type per process via a SharedInformerFactory.
  • Reflector : LIST, then WATCH from its resourceVersion. On 410 Gone it relists (API Machinery).
  • Lister : reads the cache, never the API server.
  • factory.Start(), then cache.WaitForCacheSync(), only then the workers.
  • Handlers only Add keys, the owner's key for a child (metav1.GetControllerOf()). Deletes may arrive as a DeletedFinalStateUnknown.

2.The workqueue

  • Fair (FIFO) and stingy : a key added many times before a worker gets it runs once.
  • Two workers never hold the same key at once.
  • Re-added while processing ? It runs again after Done().
  • N workers = N goroutines calling Get().
CallWhen
Done(key)Always, in a defer
Forget(key)Success : reset its backoff
AddRateLimited(key)Error : retry with backoff
AddAfter(key, d)Recheck after d

Default limiter : max of per-key 5ms x 2n (cap 1000s) and a 10 qps, burst 100 token bucket.

3.kube-controller-manager

kube-controller-manager (kcm) is one binary with 49 built-in loops in v1.37, each with its own workers (Architecture). --controllers defaults to * (every on-by-default loop), and -name turns one off. Three are off by default : bootstrap-signer, selinux-warning and token-cleaner.

--controllers nameReads, then managesTuning flag (default)
deployment-controllerDeployment  ReplicaSets--concurrent-deployment-syncs (5)
replicaset-controllerReplicaSet  Pods--concurrent-replicaset-syncs (5)
daemonset-controllerDaemonSet  one Pod per eligible node--concurrent-daemonset-syncs (2)
job-controllerJob  Pods, then marks the Job finished--concurrent-job-syncs (5)
endpointslice-controllerService  EndpointSlices--concurrent-service-endpoint-syncs (5)
namespace-controllerTerminating ns  deletes all inside--concurrent-namespace-syncs (10)
garbage-collector-controllerownerReferences  dependents of gone owners--concurrent-gc-syncs (20)
node-lifecycle-controllerNode Leases  node health--node-monitor-grace-period (50s)

4.Leader election on a Lease

  • Replicas race to update one coordination.k8s.io/v1 Lease. resourceVersion lets exactly one write win.
  • The leader keeps updating renewTime. Standbys wait leaseDurationSeconds after they last saw it change (own clock).
  • kcm defaults : --leader-elect=true, lease 15s, renew deadline 10s, retry 2s.
  • Lost the Lease ? kcm exits on purpose and the kubelet restarts it.
kcm-a (leader) OnStartedLeading controllers leader only kcm-b (standby) kcm-c (standby) Lease kube-system/ kube-controller-manager holderIdentity renewTime renew every 2s read 15s with no renew : standbys race, one write wins, rest get 409

holderIdentity is <hostname>_<uuid>. Not fencing : keep reconcile idempotent.

In short : Informers share one cache per type, handlers only queue keys, workers reconcile one key at a time with backoff, and one kube-controller-manager replica leads through a Lease.

Extending the API

Custom Resources (CRDs)

1.What is a CRD ?

A CustomResourceDefinition teaches kube-apiserver a new kind, with its schema, validation, defaults and versions. The API server serves and stores the objects itself.

  • The CRD is cluster-scoped (apiextensions.k8s.io/v1). Its metadata.name must be <plural>.<group>.
  • spec.scope (Namespaced or Cluster) is for the custom objects, not the CRD.
  • A CRD only stores data. A controller acts on it (Operators).
kubectl 1. apply crd.yaml 2. get ct kube-apiserver CRD handler CRD status NamesAccepted, Established Discovery kind, plural, shortNames OpenAPI v3 schema read by kubectl explain REST path /apis/stable.example.com/v1/ namespaces/*/crontabs storage version etcd publishes serves

Use the kind once the CRD is Established.

CRD or aggregated API ?CRD : kube-apiserver serves it and stores it in etcd, with no server to run and validation from the schema and CEL. Aggregated API : an APIService routes /apis/<group>/<version> to your own server (like metrics.k8s.io from metrics-server), which picks its own storage, verbs and subresources. Start with a CRD.

2.A CRD in YAML

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com # plural.group
spec:
  group: stable.example.com
  scope: Namespaced       # of the CronTabs
  names:
    kind: CronTab
    plural: crontabs      # URL segment
    shortNames: ["ct"]
    categories: ["all"]   # kubectl get all
# then, an object of the new kind
apiVersion: stable.example.com/v1
kind: CronTab
metadata: {name: nightly}
spec: {cronSpec: "0 2 * * *", image: "backup:1.4"}
  versions:
  - name: v1
    served: true
    storage: true         # exactly one
    subresources:
      status: {}
      scale:
        specReplicasPath: .spec.replicas
        statusReplicasPath: .status.replicas
    additionalPrinterColumns:
    - name: Image
      type: string
      jsonPath: .spec.image
      priority: 1         # only in -o wide
    schema:
      openAPIV3Schema: {...}  # structural
  conversion:
    strategy: None        # the default

3.Field by field

FieldRule / defaultPurpose
names.singular, names.listKinddefault lowercased kind, <Kind>ListCLI alias, kind of list responses
names.shortNames, names.categoriesoptional, lowercasekubectl get ct, kubectl get all
versions[].served, storagerequired, exactly one storage: trueserve over REST, version written to etcd
versions[].schema.openAPIV3Schemastructuralvalidation, pruning, defaulting
versions[].subresourcesstatus: {} and/or scale/status for controllers, /scale for kubectl scale and HPA (labelSelectorPath)
versions[].additionalPrinterColumnsnone = NAME, AGEkubectl get columns
versions[].selectableFieldsmax 8--field-selector on custom fields
versions[].deprecateddefault falsewarning header on requests
spec.conversion.strategyNone (default) or Webhookhow objects convert between versions
status.storedVersionsAPI server adds, migration trimsevery version ever stored
In short : A CRD adds a kind (names, scope, versions, schema) that kube-apiserver serves.

1.Structural schema and pruning

  • Structural (a must in v1) : a type on every node, no type or default inside allOf, anyOf, oneOf, not.
  • Pruning drops unknown fields (API warns, kubectl rejects). x-kubernetes-preserve-unknown-fields opts a subtree out.
  • default: on request, on read and after a webhook patch. nullable: true keeps a null.
  • x-kubernetes-list-type (atomic by default, set, map) and x-kubernetes-list-map-keys tell server-side apply how to merge a list.
Requestserved version Decode, pruneunknown fields Defaultsrequest version Mutatingadmission Defaults againif webhook patched Validationschema + CEL ratcheting on update Validatingadmission Convertto storage version etcd

Dashed : admission. On read : storage-version defaults, then conversion.

2.CEL validation rules

image:
  type: string
  maxLength: 256        # bounds the CEL cost
  x-kubernetes-validations:
  - rule: "self == oldSelf"   # transition
    message: "image is immutable"
replicas: {type: integer, default: 1}

A rule also takes messageExpression, reason, fieldPath, optionalOldSelf.

  • x-kubernetes-validations holds CEL rules (Stable 1.29). self is the node the rule sits on.
  • Uses oldSelf ? It is a transition rule, run only on updates where old and new both exist.
  • optionalOldSelf: true (Stable 1.33) runs it on create too.
  • Ratcheting (Stable 1.33, on) passes an update if the invalid part did not change.

3.Versions and conversion

  • Exactly one version has storage: true. Each has its own schema, subresources and columns.
  • None (default) only rewrites apiVersion. A renamed or split field needs a Webhook.
  • status.storedVersions lists every version ever stored. Drop a version only once it has left the list.
  • v1.37 : StorageVersionMigration is GA and on by default. It rewrites every object at the storage version.
Client Aasks for v1beta1 Client Basks for v1 kube-apiserver ConversionReview Conversion webhook :443, https writes at v1 etcd objects at v1beta1 and v1 storedVersions [v1beta1, v1], then [v1] StorageVersionMigration GA in v1.37 rewrites at v1 once Succeeded

Changing storage: true never rewrites old objects. Real example : Gateway API CRDs (channels, v1beta1 to v1, storedVersions).

4.Commands

CommandDoes
kubectl wait --for=condition=Established crd/<name>Block until the new endpoint is served
kubectl api-resources --api-group=stable.example.comKinds, shortNames and scope the CRD added
kubectl explain crontabs.spec --api-version=stable.example.com/v1Field docs from the published OpenAPI v3 schema
kubectl apply --dry-run=server -f my-crontab.yamlFull server check (schema, CEL, admission), nothing saved
kubectl get ct -o widePrinter columns, plus those with priority > 0
kubectl get crd <name> -o jsonpath='{.status.storedVersions}'Versions that may still have objects in etcd
Gotchas
  • Deleting a CRD deletes every object of that kind.
  • With status on, a normal update ignores .status. Use --subresource=status.
  • A failing conversion webhook can block reads, updates, even deletes.
Good to know
  • Most RBAC roles miss new kinds. Add Roles.
  • *.k8s.io and *.kubernetes.io groups need the api-approved.kubernetes.io annotation.
  • CEL over budget rejects the CRD. Bound it with maxItems, maxProperties, maxLength.
In short : A structural schema lets the API server prune, default and check every write with CEL. Versions share one storage version, and a StorageVersionMigration moves old objects to it.

Extending the API

Operators

1.What is an operator ?

An operator is CRDs + a custom controller. It turns a human operator's runbook (install, upgrade, backup, failover) into a reconcile loop that runs forever.

  • k8s.io : "software extensions to Kubernetes that make use of custom resources to manage applications and their components." The idea came from a CoreOS blog post.
  • Every operator is a controller, not every controller is an operator. The Deployment controller is generic. CloudNativePG knows Postgres : promotion, WAL archiving, backups.

2.What it automates

CRDs + controller + runbook = Operator new kinds watch, diff, act ops know-how runs forever
Jobs k8s.io lists
  • Deploy an app on demand
  • Take and restore backups
  • Upgrades with schema or config changes
  • Publish a Service for apps that cannot use the Kubernetes API
  • Simulate failure to test resilience
  • Leader election for apps that have none

3.Anatomy (k8s.io SampleDB)

Deploy it like any app : CRDs plus a controller Deployment, outside the control plane.

  • Users touch only the CR : kubectl edit SampleDB/example-database.
  • You write .spec (a change bumps metadata.generation). The operator writes .status.conditions with observedGeneration.
  • Children carry an ownerReference, so a StatefulSet change re-runs reconcile.
  • Delete the SampleDB : a finalizer takes a snapshot, then the StatefulSet and volumes go.
You / Flux apply .spec SampleDB CR .spec : you .status : operator kube-apiserver every write goes through here watch SampleDB + owned kinds writes .status Operator Pod controller-manager status change re-runs it (Owns) create / update + ownerRef StatefulSet PVCs Init Job Backup Pod Upgrade Job first run on schedule old version

Every write goes through kube-apiserver. Arrows to the children are shorthand.

4.Capability levels

L5 Auto Pilotauto-scale, self-heal, auto-tune L4 Deep Insightsmetrics, alerts, events L3 Full Lifecyclebackup, restore, failover L2 Seamless Upgradesoperand + operator upgrades L1 Basic Installinstall via CR, ready in status L1 to L2 L1 to L5 L1 to L5 Helm Ansible Go
In these notesLevel*
CloudNativePG5 Auto Pilot
External Secrets4 Deep Insights
OTel Operator4 Deep Insights
cert-manager3 Full Lifecycle
Knative Operator2 Seamless Upgrades
Flux CD2 Seamless Upgrades

* OperatorHub level, self-declared by the publisher in the CSV (metadata.annotations.capabilities). Read the docs before you trust it.

5.Write one, or not ?

You needReach forWhy
Install / upgrade a stateless appHelm chart + Flux HelmReleaseLevels 1-2, no Go code to maintain
App reads a config fileConfigMapk8s.io : a file a Pod reads is not a CR
Logic on built-in kindsPlain controller, --resource=falseNo new API needed
Validate or default fieldsCRD schema + CEL, or admission policyNo webhook (CRDs, Admission)
Parent-to-children mappingMetacontrollerOne JSON webhook, it does the watching
Non-declarative APIA stand-alone APIk8s.io : CRDs and aggregation fit declarative only
It already existsInstall itSomeone else carries the pager
Backup, failover, ordered upgradesWrite an operatorLevel 3+ is why operators exist
In short : An operator is CRDs plus a controller that runs a runbook as a reconcile loop : you write .spec, it owns the children and reports .status. Write one only for level 3+ work.

1.One reconcile pass

Request : ns/name Get CR return nilNotFound : gone being deleted ?deletionTimestamp set yes Cleanupfinal backup no Add finalizer Remove finalizer CreateOrUpdate children+ SetControllerReference list empty API serverdeletes the CR Set status conditions periodic work return RequeueAfter return errrequeue with backoff

Red dashed : a step failed, Result ignored, retry with backoff. Each pass reads the current state, never "what changed". Finalizer and ownerReference rules : Owners, Finalizers and GC.

2.The calls (controller-runtime v0.25)

func (r *SampleDBReconciler) Reconcile(ctx context.Context,
    req ctrl.Request) (res ctrl.Result, err error) {
  var db dbv1.SampleDB
  if err = r.Get(ctx, req.NamespacedName, &db); err != nil {
    return res, client.IgnoreNotFound(err) } // gone : nil
  if db.DeletionTimestamp != nil { // final backup first
    controllerutil.RemoveFinalizer(&db, fin)
    return res, r.Update(ctx, &db) }
  if controllerutil.AddFinalizer(&db, fin) {
    return res, r.Update(ctx, &db) }
  sts := child(&db) // *appsv1.StatefulSet, name + namespace
  _, err = controllerutil.CreateOrUpdate(ctx, r.Client, sts,
    func() error { // set sts.Spec, then own it
      return ctrl.SetControllerReference(&db, sts, r.Scheme) })
  if err != nil { return res, err }
  meta.SetStatusCondition(&db.Status.Conditions, ready(&db))
  res.RequeueAfter = time.Hour // periodic work
  return res, r.Status().Update(ctx, &db)
}

Builder : For(&dbv1.SampleDB{}).Owns(&appsv1.StatefulSet{}), a child change requeues. Requeue: true is deprecated, reconcile.TerminalError(err) stops retries.

3.Pick a toolkit

Tool (language)LatestYou writeUsed here / notes
controller-runtime (Go)v0.25.2Reconcile(ctx, req) (Result, error)Flux CD, CNPG, ESO
Kubebuilder (Go)v4.16.0Types, markers, ReconcileCNPG, ESO, OTel
Operator SDK (Go, Ansible, Helm)v1.42.3Go via Kubebuilder, a playbook or a chartOTel plugins. Helm type : level 2 max
Java Operator SDK (Java)v5.6.2A Reconciler class (fabric8 client)Quarkus extension
Kopf (Python 3.11+)1.44.6@kopf.on.create, timer, daemonFeature-complete, stable
kube-rs (Rust)4.2.0async fn reconcile(obj, ctx)In kube::runtime
Metacontroller (any)v4.17.2A JSON sync webhookAPI still v1alpha1
OLM v1 operator-controllerv1.12.0A ClusterExtension : package + channelOperatorHub.io catalog, cluster-admin

4.Build, test, ship (Kubebuilder)

CommandDoes
kubebuilder init --domain <domain> --repo <go-module>Scaffold the project and a Kind e2e suite
kubebuilder create api --group db --version v1alpha1 --kind SampleDBCRD Go type, controller, envtest suite
make manifests generateCRD YAML + ClusterRole from markers, DeepCopy
make install runCRDs in, controller runs on your laptop
make testenvtest : etcd + kube-apiserver from setup-envtest (KUBEBUILDER_ASSETS)
make test-e2eKind cluster + Ginkgo, deleted on pass
make docker-build docker-push IMG=<image> && make deploy IMG=<image>Push image, deploy CRDs, RBAC, Deployment

5.Gotchas & good to know

Gotchas
  • Finalizer deadlock : remove the operator before its CRs and they hang in Terminating.
  • ownerReferences cannot cross namespaces (OwnerRefInvalidNamespace events).
  • envtest has no controller-manager : no GC, no Pods. Assert ownerReferences, .status.
Good to know
  • Keep --leader-elect on before replicas > 1.
  • Upgrade the operator first. It only knows operands up to its own version.
  • Operator down : the app keeps serving, only automation pauses.
  • OLM v1 installs, then upgrades along the channel.
In short : Reconcile is an idempotent pass : get, finalize, create-or-update owned children, write status. Scaffold with Kubebuilder, test on envtest, ship with kustomize, OLM v1 or Helm.

API machinery

API Machinery and Server-Side Apply

1.Resources, kinds and URLs

  • Resource vs kind : the URL uses the resource (deployments), the body the kind (Deployment).
  • Core group : /api/v1, apiVersion v1. Named : /apis/GROUP/VERSION, apiVersion apps/v1.
  • Identity = group + resource + namespace + name. v1beta1 and v1 are two views of one stored object, so names clash.
/apis /apps /v1 /namespaces/prod groups group version namespace /deployments /web /status resource name subresource bodyapiVersion: apps/v1, kind: Deployment
Discovery/api and /apis list what is served. Aggregated discovery (stable since 1.30) returns every group in 2 requests. Schemas at /openapi/v3.

2.A request, step by step

  • Every request : authentication (401), audit and impersonation, APF (429), authorization (403, RBAC and Service Accounts).
  • Writes : mutating admission, object schema validation (422), validating admission (Admission Control), then etcd. Stale resourceVersion : 409.
  • Unknown fields are dropped with a Warning. fieldValidation=Strict makes them a 400.
  • Reads skip admission and come from the watch cache. While it is cold, watches and unpaginated or filtered lists get 429 with Retry-After.
kubectl / client-go Authentication 401 Audit + impersonation APF (FlowSchema, level) 429 Authorization 403 writes Mutating admission Schema validation 422 Validating admission etcd 409 reads : get, list, watch skip admission Watch cache serves reads change stream

3.resourceVersion and conflicts

  • resourceVersion (RV) : a string, new on every write. Pass it back unmodified. Compare as integers only within one group + resource (required since 1.35).
  • PUT sends metadata.resourceVersion. Stale ? 409 Conflict. GET again, reapply your change, retry.
  • Patches skip this check unless they carry the RV or a JSON Patch test op.
FieldChanges on
resourceVersionEvery write, status and labels too
generationDesired state (spec) changes
observedGenerationSet in status by the controller. Equal to generation : status is current

How controllers use observedGeneration : Controllers and Reconciliation.

In short : Every request passes authentication, APF and authorization, and writes also pass admission and validation. resourceVersion changes on every write, so a stale PUT gets 409, while generation moves only when the desired state does.

1.List, then watch

  • A LIST returns the items and the collection's RV. WATCH from there, not from an item's RV.
  • etcd keeps about 5 min of history. Older RV : 410 Gone. Clear the cache, re-list.
  • Bookmarks carry only a fresh RV for reconnects. No interval promise.
  • Pages : limit=N returns metadata.continue. All pages share one RV. Tokens expire in about 5 min.
  • Streaming list (WatchList, beta, on) : one WATCH with sendInitialEvents=true sends the state as ADDED events, then a BOOKMARK.
Client Reflector / informer kube-apiserver served from watch cache 1. LIST ?limit=500 items + RV 10245 + continue token 2. LIST &continue=... until empty 3. WATCH ?resourceVersion=10245 ADDED / MODIFIED / DELETED BOOKMARK : fresh RV only 4. dropped ? re-WATCH from last RV 410 Gone : RV compacted (~5 min) clear the cache, back to 1

2.Ways to write an object

MethodHTTP / Content-TypeConflictsUse for
updatePUT, the whole object409 if metadata.resourceVersion is staleRead-modify-write, you write the retry loop
JSON PatchPATCH application/json-patch+jsonOnly via a test op or the RVExact path ops, compare-and-set
JSON Merge PatchPATCH application/merge-patch+jsonNone, last write winsSmall partial edits
Strategic Merge PatchPATCH application/strategic-merge-patch+jsonNoneBuilt-ins only, never CRDs. Superseded by SSA
Server-Side ApplyPATCH application/apply-patch+yaml ?fieldManager=409 on another manager's field unless force=trueDeclarative config, controllers, create-or-update

3.Server-Side Apply (SSA)

  • Stable since 1.22. The body is your partial intent : it creates the object if missing, patches it otherwise.
  • managedFields lists each manager, its operation (Apply or Update) and the fields it owns.
  • Apply over another manager's field : 409 unless force=true. Update never conflicts, it takes ownership. Same value : shared.
  • A field you drop is removed only if no one else owns it. Controllers should always force.
  • Lists merge by x-kubernetes-list-type (atomic, set, map). CRD lists are atomic unless marked (Custom Resources (CRDs)).
1. you apply 3 2. controller sets 5 3. apply 3 again replicas: 3 kubectl owns it operation : Apply replicas: 5 controller owns it Update, no conflict 409 Conflict field owned by the controller A : force --force-conflicts you own it alone B : drop it remove replicas give up ownership C : match apply the live 5 shared ownership
Gotchas
  • A renamed manager stays on as a second owner.
  • Drop replicas before the HPA owns it : it resets to 1.
  • Clear managedFields with [{}], not [].
In short : Watch from the list's RV and re-list on 410. A PUT checks resourceVersion, most patches do not, and SSA tracks an owner per field instead : apply over another manager's field and you get 409.

1.API Priority and Fairness

  • Stable since 1.29. Each apiserver has 600 seats : 400 inflight + 200 mutating.
  • Priority levels split them by nominalConcurrencyShares, and lend and borrow.
  • FlowSchemas match by ascending matchingPrecedence. First match wins.
  • A flow = FlowSchema + distinguisher (ByUser, ByNamespace). Shuffle sharding picks its queue.
  • A LIST costs seats by object count. exec and logs bypass APF.
Request user, verb, ns FlowSchema first match wins Priority level Limited or Exempt Exempt : no queue Limited Exempt level system:masters Queues shuffle sharding Flow user or namespace seat frees up full or waited Execute holds its seats 429 Too Many Requests
Gotchas
  • Edits to suggested objects revert each minute unless annotated apf.kubernetes.io/autoupdate-spec: "false".
  • Replicas under one ServiceAccount are one flow.

2.Deprecation and version skew

API level or componentRule
Alpha APIOff by default. Can be removed in any release without notice
Beta APINew betas off by default since 1.24. Still served 9 months or 3 releases after deprecation, whichever is longer
GA APICan be deprecated, never removed within v1
kube-apiserver (HA)Instances within 1 minor of each other
controller-manager, schedulerNot newer than kube-apiserver, up to 1 older
kubelet, kube-proxyNot newer than kube-apiserver, up to 3 older
kubectlWithin 1 minor of kube-apiserver, either way

3.Commands

CommandShows
kubectl api-resources -o wideDiscovery as a table : APIVERSION, NAMESPACED, KIND, VERBS
kubectl get pods -w --output-watch-eventsList, then watch, printing each event type
kubectl apply --server-side --field-manager=ci-cd -f deploy.yamlSSA as manager ci-cd (default kubectl). A conflict fails with 409
kubectl apply --server-side --force-conflicts -f deploy.yamlTake the conflicting fields, become sole owner
kubectl get deploy web -o yaml --show-managed-fieldsmanagedFields : who owns which field, Apply or Update
kubectl get flowschemas,prioritylevelconfigurationsAPF config
In short : APF queues each flow fairly, so one noisy client cannot starve the rest. Alpha APIs can vanish any time, a deprecated beta is served for at least 3 releases, GA stays for all of v1, and the kubelet may trail kube-apiserver by 3 minors but never lead it.

API machinery

Owners, Finalizers and Garbage Collection

1.Owners and dependents

The garbage collector (GC) deletes a dependent once all its owners are gone. The link lives on the dependent. Deployment, ReplicaSet, StatefulSet, DaemonSet, Job and CronJob write it for you :

ownerReferences:            # on the Pod
- apiVersion: apps/v1
  kind: ReplicaSet
  name: nginx-deployment-6b474476c4
  uid: <owner uid>
  controller: true          # at most one
  blockOwnerDeletion: true  # default false
Deployment nginx-deployment ReplicaSet nginx-deployment-6b474476c4 Pod ...476c4-a Pod ...476c4-b ownerReferences[0] controller: true ownerReferences controller: true garbage collector kube-controller-manager watches every listable resource, builds the graph deletes it once ALL owners are gone the owner keeps no list of dependents

Arrows point from dependent to owner. Labels pick what a controller manages, owners pick what gets cleaned up.

2.Same namespace, or cluster-scoped

  • A reference has no namespace field, so the owner must sit in the dependent's namespace, or be cluster-scoped.
  • Owner in another namespace : the ref counts as absent, so the GC may delete your dependent.
  • Cluster-scoped dependent with a namespaced owner : never collected (v1.20+). Both raise a Warning event, OwnerRefInvalidNamespace.

3.Cascading deletion : three modes

Deleting an owner picks the order things disappear in. kubectl --cascade maps to DeleteOptions.propagationPolicy.

Mode, kubectl flagOwner getsWhat happens
Background (default)
--cascade=background
no finalizerOwner goes at once. The GC deletes the dependents afterwards. Everyday deletes.
Foreground
--cascade=foreground
foregroundDeletionOwner stays (deletionTimestamp set) until the GC has deleted each dependent with blockOwnerDeletion=true. For teardown that must not lose the owner first.
Orphan
--cascade=orphan
orphanThe GC strips the owner uid from each dependent, then the owner goes. Dependents keep running, so you can swap a controller without killing its Pods.

A raw API DELETE of a batch/v1 Job defaults to orphan. orphanDependents is deprecated. blockOwnerDeletion only matters for foreground, and the plugin that polices it (OwnerReferencesPermissionEnforcement) is off by default in v1.37.

4.Other collectors

Who, knobWhat it collects
Job TTL controller
ttlSecondsAfterFinished (unset = keep)
A finished Job and its Pods, N seconds after it finishes (0 = at once). Stable v1.23
PodGC
--terminated-pod-gc-threshold (12500)
Succeeded / Failed Pods above the count
kubelet GC
  • Images, 5 min : disk over imageGCHighThresholdPercent (85), least recently used first, down to imageGCLowThresholdPercent (80). imageMinimumGCAge 2m, imageMaximumGCAge 0 (off).
  • Containers, 1 min : MinAge 0, MaxPerPodContainer 1, MaxContainers -1 (flags deprecated).

5.Commands

CommandWhat it does
kubectl get pod <pod> -o jsonpath='{.metadata.ownerReferences}'Lists the owners. "controller":true marks the controller
kubectl delete deployment nginx-deployment --cascade=foregroundOwner stays (foregroundDeletion) until its ReplicaSet and Pods are gone
kubectl patch job pi --type=merge -p '{"spec":{"ttlSecondsAfterFinished":600}}'Makes a finished Job eligible for TTL cleanup
In short : The owner is written on the dependent, the garbage collector deletes a dependent once all its owners are gone, and the cascade mode (background, foreground, orphan) sets the order.

1.Finalizers : deletion gates

A finalizer is a key in metadata.finalizers. While any key is left, the object stays.

  • DELETE sets deletionTimestamp and returns 200 OK (202 only for the deprecated orphanDependents=false). Terminating : keys can only be removed, no undelete.
  • Custom keys need a domain, example.com/cleanup (CRs only warn). Bare : kubernetes, orphan, foregroundDeletion.
  • --force --grace-period=0 skips graceful termination, not finalizers.
kubectl kube-apiserver etcd 1. DELETE 3. 200 OK 7. removed 2. set deletionTimestamp Object, Terminating finalizers: [example.com/cleanup] Owning controller External resource 4. watch event 6. remove only its own key, then Update 5. cleanup

2.In your controller

  1. Add your key before making anything external.
  2. deletionTimestamp set ? Clean up (idempotent).
  3. Remove only your own key, then Update.
  4. Failed ? Return the error and retry. The key stays.
  5. RBAC : plain update on the kind. <kind>/finalizers only gates setting blockOwnerDeletion on children.

More : Controllers and Reconciliation, Operators.

Why it hangs
  • Namespace : an aggregated APIService is down (NamespaceDeletionDiscoveryFailure), or CRs outlived their operator (NamespaceFinalizersRemaining).
  • PVC : a Pod still mounts it (pvc-protection).
  • CR : its operator was uninstalled before it.

3.Built-in finalizers you will meet

KeyOn (added by)Gone when
kubernetes.io/pvc-protection,
kubernetes.io/pv-protection
PVC, PV (StorageObjectInUseProtection)No Pod uses the PVC, the PV is unbound
kubernetes.io/pv-controller (in-tree),
external-provisioner.volume.kubernetes.io/finalizer (CSI)
PV with reclaim Delete (v1.33)Backend volume deleted
foregroundDeletion, orphanAny owner (API server)GC is done with the dependents
kubernetes (in spec.finalizers)Namespace, on createAll content gone, Pods first
batch.kubernetes.io/job-trackingJob Pods (v1.26)Job controller counted the Pod

4.Stuck in Terminating ?

A finalizer is a promise that someone will clean up. Find the someone before you delete the promise.

What holds it ?metadata.finalizers
Read the keyprefix names the controller
Check itrunning ? RBAC ? logs ?
Fix itit drops its key

5.Commands

CommandWhat it does
kubectl get pvc -n shop -o custom-columns=NAME:.metadata.name,FINALIZERS:.metadata.finalizersWhat is Terminating, and which key holds it
kubectl get ns shop -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'Why a namespace is stuck. On a DiscoveryFailure, fix the APIService that kubectl get apiservices shows as False
kubectl describe pvc data-postgres-0 -n shopUsed By lists the Pods that keep pvc-protection in place
kubectl patch widgets.example.com w1 -n shop --type=json -p='[{"op":"remove","path":"/metadata/finalizers"}]'Last resort, drops ALL keys. Only if the controller is gone for good and you did its cleanup
In short : A finalizer holds an object until its controller cleans up and drops its own key. When one is stuck, fix the controller before you touch the key.

API machinery

Admission Control

1.What is admission control ?

The API server's last gate before etcd. Admission plugins can change a write, then reject it.

  • Runs in kube-apiserver after authn and authz. Writes only : create, update, delete, connect.
  • Mutating plugins may change the object. Validating plugins only say yes or no.
  • One rejection, in either phase, fails the request.
  • Only new writes are checked. Existing objects are left alone.
Client kubectl Authentication fail = 401 Authorization fail = 403 Admission, writes only Mutating, then validating (see 2.) may change it, then yes or no saved only if nothing rejected etcd get / list / watch skip it

Any reject in either phase ends the request with an error.

2.Inside admission, in order (v1.37)

  • Built-in mutators first : ServiceAccount, DefaultStorageClass, LimitRanger ...
  • Then each bound MutatingAdmissionPolicy (CEL).
  • Then mutating webhooks, serial, any order.
  • Object changed ? Built-ins run again, and reinvocationPolicy: IfNeeded hooks may too.
  • Schema check, built-in validators (PodSecurity ...), ValidatingAdmissionPolicies, validating webhooks (parallel), ResourceQuota last. One deny is enough.
Mutating phase Built-in mutators e.g. ServiceAccount Mutating policies CEL, in-process Mutating webhooks serial, one at a time changed ? loop again Object validation the final object Validating phase Validating policies after built-in validators Validating webhooks all in parallel Response any deny = rejected Webhook order is not guaranteed, so every mutator must be idempotent.

3.Built-in plugins

Compiled into kube-apiserver, 25 on by default in v1.37. Change the set with --enable-admission-plugins / --disable-admission-plugins (order does not matter).

PluginTypeWhat it does
NamespaceLifecycleMRejects objects in missing or terminating namespaces. default, kube-system, kube-public cannot be deleted
LimitRangerM + VApplies LimitRange defaults, enforces the namespace min / max
ResourceQuotaVRejects a write that would push the namespace past its quota
ServiceAccountM + VDefaults the SA to default, rejects a missing SA, adds the API token volume
PodSecurityVPod Security Standards from namespace labels (stable v1.25), see Pod Security
DefaultStorageClassMNew PVC with no class gets the default one. Create only, newest default wins

M = mutating, V = validating.

4.Picking a mechanism

MechanismRuns in, logic inGood for
ValidatingAdmissionPolicykube-apiserver, CEL (GA v1.30)Deny / warn / audit rules, no server to run
MutatingAdmissionPolicykube-apiserver, CEL (GA v1.36)Simple defaults : labels, sidecars, PSA labels
Validating webhookyour HTTPS server, any languageComplex rules, external data. Called in parallel
Mutating webhookyour HTTPS server, any languageComplex mutation, external APIs. Called serially
Kyverno (v1.19)webhooks, YAML + CELPolicy library, generate / cleanup, image checks
OPA Gatekeeperwebhooks, RegoConstraintTemplate + Constraint, audits old objects
Rule of thumb : a CEL policy whenever it can express the rule, a webhook for complex logic or outside data. For CRDs, use their built-in validation rules and defaulting, not a webhook (see CRDs).
In short : Admission is the last gate before etcd : mutating plugins may change a write, validating plugins may reject it, and your own rules go in CEL policies or webhooks.

1.Webhooks : how a call works

  • Register in a Mutating or ValidatingWebhookConfiguration (v1).
  • The API server POSTs an AdmissionReview. The reply echoes request.uid and sets allowed.
  • Changes return as a base64 JSONPatch. warnings[] reach the client as Warning headers (code 299).
  • clientConfig : a Service (port 443, path / by default) or an https URL.
  • caBundle : CA for the serving cert, valid for <svc>.<ns>.svc. Filled by cert-manager's cainjector, see cert-manager.

2.Webhook settings

FieldValuesDefaultNotes
failurePolicyFail, IgnoreFailErrors and timeouts only. allowed: false always denies
timeoutSeconds1 to 3010On timeout, failurePolicy decides
sideEffectsNone, NoneOnDryRunrequiredLegacy v1beta1 Some / Unknown fail dry-run
matchPolicyExact, EquivalentEquivalentAlso catches other group / versions of the resource
namespaceSelectorlabel selectorallLeave out kube-system and the webhook's own namespace
objectSelectorlabel selectorallOpt-in hooks only : whoever writes the labels can skip it
matchConditionsup to 64 CELnoneCalled only if all are true (GA v1.30)
reinvocationPolicyNever, IfNeededNeverMutating only. IfNeeded needs an idempotent hook

Matching : rules (operations, apiGroups, apiVersions, resources, scope), then the two selectors and matchConditions. Since v1.37 (beta, on by default), TokenReview, SubjectAccessReview and other virtual auth resources skip webhooks.

3.ValidatingAdmissionPolicy : CEL, no server

# demo : at most 5 replicas per Deployment
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: max-replicas.example.com
spec:
  matchConstraints:
    resourceRules:
    - apiGroups:   ["apps"]
      apiVersions: ["v1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["deployments"]
  validations:
  - expression: "object.spec.replicas <= 5"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: max-replicas-demo.example.com
spec:
  policyName: max-replicas.example.com
  validationActions: [Deny]  # [Warn, Audit] to trial
  matchResources:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: demo
  • GA since v1.30. Does nothing until a binding names it in policyName. failurePolicy defaults to Fail.
  • validationActions : Deny, Warn, Audit. Never Deny and Warn together.
  • paramKind + paramRef pass a ConfigMap or any CR to CEL as params. parameterNotFoundAction (Allow / Deny) is required.
  • CEL sees object, oldObject (null on CREATE), request, namespaceObject, variables, authorizer. Type errors : status.typeChecking.
MutatingAdmissionPolicy (GA v1.36, on by default) : CEL emits an ApplyConfiguration (merged like server-side apply) or JSONPatch ops. reinvocationPolicy is required. Binds the same way.

Manifest-based admission (beta v1.37) : staticManifestsDir loads webhooks and policies from disk, enforced from API server start, even with etcd down. Names end in .static.k8s.io.

4.When a webhook goes down

Node fails replicas ran there Webhook Pods gone ReplicaSet controller new Pod kube-apiserver Webhook Svc no endpoints error or 10s timeout Pod CREATE rejected failurePolicy: Fail never comes back Configs never go to webhooks : the delete works.
Avoid the deadlock
  • namespaceSelector : skip kube-system, own ns.
  • 2+ replicas behind the Service.
  • Mutating hooks fail open, a validator checks.
  • Small timeoutSeconds : mutating hooks are serial.
  • Break glass : kubectl delete the Validating / MutatingWebhookConfiguration
In short : A webhook puts your HTTPS server on the write path, so scope it and keep it up. A CEL policy runs inside the API server and acts only once a binding points at it.

Security

RBAC and Service Accounts

1.Two questions per request

Every API call is checked twice. Authentication asks who you are. Authorization asks if that identity may do this verb on this resource.

  • Authentication returns only strings : username, UID, groups, extra.
  • Bad credential : 401. Nobody allows : 403, the default deny.
  • Authorizers run in order, e.g. Node, RBAC, Webhook. The first allow or deny wins.
  • RBAC allows or has no opinion. It never denies.
  • Admission runs only after an allow.
Client cert, JWT, OIDC, proxy Authentication invalid 401 Unauthorized user, groups, extra UserInfo no credential :system:anonymous verb, resource, ns Authorizer chainNode, RBAC, Webhook no allow 403 Forbidden first allow wins Admission saved etcd

Members of system:masters skip RBAC and webhook authorizers.

2.Users come from outside

There is no User object. You cannot add a normal user with an API call.

  • X.509 certCN = username, each O = a group
  • OIDC / JWTtoken from an identity provider
  • Webhookan outside service checks the token
  • Auth proxya proxy in front sets the user
  • Nothingsystem:anonymous, group system:unauthenticated

Every authenticated request also gets the group system:authenticated.

3.Inside a rule

  • apiGroups"" = the core group
  • resourcespods, secrets, pods/log (get), pods/exec (create)
  • verbsget, list, watch, create, update, patch, delete, deletecollection
  • specialescalate, bind, approve, sign, attest, impersonate (beta : impersonate-on:MODE:VERB)
  • resourceNamesonly these objects
  • nonResourceURLs/healthz, ClusterRoleBinding only

4.Four RBAC kinds

KindScopeHoldsEffect
RoleNamespacedrules[]Permissions usable only inside its namespace
ClusterRoleClusterrules[] or aggregationRuleReusable. The only way to cover nodes, nonResourceURLs or all namespaces
RoleBindingNamespacedsubjects[] + roleRef (Role or ClusterRole)Grants the role only in the binding's namespace
ClusterRoleBindingClustersubjects[] + roleRef (ClusterRole only)Grants the role in every namespace and cluster-wide

All in rbac.authorization.k8s.io/v1. No deny rules, permissions only add up. Turn it on with --authorization-mode=...,RBAC or an --authorization-config file (stable since v1.32).

5.Subjects, bindings, roles

  • A RoleBinding can reuse one ClusterRole in many namespaces, granting only in its own.
  • roleRef is immutable. Delete and recreate, or kubectl auth reconcile.
  • A ServiceAccount subject uses apiGroup "". A ClusterRoleBinding must set its namespace.
  • Users are just strings. Recreate a deleted name and it inherits its old bindings.
User Group ServiceAccount subjects[] RoleBindingone namespace ClusterRoleBindingwhole cluster roleRef reuse Role ClusterRole labelledClusterRoles aggregation
In short : Authentication turns a credential into strings. Roles list what is allowed, bindings hand a role to subjects in one namespace or the whole cluster, permissions only add up, and anything no rule allows gets 403.

1.ServiceAccounts

The only identity stored as an API object. Namespaced, core v1.

  • Every namespace gets a default SA. It may only do discovery and read ClusterTrustBundles.
  • A Pod picks one with serviceAccountName (else default). A missing SA gets the Pod rejected.
  • automountServiceAccountToken: false on the SA or the Pod skips the token. The Pod wins. Use it when the app never calls the API.
Singular vs plural
  • usersystem:serviceaccount:shop:log-reader
  • groupssystem:serviceaccounts and system:serviceaccounts:shop
kind: Pod
spec:
  serviceAccountName: log-reader
  automountServiceAccountToken: false  # no token

Identity outside the cluster, with mTLS : see SPIFFE, SPIRE and Pod Certificates.

2.Bound tokens in a Pod

  • Mount : /var/run/secrets/kubernetes.io/serviceaccount holds token, ca.crt and namespace.
  • Lifetime : asks for 3607s, and the API server may extend it to a year. Renewed past 80% of TTL or 24h.
  • Revoke : not per token. Delete the Pod or the SA; it still works up to 10 s (cache).
  • Legacy Secret tokens : not auto-made since v1.24. A hand-made one never expires.
Pod specserviceAccountName SA admissionadds the volume kube-api-accesstoken, ca.crt, namespace reads Appre-reads, file rotates TokenRequestbound to this Pod asks kubeletrenews at 80% of TTL writes token kube-apiserversig, exp, aud, Pod UID Bearer RBACas system:serviceaccount:ns:sa Pod deleted :token rejected

3.Default roles

  • cluster-admin : everything. Bound to system:masters and kubeadm:cluster-admins.
  • admin : most namespaced objects, Roles too. Cannot write ResourceQuota or the Namespace.
  • edit : read / write most objects and Secrets, run Pods as any SA. Cannot see Roles.
  • view : read-only. No Secrets, no Roles.

admin, edit and view have no default binding. They aggregate any ClusterRole labelled rbac.authorization.k8s.io/aggregate-to-edit: "true" (or -admin, -view), so CRDs can extend them. view flows into edit, edit into admin.

4.Risky permissions

PermissionWhy
get / list / watch secretsAll three return the data, legacy SA tokens too
create podsMount any Secret, run as any SA in the namespace. See Pod Security
escalate, bindWrite or bind roles you do not hold. Without them the RBAC API blocks that
impersonateAct as anyone, even group system:masters
get nodes/proxyExec in every Pod on the node, no audit
create serviceaccounts/tokenMint tokens for existing SAs

5.Commands

CommandDoes
kubectl auth whoami -o jsonUsername, groups and extra, as the cluster sees you
kubectl auth can-i get pods --subresource=log -n shopAsk the authorizer. The exit code is the answer
kubectl auth can-i list pods -n prod --as=system:serviceaccount:dev:fooAsk as someone else. Needs impersonation rights
kubectl auth can-i --list -n shopEverything you may do in a namespace
kubectl create rolebinding log-reader -n shop --role=pod-log-reader --serviceaccount=shop:log-readerGrant a Role (or --clusterrole) in one namespace
kubectl create token log-reader -n shop --duration 10mShort-lived token through TokenRequest
In short : A Pod runs as a ServiceAccount with a short-lived token bound to that Pod. Grant the least it needs, and treat the risky permissions like admin rights.

Security

Pod Security

1.Standards vs admission

Pod Security Standards (PSS) define three levels. Pod Security Admission (PSA) enforces them per namespace.

  • PSA is the built-in PodSecurity validating admission plugin. GA since v1.25, on by default.
  • It runs after mutating admission, so it judges the final Pod. See Admission Control.
  • Pods comply through securityContext and, since v1.35, hostUsers: false.
  • PodSecurityPolicy : removed in v1.25. Use PSA, a third-party admission plugin, or both.
Privileged no restrictions Baseline a default Pod passes bans : host namespaces, privileged, hostPath, hostPort, extra caps, Unconfined seccomp or AppArmor, Unmasked procMount, unsafe sysctls, probe host field, HostProcess, SELinux Restricted best practice adds : allowPrivilegeEscalation false, runAsNonRoot, runAsUser not 0, seccomp RuntimeDefault or Localhost, drop ALL, volume allow-list

A Pod that passes restricted also passes baseline.

2.One label per mode

Label the namespace pod-security.kubernetes.io/<MODE>: <LEVEL>. The modes are independent.

ModeOn a violationChecks
enforceRejects the Pod (403)Pods only
auditAudit annotation, allowedWorkloads + Pods
warnClient warning, allowedWorkloads + Pods

Common rollout : enforce baseline, warn and audit restricted.

Versions, defaults, exemptions
  • -versionv1.37 or latest (default). Pin enforce-version so an upgrade cannot start rejecting Pods.
  • No labelsCluster defaults from PodSecurityConfiguration. Built in : privileged / latest.
  • Exemptusernames, runtimeClasses, namespaces skip all three modes. Avoid controller service accounts.

3.Deployment path vs Pod path

Namespace labels level + version per mode reads kubectl apply a Deployment kube-apiserver Mutating admission final object PodSecurity validating store etcd if allowed warn audit Warning to the client Audit log annotation warn, audit : Deployment and Pod ReplicaSet controller creates the Pod enforce fails : 403 Forbidden, no Pod ReplicaSet controller kubectl apply a Deployment a Pod kube-apiserver Mutating admission final object PodSecurity validating Namespace labels Warning to client Audit log annotation etcd if allowed warn, audit : Deployment and Pod red : enforce fails, 403, no Pod

enforce checks Pods only : a bad Deployment is stored, its Pods are not.

4.The checks, field by field

Baseline bansAllowed
hostNetwork / PID / IPCunset or false
privilegedunset or false
hostPath volumesnot used
ports[*].hostPortunset or 0
appArmorProfile.typeunset, RuntimeDefault, Localhost
seccompProfile.typeanything but Unconfined
Restricted addsAllowed
allowPrivilegeEscalationfalse
capabilitiesdrop ALL, add only NET_BIND_SERVICE
runAsNonRoottrue (pod, or every container)
runAsUsernon-zero or unset
seccompProfile.typeset : RuntimeDefault or Localhost

Restricted volumes : configMap, csi, downwardAPI, emptyDir, ephemeral, persistentVolumeClaim, projected, secret. A rejection names the check, for example allowPrivilegeEscalation != false or runAsNonRoot != true.

In short : PSS defines three levels and PSA enforces them per namespace through labels. enforce rejects Pods, warn and audit only report.

1.A Pod that passes restricted

apiVersion: v1
kind: Pod
metadata: {name: app, namespace: team-a}
spec:
  securityContext:        # pod level
    runAsNonRoot: true
    runAsUser: 10001
    runAsGroup: 10001
    fsGroup: 10001
    seccompProfile: {type: RuntimeDefault}
  containers:
  - name: app
    image: busybox:1.28
    command: ["sh", "-c", "sleep 1h"]
    securityContext:      # container level
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities: {drop: ["ALL"]}
    volumeMounts: [{name: tmp, mountPath: /tmp}]
  volumes: [{name: tmp, emptyDir: {}}]  # scratch

2.securityContext, two levels

spec.securityContext sets defaults for every container. containers[*].securityContext wins when both set a field.

FieldLevelIf unset
runAsUserbothimage USER
runAsNonRootbothno check
seccompProfilebothUnconfined *
appArmorProfilebothRuntimeDefault **
fsGrouppodowners untouched
allowPrivilegeEscalationcontainertrue
readOnlyRootFilesystemcontainerfalse
capabilitiescontainerruntime default
privilegedcontainerfalse

* RuntimeDefault if kubelet seccompDefault: true (GA v1.27, off by default). ** on AppArmor nodes. fsGroup, supplementalGroups, sysctls : pod only.

3.User namespaces : hostUsers: false

  • GA in v1.36. Root in the Pod maps to an unprivileged host UID range, 65536 IDs per Pod by default.
  • PSA (policy v1.35+) skips runAsNonRoot and runAsUser : root inside passes restricted.
  • Not with hostNetwork, hostPID, hostIPC or volumeDevices.
  • Nodes : Linux 6.3+, containerd 2.0+ or CRI-O 1.25+, runc 1.2+ or crun 1.9+.
Pod user namespace process, UID 0 root inside the Pod capabilities work inside it only Volume idmap mount in-Pod owners, disk unchanged uid_map : 0 to 833617920 count 65536 Node : host UID 833617920 unprivileged, a breakout lands here

4.Commands

CommandDoes
kubectl label --overwrite ns team-a pod-security.kubernetes.io/enforce=restricted pod-security.kubernetes.io/enforce-version=v1.37Enforce restricted, pinned to v1.37
kubectl label --dry-run=server --overwrite ns <ns> pod-security.kubernetes.io/enforce=baselineRunning Pods that would fail
kubectl apply --dry-run=server -n <ns> -f <file>Runs admission, stores nothing
kubectl debug -it <pod> --image=<nonroot-image> --target=<container> --profile=restrictedRestricted-safe debug container
kubectl exec <pod> -- cat /proc/self/uid_mapExpect 0, a high host UID, 65536

5.Gotchas & Good to know

Gotchas
  • enforce never blocks a Deployment. Read the warnings and the ReplicaSet events.
  • New enforce labels do not evict running Pods. Dry-run the label first.
  • runAsNonRoot with an image USER of root, or a name : CreateContainerConfigError. Set a numeric runAsUser.
Good to know
  • An explicit appArmorProfile RuntimeDefault fails on nodes without AppArmor.
  • PSA checks debug containers too : use --profile=restricted.
  • hostPID sees node processes, hostNetwork Pods may skip NetworkPolicy, privileged or hostPath reach the node's Secrets.
In short : Share settings through the pod securityContext, use a numeric non-root runAsUser, and hostUsers: false maps root in the Pod to an unprivileged UID on the node.

Identity and trust

OIDC Authentication for the API Server

1.No users, only strings

  • Kubernetes has no user objects. Every authenticator returns four attributes : username, UID, groups, extra. They mean nothing until RBAC matches them.
  • Success adds group system:authenticated. Anonymous : system:anonymous, group system:unauthenticated.
  • First success wins; the API server does not guarantee the order. Docs : ServiceAccount tokens plus at least one method for people.
  • See RBAC, API Machinery. Workloads : SPIFFE.

2.Ways in, and when each one ends

MethodSet withWho uses itEnds when
X.509 client cert--client-ca-filebreak-glassat expiry, or rotate the CA
ServiceAccount token--service-account-*Podsbound ones die with their Pod or Secret
Bootstrap token--enable-bootstrap-token-authnode joindelete the token Secret
Static token file--token-auth-filenot recommendededit the file, restart
OIDC / JWT--oidc-* or --authentication-configpeopleat exp. No revocation
Webhook--authentication-token-webhook-config-filea serviceafter the cache TTL (2m)
Authenticating proxy--requestheader-client-ca-filea front proxyat the proxy

X.509 : CN = user, each O = a group. v1.37 has no certificate revocation. Issuing them : cert-manager, Private PKI.

3.What an ID token is

  • issequals the issuer URL exactly, trailing slash too
  • subthe subject : stable but opaque
  • audone entry must match an audience
  • expexpiry; iat, nbf checked too
ID token, not access token

kubelogin sends the id_token; its aud is the client id. A Keycloak access token usually shows aud: account, so it fails the audience check. See Keycloak.

Also mapped : email, preferred_username, groups. IdP-signed; the API server caches the jwks_uri keys from {issuer}/.well-known/openid-configuration, checks offline. Issuer : https, not --service-account-issuer (fatal, 1.30+).

4.One kubectl request

  1. kubectl runs the exec plugin named in kubeconfig.
  2. kubelogin returns a cached ID token, refreshes it, or opens a browser login.
  3. kubectl sends Authorization: Bearer <id_token>.
  4. The authenticator matches iss, checks the signature with cached JWKS, then aud and exp.
  5. Claims become username, groups, uid and extra, with prefixes.
  6. RBAC allows it, or the request is authenticated but Forbidden.

Then Admission. The login itself : Keycloak.

kubectl kubelogin token cache exec id_token only with no refreshable token browser login at IdP code + PKCE Bearer id_token API server match iss, verify with JWKS check aud, exp claims to user, groups no call to the IdP per request cached keys RBAC allowed 403 Forbidden

5.Why OIDC for people

  • One identity per person, groups managed in the IdP, short-lived tokens.
  • Not a shared admin cert : k3s's CN=system:admin, O=system:masters skips RBAC and cannot be revoked short of rotating the CA. Keep it offline.
Seen in practice

No OIDC at the API server: people use a client-cert context, and SSO users reach RBAC only through a web UI that impersonates their email and groups. Pitfall : empty claims make the UI act as its own ServiceAccount, often cluster-admin. Do better : validate claims with CEL before impersonating, and keep client certs for break-glass.

In short : The API server never logs anyone in. It checks a signed ID token against the issuer's keys, turns its claims into a username and groups, and RBAC decides what they may do.

1.Flags or a file, never both

  • Legacy --oidc-* flags : supported, not deprecated. One issuer, one audience.
  • --authentication-config : AuthenticationConfiguration in apiserver.config.k8s.io/v1, GA since 1.34, gate removed in 1.37.
  • Any --oidc-* flag plus any config file, even one with only anonymous, stops the API server : mutually exclusive.
  • The removed StructuredAuthenticationConfiguration gate on 1.37 : unrecognized feature gate.
TopicFlagsFile
Issuers, audiencesone each, audience = --oidc-client-idup to 64 issuers; many audiences with MatchAny
Username prefix<issuer># unless the claim is email; - disablesno default : prefix required with claim
Claim rules, maps--oidc-required-claim; claim names onlyclaim or CEL, plus uid, extra
Changerestart the API serverhot reload
Signing algs--oidc-signing-algs, default RS256all known algs (RS, ES, PS)
IdP CA--oidc-ca-file (a path)certificateAuthority (PEM), see Private PKI

2.A minimal v1 file

apiVersion: apiserver.config.k8s.io/v1
kind: AuthenticationConfiguration
anonymous:
  enabled: true
  conditions: [{path: /livez},
    {path: /readyz}, {path: /healthz}]
jwt:
- issuer:
    url: https://auth.example.com/realms/demo
    audiences: [kubernetes]
  claimValidationRules:
  - expression: 'claims.azp == "kubernetes"'
  claimMappings:
    username: {claim: preferred_username,
               prefix: "oidc:"}
    groups: {claim: groups, prefix: "oidc:"}
    uid: {claim: sub}
  userValidationRules:
  - expression: "!user.username.startsWith('system:')"
    message: no system names

apiserver.k8s.io is the wrong group : no kind "AuthenticationConfiguration" is registered. username.claim: email rejects a token with email_verified: false.

3.Claims to user

JWT claims claimValidationRules claimMappings username, groups, uid, extra + prefix userValidationRules UserInfo RBAC false 401 false 401
  • CEL : claims in claim rules and mappings, user in user rules. An empty username fails.
  • Prefixes stop IdP system: names and clashes. File : no defaulting, prefix: "" = none.
  • Flags, --oidc-username-prefix unset, claim not email : <issuer>#<claim value> (sub default). Groups prefix : no default (not oidc: as the CLI docs say).
  • Email (1.33) : claims.email in the username needs claims.email_verified in it, an extra or a claim rule : claims.?email_verified.orValue(true) == true.

4.Reload

  • Bad parse or validation : the old config stays.
  • A valid file applies once every new issuer answers discovery within 1 minute, else failed to update authentication config.
  • anonymous cannot change. Write atomically.
  • 10s success cache : a token works up to 10s after a reload.

Check apiserver_authentication_config_controller_automatic_reloads_total{status="failure"} (kubectl get --raw /metrics).

5.On k3s

  1. File on every server node, mode 0600.
  2. In /etc/rancher/k3s/config.yaml, a list item under kube-apiserver-arg : - authentication-config=<path>.
  3. Drop oidc-* entries and the old StructuredAuthenticationConfiguration gate.
  4. systemctl restart k3s once; later edits reload.
  5. Keep an anonymous block : with this flag k3s drops anonymous-auth=false, so anonymous is enabled.

Never the k3s ServiceAccount issuer https://kubernetes.default.svc.<cluster-domain>.

In short : Flags or the file, never both. The file maps claims with CEL, needs explicit prefixes and reloads once the issuer answers. On k3s it goes in kube-apiserver-arg with its own anonymous block.

1.Exec credential plugins

  • kubeconfig users[].user.exec : command, apiVersion (client.authentication.k8s.io/v1), args, installHint, provideClusterInfo.
  • interactiveMode (Never, IfAvailable, Always) is required in v1.
  • The plugin prints an ExecCredential with status.token and expirationTimestamp. kubectl caches it until expiry or a 401.

kuberc credentialPluginPolicy: Allowlist (entries keyed command; name deprecated in 1.36) refuses other plugins.

users:
- name: oidc
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1
      command: kubectl
      interactiveMode: IfAvailable
      args: [oidc-login, get-token,
        --oidc-issuer-url=<issuer>,
        --oidc-client-id=kubernetes,
        --oidc-extra-scope=groups]

2.kubelogin

  • kubectl oidc-login : int128/kubelogin v1.36.4 (not the AKS tool). Versions do not track Kubernetes.
  • Auth code + PKCE on 127.0.0.1:8000 (or 18000). Cache : ~/.kube/cache/oidc-login (disk; keyring optional).
  • IdP client : public with PKCE, no secret on laptops. Register http://localhost:8000 and http://localhost:18000 exactly.
  • Ask for the groups scope; its mapper : Keycloak.

3.RBAC for OIDC groups

IdP group k8s-viewers groups: "k8s-viewers" + prefix "oidc:" oidc:k8s-viewers ClusterRoleBinding subject kind: Group ClusterRole view Full group path ON "/k8s-viewers" never matches
  • Bind kind: Group with the prefixed name. RBAC rules : RBAC and Service Accounts.
  • Check with kubectl auth whoami : the final username and groups, after prefixes.
$ kubectl auth whoami
ATTRIBUTE   VALUE
Username    oidc:alice
Groups      [oidc:k8s-viewers system:authenticated]

Groups can vanish after a refresh (kubelogin may not resend extra scopes) : run whoami after one too.

4.Operations

SituationWhat happensWhat to do
Logoutworks until exp, plus the 10s cacheshort tokens, end the IdP session, oidc-login clean
Removed from a grouptakes effect at the next refreshkeep tokens short
IdP downissued tokens work; logins, refreshes faila break-glass cert, offline
API server up before the IdPtokens fail, oidc: authenticator not initialized, 10s retrieswait : certs and SA tokens still work
  • The API server process must reach the issuer (host network on k3s). An in-cluster issuer is a bootstrap loop; discoveryURL (https only, keys still from jwks_uri) does not fix it cheaply.

5.Commands

CommandDoes
kubectl auth whoamiwho the API server thinks you are
kubectl auth can-i list pods --as=oidc:alice --as-group=oidc:k8s-viewerstest RBAC for an OIDC identity (impersonation)
kubectl oidc-login setup --oidc-issuer-url=<issuer> --oidc-client-id=<client>log in, print the ID token claims
kubectl oidc-login cleandelete the local token cache (disk, keyring)
curl -s <issuer>/.well-known/openid-configurationcheck issuer and jwks_uri
In short : kubelogin, an exec plugin, caches the ID token. Bind the prefixed group and check with kubectl auth whoami. Tokens cannot be revoked : keep them short, keep a break-glass cert.

Identity and trust

Keycloak and Single Sign-On

1.What single sign-on is

Keycloak is an OpenID Connect and SAML identity provider (IdP). Apps send the browser to it, it checks the user (password, OTP, passkey or a brokered IdP) and returns signed tokens. One login leaves a session cookie at Keycloak that every app in the realm reuses : that is single sign-on. Keycloak 26.8.0, CNCF incubating.

Neighbours : the API server trusting these tokens is OIDC for the API server; what a groups claim grants is RBAC.

Four strings break SSOThe issuer URL, the redirect URI, the audience and the groups claim. Most login bugs are one of them not matching.

2.The model

ObjectWhat it isWatch out
RealmA tenant : own users, keys, issuer .../realms/<realm>master is for admins. No /auth since 17
ClientOne app that asks for tokensIts clientId shows up as azp in tokens
Public / confidentialNo secret (browser, CLI) / a secret or key for /tokenPublic : PKCE, tight redirect URIs
Client scopeBundle of mappers. Default : always. Optional : on requestNew realm defaults skip old clients
Protocol mapperPuts a claim in the ID token, access token or userinfoGroup Membership, Audience, ...
User, group, roleIdentities, memberships, realm and client rolesGroups nest, roles can be composite

3.Authorization code with PKCE

browser app (client) Keycloak (realm) 1 redirect to /auth + code_challenge 2 login (skipped with an SSO cookie) 3 back with code (60 s) via the browser 4 code + verifier back channel, /token 5 ID, access, refresh 6 check signature JWKS keys from /certs

Paths under https://<host>/realms/<realm>

EndpointPath
Discovery/.well-known/openid-configuration
Authorization/protocol/openid-connect/auth
Token.../token
Userinfo.../userinfo
Logout.../logout
JWKS keys.../certs

Grants : implicit SHOULD NOT be used, password grant MUST NOT be used (Keycloak, citing RFC 9700). Client credentials only for a client acting for itself. PKCE is optional per client until you set its method to S256.

4.Tokens and lifetimes

The basic claims are on OIDC for the API server. Keycloak adds :

  • azp : the client the token was issued to.
  • aud on a plain user token is account (Audience Resolve mapper plus the default account roles). A service that checks aud needs an Audience mapper.
  • groups comes from a Group Membership mapper. Full group path is ON in the admin console (/team). A mapper made by API or import without full.path acts OFF (team).
New realmDefault
Access token5 min (master 60 s)
SSO Session Idle30 min
SSO Session Max10 h
Offline Session Idle30 days
Authorization code60 s
  • microprofile-jwt writes a groups claim from realm roles, not groups.

The ID token's exp equals the access token's. Since 26.8.0 every refresh token's exp is capped by the session idle timeout, and a per-client lifespan cannot outlive SSO Session Max.

In short : A realm holds users, clients and keys behind one issuer URL. Apps use the code flow with PKCE and break when the issuer, redirect URI, audience or groups claim does not match.

1.Install paths

  • Official operator (lockstep) : kustomize ?ref=26.8.0 into a keycloak namespace you create first, raw manifests, experimental Helm chart, OLM. It runs the server, not the database : bring PostgreSQL 14 to 18 (CNPG).
  • codecentric keycloakx lags one release; default relativePath: /auth puts /auth in the issuer. Plain manifests : you own clustering and probes.

2.The Keycloak CR and what it makes

apiVersion: k8s.keycloak.org/v2beta1  # since 26.6.0
kind: Keycloak
spec:
  instances: 2
  image: registry.example.com/keycloak-custom:26.8.0
  db: {vendor: postgres, host: keycloak-pg}  # + Secrets
  http: {httpEnabled: true}    # TLS at the Gateway
  hostname: {hostname: auth.example.com}
  proxy: {headers: xforwarded}
  ingress: {enabled: false}    # HTTPRoute instead
  features: {enabled: ["client-admin-api:v2"]}
Gateway TLS ends here HTTPRoute Service:8080 / :9000 Keycloak CR operator makes to pods StatefulSetkeycloak-0, keycloak-1 keycloak-discoveryheadless, :7800 jdbc-ping : pods findeach other via the DB PostgreSQLstate + discovery

The CR also makes a NetworkPolicy, a temporary <name>-initial-admin Secret (replace it), a ServiceMonitor if metrics are on. Default memory: request 1700MiB, limit 2GiB. Never expose port 9000 (/health, /metrics).

3.Build time vs runtime options

OptionKindNotes
dbbuildDefault dev-file, not for production
health-enabled, metrics-enabledbuildDefault false
featuresbuild--features replaces the stored list
hostname, proxy-headers, http-enabledruntimeFine at each start
Exit code 2A custom spec.image (or startOptimized: true) starts --optimized : a build option passed at runtime with a different value exits with code 2. The stock image rebuilds at start. features.enabled must repeat the baked list.

4.Realms and clients as code

  • KeycloakRealmImport creates a realm once.
  • KeycloakOIDCClient (v2alpha1, Preview) needs client-admin-api:v2 and a Secret <cr-name>-admin. Name = clientId. No mappers, scopes, PKCE, lifetimes.
  • Since 26.8.0 its Secret needs label operator.keycloak.org/kind: KeycloakOIDCClient, else HasErrors.

5.Upgrades

update.strategyOn an image change
RecreateOnImageChangeDefault. Scales to 0 first : short outage
AutoRolls a newer patch, same major.minor
ExplicitRolls until you change spec.update.revision

Schema migration is one way : back up first. Roll back by restoring a dump into a fresh database.

Seen in practiceBuild options repeated at runtime crashloop an optimized image. Do better : bake them once, run 2+ instances, give the DB HA (one Postgres on local disk is a single point of failure), dump it nightly.
CommandWhat it does
kubectl apply -k 'github.com/keycloak/keycloak-k8s-resources/kubernetes?ref=26.8.0'Operator, into ns keycloak
kubectl -n keycloak get keycloaks/<name> -o jsonpath='{.status.conditions}'Ready, HasErrors
kubectl -n keycloak get keycloakoidcclient <name> -o yamlClient errors (only here)
kc.sh show-configBaked values show (Persisted)
In short : The operator runs Keycloak from one CR, not its DB. Bake build options into the image, never at runtime. Upgrade after a backup; roll back from a dump. Client CRs cover part of a client.

1.Clients that work

  • Redirect URIs match exactly, case-sensitive. * only as the last character : then a prefix match.
  • https://*.example.com/cb is a literal : it never matches.
  • Since 26.6.3 https://example.com* means https://example.com/*.
  • A portless plain-http loopback URI (localhost, 127.0.0.1) matches any port, for CLIs (OIDC auth).
  • Since 26.7.3 a redirect URI carrying OIDC response parameters is refused.
aud: account is normalAdd an Audience mapper (often in its own client scope) when a proxy or gateway checks aud. Since 26.6.2 introspection checks it too.

Post-logout redirect URIs are a separate list. A broken one does not break login, so it is easy to miss.

2.Ways to put SSO or a token check in front of an app

PatternCode flow runs inGood forWeak spot
Native OIDC in the appThe appRoles map to app rolesConfigure each app
oauth2-proxyThe proxy, serves /oauth2/*Mature, group checksExtra hop per app; no gRPC ext_authz mode
Envoy ext_authz serverThe auth server, inside ext_authz repliesOne deployment for many hostsIn-memory sessions mean one replica
Gateway-native OAuth2Envoy oauth2 filter : kgateway OAuth2, Envoy Gateway SecurityPolicyNo extra serviceGateway-specific config
Gateway JWT validationNone : client brings a bearer tokenAPIs and machine clientsNo browser login

3.The ext_authz path

GatewayExtensionextAuth: grpcService TrafficPolicytargetRefs: HTTPRoute browser Gateway(Envoy) app allow auth server Keycloak login Check deny + 302or allow login at Keycloak, then the callback returns through the Gateway : Set-Cookie
  • A kgateway GatewayExtension holds extAuth, oauth2 or jwt; a TrafficPolicy attaches it to an HTTPRoute by targetRefs. type is deprecated, a no-op.
  • Across namespaces it needs a ReferenceGrant, or it silently fails to attach.
# oauth2-proxy flags
--provider=keycloak-oidc
--oidc-issuer-url=https://auth.example.com/realms/demo
--code-challenge-method=S256
--reverse-proxy=true
--cookie-refresh=4m     # under the 5 min token
--allowed-group=/admins # full path

The kgateway jwt gate answers 401 for a missing or bad token and 403 when its CEL rule misses. CEL rbac entries are ORed : put AND inside one expression, and use in for list claims.

4.The issuer is the public URL

  • In-cluster clients that check iss must use the public URL, not keycloak-service.<ns>.svc.
  • The JWKS may come from the in-cluster Service. With a 5 minute JWKS cache, a longer Keycloak outage breaks token checks.
  • One Keycloak outage takes out every SSO app. Keep an escape hatch per app where one exists.
Seen in practiceAll four patterns can live in one cluster. Pitfalls : a hostname-wildcard redirect URI never matches, and a logout redirect left on one fails quietly. Pods may not hairpin to the LB IP. Do better : exact URIs per app, and reach the public issuer with hostAliases or split DNS.
In short : Register exact redirect URIs, add an Audience mapper where aud is checked, pick the pattern by where the code flow runs, and use the public issuer URL everywhere.

Identity and trust

OpenBao: Secrets as a Service

1.What is OpenBao ?

  • The open-source fork of HashiCorp Vault, which is BSL 1.1 from 1.15.0. Forked from Vault's release/1.14.x branch (November 2023). 1.14.8 was the last MPL release.
  • MPL-2.0, an OpenSSF Sandbox project. Latest v2.7.1.
  • API compatible, not storage compatible. X-Vault-* headers, vault:v1: ciphertext, vault.* metrics and vault.hashicorp.com/* annotations stay. CLI bao, env BAO_*.
OpenBao 2.7.1Vault 2.1.2
LicenceMPL-2.0BSL 1.1
CLIbaovault
Tokenss. b. r. by defaulthvs. hvb. hvr.
Cloud sealsplugins only, since 2.7built in
Namespacesyes, since 2.3Enterprise only

2.One request path

client log in auth method kubernetes, jwt, oidc, approle ... token : TTL + policies policy check on the path secrets engine barrier (AES-256-GCM) storage who are you ? short-lived deny by default kv, database ... seal guards the keyring
  • Everything is a path : secret/data/app, database/creds/app, pki/issue/web, auth/kubernetes/login, sys/health.
  • Policies are written against paths. No matching rule means no access.
  • Capabilities : create, read, update, patch, delete, list, scan (OpenBao only, 2.2, a recursive list), sudo.
  • deny always wins over every other capability.
path "secret/data/app/*" { capabilities = ["read"] }

3.Engines and auth methods

EngineWhat it doesSee
KV v2Versioned values : secret/data/..., versions secret/metadata/..., keeps 10ESO
databaseA real DB user per lease (v-openbaouse-...), dropped when it endsCloudNativePG
transitEncryption as a service, the key never leaves. aes256-gcm96, vault:v1:...SOPS key
PKIA CA that issues short-lived certificates on requestPrivate PKI
auth methodsapprole, cert, jwt, oidc, kubernetes, userpass, tokenOIDC, Keycloak
not built inLDAP, Kerberos, RADIUS : plugins since 2.7. AWS, GCP, Azure engines : plugins, never built inplugins

4.Tokens, leases, TTLs

  • serviceRenewable, revocable, accessor, children.
  • batchLight. Not renewable, no accessor.
  • periodicRenews forever within its period.
  • rootFrom init, a root, or generate-root.
  • Default and max lease TTL 768h (32 days). Mounts, roles, tokens only go lower.
  • bao lease renew / revoke [-prefix]. sys/renew, sys/revoke* gone in 2.5.4.

5.Sealed by default

  • A new server is sealed : it cannot read its own data.
  • Shamir : init makes 5 shares, threshold 3. Each restart needs 3 on every node.
  • Auto-unseal (KMS plugin, transit, static, kmip) : the node unseals itself. Init gives recovery keys : they authorize generate-root, never decrypt the root key.

GET /v1/sys/health : 200 active, 429 standby, 503 sealed, 501 uninitialized.

In short : OpenBao is the open fork of Vault. A caller logs in with an auth method, gets a short-lived token whose path policies decide what it may touch, and engines serve static or on-demand secrets behind a sealed barrier.

1.Seals in 2.7

  • 2.7.0 moved awskms, gcpckms, azurekeyvault, ocikms, alicloudkms, pkcs11 into KMS plugins.
  • A same-named plugin "kms" shadows the builtin : no seal migration.
  • Built in : transit, static, kmip, Shamir.
  • Shamir to KMS is a real migration : downtime, backup, both seals up, unseal -migrate.
plugin_directory = "/openbao/plugins"
plugin "kms" "awskms" { command = "openbao-plugin-kms-aws" }
seal "awskms" {
  region     = "us-east-2"
  kms_key_id = "<kms-key-id>"
}
Upgrading to 2.7 with a builtin cloud seal leaves the cluster sealed. Add the plugin stanza and binary on 2.6 first.

2.Storage

raft (Integrated Storage) : HA, 3 or 5 voters. postgresql : HA, the docs now suggest it to newcomers. pebbledb (2.7) : durable, not HA, chart standalone default since 0.30.0. file : removed in 2.7.0, a migrate source until 2.8.0.

3.Raft on a StatefulSet

active Service openbao-active=true KMS plugin seal unseal each Pod openbao-0 active + own PVC openbao-1 standby + own PVC openbao-2 standby + own PVC 8201 : standbys forward writes headless Service a DNS name per Pod, the retry_join targets
  • Each node needs a unique node_id. Set it per Pod, or leave it unset and raft makes a UUID in its data dir. Never hardcode one id in a shared ConfigMap.
  • Quorum is a majority of voters : 2 of 3.
  • Standbys forward writes and, since 2.5, serve reads themselves (eventually consistent).
  • service_registration "kubernetes" labels each Pod openbao-active. Without it : no labels, no active-only Service.
  • Pods, PVCs, stable names : StatefulSets.

4.The Helm chart (0.30.3)

SettingDefault in 0.30.3
ModeStandalone with pebbledb. HA with server.ha.enabled and server.ha.raft.enabled
retry_joinNot in the default raft HCL : add it, or join by hand
Update strategyOnDelete, so you choose the order
authDelegatorOn : OpenBao's ServiceAccount may call TokenReview
InjectorOn, with HashiCorp's vault-k8s 1.7.6 image
TLS, liveness, audit storageOff. Not a production default

Optional : a CSI provider, and a snapshot CronJob (chart 0.22.0), aimed at the active Service since 0.23.5.

5.Upgrades, backups, audit

  1. Back up : a raft snapshot. No downgrade guarantee.
  2. Replace standbys one at a time (-l openbao-active=false), the active Pod last.
  3. Never fail over from a newer to an older version.
  • Snapshots come only from the active node : a standby redirects. No snapshot inspect, list with gunzip -c f.snap | tar tv.
  • Audit devices go in the config audit stanza : the API refuses them since 2.3.2 unless an unsafe switch is set. If every audit device fails, requests fail.
CommandWhat it does
helm install openbao openbao/openbao --set server.ha.enabled=true --set server.ha.raft.enabled=true3-replica raft HA install
bao statusSealed ? Active or standby ?
bao operator initInit : Shamir 5 / 3
bao operator raft list-peersVoters, leader, state
bao operator raft snapshot save <file>Back up, on the active node
bao operator unseal [-migrate]Give one unseal share. -migrate in a seal migration
In short : Run three raft voters with unique node ids and auto-unseal, move cloud seals to KMS plugins before 2.7, upgrade standbys first, and take snapshots from the active node.

1.Kubernetes auth

Pod ServiceAccount token OpenBao auth/kubernetes/ login 1 role + JWT 4 token with role's policies kube-apiserver TokenReview API 2 TokenReview 3 valid : SA, ns 5 read secret/data/app/* with the token
bao auth enable kubernetes
bao write auth/kubernetes/config \
  kubernetes_host=https://kubernetes.default.svc
bao policy write app-read app-read.hcl
bao write auth/kubernetes/role/app \
  bound_service_account_names=app \
  bound_service_account_namespaces=app \
  policies=app-read ttl=1h
  • OpenBao calls the TokenReview API, so its own ServiceAccount needs system:auth-delegator. The chart sets it. A hand-written install must add it, or every login is "permission denied".
  • No reviewer token : the client's JWT is the bearer, and each client SA needs the binding instead.
  • OpenBao re-reads its own SA token every minute, so short-lived projected tokens work (RBAC).
  • A role must bind ServiceAccount names and namespaces.
  • Since 2.7.1 and 2.6.4 the SA JWT is checked again on renewal : a revoked SA token can no longer renew.
  • app-read.hcl : a read-only policy on secret/data/app/*, plus read, list on secret/metadata/app/*.

2.JWT auth, SPIFFE and people

  • JWT auth checks a signed token against one issuer per mount (oidc_discovery_url, jwks_url or fixed keys), like the API server (OIDC). No TokenReview call, no early revocation.
  • SPIFFE : a JWT-SVID from the SPIRE agent, posted to auth/jwt/login. Pin bound_subject=spiffe://<td>/ns/<ns>/sa/<sa>, or any SVID with the right audience gets the role. No SPIFFE-specific method (SPIFFE, SPIRE).
  • People : a second JWT/OIDC mount that trusts the IdP (Keycloak).

3.Ways to get a secret out

PathSecret lands asAuthNote
ESOa Secret, per refreshIntervalESO's SAGitOps-friendly. The value sits in etcd
injectorsidecar files, /vault/secrets/the Pod's SAvault.hashicorp.com/*, webhook failurePolicy: Ignore : webhook down, the Pod starts without secrets
CSI driverfiles in a volumethe Pod's SAprovider: openbao, needs the CSI driver
direct loginin the app's memoryKubernetes or JWTbest for leases and dynamic secrets
Flux SOPSkustomize-controller decryptsKubernetes (Flux 2.9)a transit key as SOPS master key
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata: {name: openbao}
spec:
  provider:
    vault:
      server: http://openbao.openbao:8200
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: external-secrets
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
  • ESO's docs point to the vault provider. A newer provider.openBao (camelCase) uses path, not mountPath.
Seen in practice
  • Lagging patch releases : renewal fixes land in patches. Do better : track the current minor's patches.
  • Cloud KMS auto-unseal ties unsealing to cloud credentials. If a client cert supplies them, its CA now guards unsealing too (Private PKI). Do better : alert on that chain and keep a tested recovery path.
  • Each auth mount fails its own way : Kubernetes auth needs the reviewer's system:auth-delegator binding, or every login fails.
  • Snapshots must hit the active node : a standby redirects to the leader's api_addr. Do better : the active Service, TLS end to end.
In short : Pods log in with their ServiceAccount or a SPIFFE JWT-SVID, then get secrets out via ESO, the injector, CSI, the API or Flux SOPS. Pin JWT roles with bound_subject.

Identity and trust

Private PKI and Trust Distribution

1.Why a private CA ?

  • For names and clients no public CA will vouch for : internal names, mTLS client certs, cloud trust anchors (AWS IAM Roles Anywhere).
  • Let's Encrypt dropped client certs : classic lost client auth on 2026-02-11, the tlsclient profile ended on 2026-07-08.
  • CA/B Forum ballot SC-081 shrinks public TLS certs : 200 days (2026-03-15), 100 days (2027-03-15), 47 days (2029-03-15). Neither rule binds a private CA.
  • An OIDC issuer on a private CA means the API server must trust that root too.
Which issuer ?
  • A real client hits a public hostname : a public CA (Let's Encrypt).
  • A workload proves who it is with a cert (mTLS, Roles Anywhere) : the private CA.
  • Anything else : no cert at all, TLS ends at the gateway.

2.Hierarchy

root CA offline if you can, 2 to 10 years signs intermediate, one per purpose 6 months to 2 years pathlen:0, name constraints signs leaf leaf leaf leaves : 30 to 90 days single tier : the root signs leaves itself, so the root key stays online

Lifetimes are OpenBao's PKI guidance. Keep the root key outside the online signer. For contrast, one real cluster runs a 10-year root, no intermediate and 1-year leaves renewed 30 days early.

3.The cert-manager bootstrap

ClusterIssuer selfSigned signs Certificate isCA: true writes Secret in cert-manager tls.crt, tls.key, ca.crt secretName ClusterIssuer ca signs leaf Certificates
  • A ClusterIssuer's CA Secret lives in the cluster resource namespace (default cert-manager).
  • Give the root a real subject : an empty issuer DN breaks RFC 5280 (BadConfig).
  • The self-signed issuer should sign only the root.

4.What the CA issuer does not do

cert-manager itself warns : CA issuers "are generally for cert-manager demos or for advanced users with experience and tooling for running a PKI".

It does notSo you
check name constraints or path length when it issuesrely on clients that enforce RFC 5280 at the handshake
make CRLs or OCSP responses (it only embeds their URLs)plan revocation elsewhere, or keep leaves short
rotate a CA Secret you made by handtrack its expiry yourself
re-issue leaves when the CA Secret changesrun cmctl renew
stop leaves outliving the CAset the leaf duration yourself

5.Offline root, online intermediate

  1. Make the root key and cert on an offline machine (CA: true, up to 10 years).
  2. Sign the intermediate there : CA: true, pathlen:0, permitted DNS names, marked critical.
  3. Load a Secret : tls.crt = intermediate then root, tls.key = the intermediate key only (kubectl -n cert-manager create secret tls).
  4. Point a ca ClusterIssuer at it. Leaves get ca.crt = the root.
  5. Publish only the root to trust stores. Track the intermediate's expiry yourself.
In short : A private CA covers internal names and client certs. cert-manager bootstraps one from a self-signed issuer, but its CA issuer checks no constraints, makes no CRLs and re-issues nothing when the CA changes.

1.Copy the root, never the key

  • Copy only the public root into a dedicated ConfigMap or Secret. Not tls.crt (it can hold intermediates), and never tls.key.
  • A dedicated source lets you choose when trust changes roll out. Then fan it out to Pods, nodes and people.
root CA Secret tls.crt, tls.key, ca.crt reflector whole copy, has tls.key copy, no key root cert only trust-manager to ConfigMaps Pods mount ClusterTrustBundle kubelet writes a file in the Pod plain HTTP SHA-256 check laptops, nodes

2.Ways to distribute trust

OptionMovesLands asKey stays out ?
reflectorwhole Secret or ConfigMapsame-named copy per namespaceno, copies tls.key
trust-manager Bundlechosen certs + public rootsConfigMap per chosen namespaceyes
ClusterTrustBundleCA certs onlykubelet-written file in the Podyes
kube-root-ca.crtthe cluster's own CA onlyConfigMap in every namespaceyes, not for your CA
node storethe root PEM/etc/ssl/certs on each hostyes

3.trust-manager

  • v0.25.0 ships one CRD, Bundle (trust.cert-manager.io/v1alpha1, cluster-scoped). Its target ConfigMap takes the Bundle's name.
  • Sources : configMap, secret, inLine, useDefaultCAs (pin it : Debian trixie since v0.23.0).
  • Unset namespaceSelector = all namespaces today, documented to change.
  • PKCS#12 target for Java (JKS is deprecated). Secret targets : off by default, grant read on all Secrets.
spec:  # a Bundle : current + previous root
  sources:
  - useDefaultCAs: true
  - configMap: {name: root-current, key: ca.crt}
  - configMap: {name: root-previous, key: ca.crt}
  target:
    configMap: {key: ca-bundle.pem}
    additionalFormats: {pkcs12: {key: bundle.p12}}
    namespaceSelector: {matchLabels: {trust: corp}}

4.ClusterTrustBundle

  • GA in v1.37 (certificates.k8s.io/v1) : cluster-scoped, CA certs only, readable by every ServiceAccount. v1beta1 : deprecated, never served by default.
  • Signer-unlinked : no colon, plain RBAC. Signer-linked : the name starts with the signer name (/ written as :) then a colon, e.g. example.com:mysigner:foo; writing needs attest on the signer (admission ctbattest).
  • Mount a projected source by name, or signerName plus labelSelector : {} matches all, unset none; the Pod will not start unless optional: true (docs wrong).
  • The kubelet keeps the file updated; trust-manager cannot write one yet. Pod Certificates signers are expected to publish here.
kind: ClusterTrustBundle  # v1, no colon in name
metadata: {name: corp-root}
spec: {trustBundle: "-----BEGIN CERTIFICATE-----..."}
- clusterTrustBundle: {name: corp-root, path: ca.crt}

5.Runtimes read trust differently

RuntimeAdd a private rootWatch out
Debian, Ubuntua .crt in /usr/local/share/ca-certificates/, then update-ca-certificatesother file extensions are silently ignored
GoSSL_CERT_FILE or SSL_CERT_DIRthe pool is cached per process
Node.jsNODE_EXTRA_CA_CERTSread only when the process starts
Javakeytool -importcert -cacerts, or a PKCS#12 truststorerestart the JVM

Python requests : REQUESTS_CA_BUNDLE (certifi, not the OS store). k3s image pulls : registries.yaml tls.ca_file, restart k3s per node.

Seen in practice

A root served over plain HTTP is fine only if people compare a published fingerprint; openssl x509 -noout -fingerprint prints SHA-1 without -sha256. Pitfall : reflector mirroring the CA Secret copies the root key too. Do better : publish the SHA-256 fingerprint out of band; fan out only ca.crt.

In short : Copy only the public root into a dedicated source, then fan it out with trust-manager or a ClusterTrustBundle, and set each runtime's trust knob. reflector copies the private key too.

1.Rotating a root with overlap

  • Same subject, different key : old leaves do not validate ("ECDSA verification failure").
  • So the new root reaches every store first, and the old one stays until its last leaf expires.
  • In OpenBao's words, the hard part is rotating root certificates. It can take months.
old root trusted new root trusted 1. new root made, new key 2. added to every trust store 3. wait for propagation 4. issuer switched, leaves re-issued 5. old root removed after last leaf

2.The rotationPolicy trap

Since cert-manager v1.18.0 a renewal makes a new private key by default (rotationPolicy: Always, locked since v1.20.0). On a CA that is a new root key : trust-manager reading that Secret swaps the root at once.

Always (default)Never
key at renewalnew key and key id, same subjectreuses tls.key; subject, key id unchanged
new leaves vs old root certfailverify, until the old notAfter
trust storesneed the new root before leaves moveneed the renewed cert before the old expires
key compromiseeach renewal limits exposurekey lives as long as the hierarchy
Never buys time, not freedom. OpenSSL and Go reject the chain once the old root cert passes notAfter. Fixes : two Bundle sources (current, previous), or Never plus the renewed cert in every store within renewBefore.

3.OpenBao PKI as the issuer

  • Root outside, intermediate in OpenBao (internal : no key export; leaf_not_after_behavior err or truncate). Classic endpoints : only permitted_dns_domains (CEL role templates : full constraints).
  • cert-manager's vault issuer calls the sign path, pki_int/sign/<role>. Roles limit names (allowed_domains).
  • serviceAccountRef : 10-minute tokens for a dedicated ServiceAccount, via a Role to create serviceaccounts/token.
kind: ClusterIssuer  # cert-manager.io/v1
metadata: {name: bao-int}
spec:
  vault:
    server: https://bao.example.com:8200
    path: pki_int/sign/internal
    caBundleSecretRef: {name: bao-ca, key: ca.crt}
    auth: {kubernetes: {role: cert-manager,
      serviceAccountRef: {name: vault-issuer}}}

Same Vault API, so expected to work, but neither project documents or tests OpenBao with this issuer.

4.Issuer choices

CA issuer : no extra service, key in a Secret, no CRL. Vault issuer + OpenBao : key stays inside, CRLs, one more service. SPIRE : short-lived, SPIFFE IDs, not DNS names. PodCertificateRequest : native, you deploy the signer.

5.Commands and one real cluster

CommandWhat it does
kubectl -n cert-manager get secret <ca> -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -subject -dates -ext basicConstraints,keyUsageshow CA:TRUE, pathlen, dates
kubectl -n cert-manager get certificate <ca> -o jsonpath='{.status.renewalTime}'when it will renew, and re-key
cmctl renew -n <ns> <certificate>re-issue now (leaves after a CA change)
openssl verify -CAfile root.crt -untrusted int.crt leaf.crtcheck a chain
Seen in practice
  • Pitfall : one online root that is also an outside trust anchor, for example a cloud IAM Roles Anywhere anchor behind SOPS decryption and OpenBao auto-unseal. When cert-manager renews it with a new key, the outside copy goes stale silently and everything behind it fails at once.
  • Do better : an offline root with an online intermediate, alerts on CA expiry and on anchor drift, and a rehearsed rotation with two anchors overlapping.
  • Deleting a CA Secret re-keys it at once. Guard it with RBAC and a backup.
In short : Rotate a root with overlap : add the new one everywhere, re-issue, drop the old. cert-manager re-keys a CA on renewal, and even Never needs the renewed cert in every store in time.

Identity and trust

Pod Certificates

1.What are Pod Certificates ?

A Pod asks for an X.509 certificate by adding a podCertificate source to a projected volume. The kubelet makes the key, sends a PodCertificateRequest (PCR), waits for a signer, then writes key plus chain into the Pod's tmpfs volume and keeps it fresh. No Secret.

  • Alpha 1.34, beta 1.35 and 1.36 (off by default). GA in v1.37 : certificates.k8s.io/v1, gate on, not locked in 1.37.
No signer in coreKubernetes ships the API, the kubelet and the admission plumbing only. Install a signer, or Pods wait forever. The one well-known name, kubernetes.io/kube-apiserver-client-pod, is documented as unimplemented.

2.The flow

Pod spec podCertificate source kubelet key in memory, empty stub kube-apiserver NodeRestriction, stub check PCR req-xxxxx in the Pod's namespace signer Issued + chain kubelet writes the PEM file app reads, must reload ClusterTrustBundle ca.pem, same volume
  • No approval step. The signer sets one of Issued, Denied, Failed via /status, then it is immutable.
  • The stub is an empty PKCS#10 request : its signature proves the kubelet holds the key.
  • The API server checks the answer : requested key, lifetimes, clock skew.
  • Trust side : mount the signer-linked bundle with signerName plus labelSelector: {} (Private PKI).

3.The volume

FieldNotes
signerNameRequired. <domain>/<path> : exactly one /, a domain of at least two labels
keyTypeRequired. RSA3072, RSA4096, ECDSAP256, ECDSAP384, ECDSAP521, ED25519
maxExpiration­SecondsDefault 86400, min 3600, max 7862400. At most 86400 for kubernetes.io/ signers
credentialBundle­PathRecommended. One PEM : PRIVATE KEY first, then the chain, leaf first
keyPath, certificate­ChainPathSeparate files. Can be read mid-rotation and not match
userAnnotationsCopied into the PCR as unverifiedUserAnnotations. Untrusted input
userAlpha, gate AtomicWriteVolume­UserFields. Owner UID of the files
securityContext: {fsGroup: 2000}
volumes:
- name: creds
  projected:
    sources:
    - podCertificate: {keyType: ED25519,
        signerName: coolcert.example.com/foo,
        credentialBundlePath: credentialbundle.pem}
    - clusterTrustBundle: {labelSelector: {},
        signerName: coolcert.example.com/foo,
        path: ca.pem, optional: true}
  • There is no field to ask for DNS or IP SANs. The signer picks them.
  • The docs example sets defaultMode: 0644 : with no file owner and no fsGroup, any UID in the container can read the key. With fsGroup or user, the mode is forced to 0600.
  • Never mount with subPath : the file never updates.

4.Who vouches

  • NodeRestriction : a node asks only for a Pod bound to it, with matching Pod, ServiceAccount and node UIDs. Never for a mirror (static) Pod. No Pod phase check.
  • Since 1.37 the Pod must mount a podCertificate source naming that signer, or the node passes an extra authorization check.
  • So a signer can trust podName, serviceAccountName, nodeName and their UIDs. Never unverifiedUserAnnotations : any Pod author writes those.
In short : A Pod names a signer and a key type. The kubelet makes the key and asks, the API server vouches for the Pod and its ServiceAccount, a signer signs, and the app reads one file.

1.The PodCertificateRequest

  • ObjectNamespaced. The kubelet creates it with generateName: req- and an ownerReference to the Pod (Garbage Collection).
  • SpecImmutable : signerName, podName/podUID, serviceAccountName/serviceAccountUID, nodeName/nodeUID, maxExpirationSeconds, stubPKCS10Request, unverifiedUserAnnotations.
  • StatusOne update, then immutable : one condition, certificateChain (leaf first), notBefore, notAfter, beginRefreshAt.
  • Deny reasonsUnsupportedKeyType, InvalidUnverifiedUserAnnotations. No short name : kubectl get pcr fails.

2.States

PCR created one answer, once, then immutable Issued Denied Failed kubelet, per projection (metric states) not_yet_issued fresh refresh : new key, new PCR, at beginRefreshAt retried denied, failed first issue or refresh permanent expired still served overdue_for_refresh 10 min late
  • Metric : kubelet_podcertificate_states {signer_name, state}, six states.
  • Refresh starts at beginRefreshAt plus 0 to 5 minutes of jitter, with a new key.
  • Files are swapped atomically. The app must notice (inotify or polling).
  • KEP suggestion : 24 h certs refreshed at 12 h ride out a 12 h control plane outage.

3.The numbers

WhatValue
Default lifetime24 h
Minimum1 h
Maximum91 days
kubernetes.io/ signersat most 24 h
Refresh jitter0 to 5 min after beginRefreshAt
Refresh passevery 1 to 2 min
Overdue event10 min past the jittered time
PCR deleted15 min after creation, any state
Cleaner runsevery 5 min
Signer notBeforewithin 5 min of the API server clock
beginRefreshAt10 min inside notBefore and notAfter

4.What goes wrong

  • No signer answers : the Pod sits in ContainerCreating, with a fresh key and PCR every 15 to 20 minutes.
  • Denied or Failed is permanent, on first issue and on refresh. Only a new Pod helps.
  • Expired is quiet : the kubelet keeps serving the old files. One CertificateExpired event, the expired state.
  • One failing source blocks the whole projected volume write.
  • A kubelet restart re-keys and re-requests everything.
  • The app must reload the file. Kubernetes sends no signal.

5.Commands

CommandShows
kubectl get podcertificaterequests -ARecent PCRs. Only the last 15 to 20 minutes survive
kubectl get podcertificaterequests -n <ns> <name> -o yamlCondition, timestamps and chain
kubectl explain pod.spec.volumes.projected.sources.podCertificateField docs from the live server
kubectl api-resources --api-group=certificates.k8s.ioIs the API served ?
kubectl get events --field-selector involvedObject.name=<pod>Denied, Failed, Certificate­OverdueForRefresh, CertificateExpired
kubelet_podcertificate_states{state=~"expired|overdue_for_refresh|denied|failed"}Kubelet /metrics. Alert on it (Prometheus)
In short : The kubelet re-keys and re-requests at beginRefreshAt, and a PCR lives only 15 minutes. Denied is permanent and expired is quiet, so alert on the kubelet metric and make the app reload.

1.Who ships a signer (2026-10-08)

  • Nobody in core. Upstream expects in-tree signers for Service DNS and SPIFFE client certs, no version given.
  • cert-manager v1.21.2 : no PCR support. Its CSR signers serve CertificateSigningRequests.
  • SPIRE v1.15.3 : no PCR signer.
  • A toy reference signer (by the KEP author) and a community signer exist : third-party, unvetted.

A signer is a controller. What it needs (RBAC) :

RightNotes
sign on signersName <domain>/<path> or <domain>/*. Checked when a status update changes the status
update on statusOn podcertificate­requests/status : write the answer
attest on signersPublish its ClusterTrustBundle

2.Writing a signer

  1. Own a name : example.com/<path>, never kubernetes.io/.
  2. Watch PCRs, filter on spec.signerName.
  3. Trust only API-checked fields. Deny with InvalidUnverifiedUserAnnotations or UnsupportedKeyType.
  4. Sign the stub's key. SANs by policy : spiffe://<td>/ns/<ns>/sa/<sa> or <svc>.<ns>.svc.
  5. Lifetime and beginRefreshAt inside the rules, clock within 5 minutes.
  6. Answer well inside 15 minutes.
  7. Publish the CA in a signer-linked ClusterTrustBundle, rotate with overlap (Private PKI).
PCR signerName, Pod, SA, node signer controller policy identity to SANs, lifetime /status Issued + chain CA key signs signer-linked ClusterTrustBundle clusterTrustBundle source Pods verify peers

Skew : a v1-only signer cannot serve kubelets older than 1.36. v1 dropped pkixPublicKey, proofOfPossession.

3.Compared

PCRCSR
Scopenamespacedcluster-scoped
Askerthe kubelet, for a Pod on its nodeany user with create
Approvenonean Approved condition
Contentempty stub, key onlythe full CSR
Lives15 min after creation1 h approved or denied, 24 h pending
Useworkload identitykubelet and user certs
Pod CertificatesSPIRE
Deliveryprojected volume, kubelet writesWorkload API socket via csi.spiffe.io
ProofAPI server NodeRestrictionnode and workload attestation
FormatsX.509 onlyX.509 and JWT SVIDs
Rotationkubelet swaps files, app reloadspushed over the Workload API
Installa signerserver, agents, CSI driver

IDs and SVIDs : SPIFFE, SPIRE. cert-manager's csi-driver and csi-driver-spiffe also put certs in Pods. Replacing a mesh is a KEP non-goal.

4.Seen in practice

  • A v1.37 cluster serves certificates.k8s.io/v1 PodCertificateRequests and ClusterTrustBundles with no flags, but nothing uses them until you deploy a signer.
  • Often the only ClusterTrustBundle is the one kube-controller-manager publishes for kubernetes.io/kube-apiserver-serving.
  • Clusters on SPIRE lean on JWT-SVIDs (OpenBao logins, object storage STS; see OIDC). X.509-only Pod Certificates do not replace them.
  • Where it fits : operators that take projected volumes but no CSI, like CloudNativePG's projectedVolumeTemplate. A signer backed by your private CA avoids a SPIRE-to-Secret bridge.
In short : No signer ships in core, so install or write one (sign, status update, attest). Pick Pod Certificates for kubelet-made X.509 without a CSI driver, SPIRE for JWTs and attestation.

Supply chain security

Images, Registries and Pulls

1.A reference, part by part

  • Shape : [host[:port]/]path[:tag][@digest]. Tag and digest both given : only the digest is pulled.
  • First part is a host only with a . or :, as localhost, or in uppercase. Else docker.io (+ library/ for one word).
  • The kubelet only adds :latest. containerd maps short names to Docker Hub. CRI-O uses its search list and can refuse an ambiguous name.
In the Podcontainerd pullsDefault policy
busyboxdocker.io/library/busybox:latestAlways
myregistry/app:1docker.io/myregistry/app:1 (no . or :, so Docker Hub)IfNotPresent
registry:5000/app:1registry:5000/app:1 (: makes it a host)IfNotPresent
nginx@sha256:<d>docker.io/library/nginx@sha256:<d>IfNotPresent
nginx:latest@sha256:<d>docker.io/library/nginx@sha256:<d> (tag dropped)Always

2.Behind one tag

  • A multi-arch tag points at an index : one manifest per platform.
  • The runtime picks the node's platform. None fits : "no match for platform in manifest".
  • A manifest names one config and its layers, all by digest.
  • Clients re-hash what they fetch and reject a mismatch ("unexpected commit digest").
  • The config digest is the ImageID (crictl IMAGE ID).
tag 3.10.2 pause registry resolves index sha256:f548e0e8... runtime picks platform linux/arm64 a4332146... +6 more linux/amd64 manifest sha256:412c4a72... fetch by digest config 4a83b15d... = ImageID layer 81ede362... gzip, as stored
signature SBOM provenance subject image manifest sha256:A client GET /referrers/sha256:A index of 3 pause:3.10.2 on 2026-10-07 : empty (legacy .sig tags)

3.Things that point at an image

  • OCI 1.1 : an artifact names the image digest in subject, its kind in artifactType or the config type.
  • GET /v2/<name>/referrers/<digest> lists them. A 404 : fall back to the tag sha256-<hex>.
  • The kubelet and the CRI never read referrers. Checks live in admission, tools or runtime policy : Image Signing, Attestations, Image Policy.

4.imagePullPolicy

  • Defaulted once, at create : Always when the parsed tag is latest (no tag, :latest, even :latest@sha256:, not :Latest), else IfNotPresent.
  • Fixed on a Pod. Moving a template to :latest later keeps the old default.
  • Always : a registry round trip per container start, not a re-download. Needs the registry up.
  • IfNotPresent keeps the old bytes of a moved tag.

5.What actually ran : imageID

  • imageID copies the CRI image_ref (Architecture). On containerd : the index digest.
  • It can name another repo with the same bytes. Seen : docker.io/rancher/mirrored-pause@sha256:f548e0e8...
  • Compare only the digest part, with crane digest without --platform.
CommandShows
kubectl get pod web -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.imageID}{"\n"}{end}'The digest that ran, per container
crane digest <image> / crane digest --platform linux/amd64 <image>Index digest / one platform's manifest
crane manifest <image> | jq -r '.mediaType, (.manifests | length)'Index or not, and how many platforms
oras discover <ref>@sha256:<d>Referrers tree. Finds nothing for legacy .sig tags
In short : A tag is a question and a digest is the answer. Pin digests, remember that pull policy is fixed when the Pod is created, and read imageID to see what really ran.

1.One container start, step by step

  • Always skips the presence check. IfNotPresent and Never ask CRI ImageStatus.
  • Never and no image : ErrImageNeverPull. Present : pull records decide if this Pod may use it.
  • Else : credentials, per-Pod back-off, a Pulling event, the queue.
  • The Pulled event shows two times : "in 209ms (209ms including waiting)". The gap is time in the queue.
Pod spec policy set at create kubelet image manager default tag present ? pull records back-off credentials PullImage (one per credential) CRI ImageService (containerd, CRI-O) normalize mirrors index : platform fetch, re-hash image_ref pulled record status.imageID

2.Queue, rate and back-off

  • Default : 1 worker, queue of 10, 1 pull at a time.
  • maxParallelImagePulls >= 2 (stable v1.35) flips an unset serializeImagePulls to false. An explicit true stops the kubelet.
  • Keep it at 31 or less before v1.37.0 (panic).
  • registryPullQPS 5, burst 10 : a request rate. Over it : pull QPS exceeded.
  • Back-off per Pod UID + image, 10s doubling to 300s. A new Pod starts fresh.

3.Where pull credentials come from

  • Order : Pod imagePullSecrets, node config.json, then plugins.
  • The ServiceAccount's secrets are copied only at create, only if the Pod lists none.
  • Only dockerconfigjson and dockercfg Secrets count : kubectl create secret docker-registry.
  • One PullImage per matching credential, first success wins. No anonymous retry.
MethodScopeRotationStatic Pods
Pod imagePullSecretsThe PodUpdate the Secretno
ServiceAccount imagePullSecretsNew Pods of the SA, listing noneUpdate the Secretno
Node config.jsonEvery Pod on the nodeRe-read within 5 minutesyes
Exec credential providerPods whose image matchesPlugin's cache durationyes
SA-token credential providerThe Pod's ServiceAccountShort-lived, no Secretonly without a token

4.Credential provider plugins

  • CredentialProviderConfig (kubelet.config.k8s.io/v1) : name = binary, matchImages, defaultCacheDuration.
  • Flags --image-credential-provider-config (a directory since v1.34) and -bin-dir.
  • stdin in, stdout out, 1-minute timeout, cached in memory. A bad config stops the kubelet.

5.Pulling with the Pod's own identity

  • tokenAttributes (beta, on since v1.34) : the plugin trades a Pod-bound token for short-lived credentials.
  • cacheType required since v1.34. ServiceAccount : one SA, shared credentials.
  • NodeRestriction : grant the kubelet request-serviceaccounts-token-audience (RBAC).
  • Required SA annotation missing : skipped, plain 401.
matchImages rules (and config.json keys)
  • *.k8s.io matches x.k8s.io, *.io does not : one label per *. A path is a plain prefix : /images also matches /images2.
  • * in a path never matches : the v1.34 blog's *.myregistry.io/* matches nothing. Write *.myregistry.io.

6.Commands

CommandShows
kubectl apply --dry-run=server -f pod.yaml -o jsonpath='{range .spec.containers[*]}{.image}{"\t"}{.imagePullPolicy}{"\n"}{end}'Defaulted pull policy, nothing created
kubectl get --raw /api/v1/nodes/<node>/proxy/configz | jq '.kubeletconfig | {maxParallelImagePulls, imagePullCredentialsVerificationPolicy}'The kubelet's effective image settings
kubectl events --for pod/<p>Pulling, Pulled (queue time), BackOff
crictl images --digests / crictl inspecti <image>Node images : DIGEST (index) vs IMAGE ID (config)
In short : The kubelet pulls one image at a time unless told otherwise, tries each matching credential in a fixed order, and can pull as the Pod's ServiceAccount with no stored Secret.

1.The cached-image leak

  • IfNotPresent or Never : any Pod can run a private image another tenant pulled there (#18787).
  • Old fix : AlwaysPullImages (admission, off by default). Skips subresources : kubectl debug escapes it.
  • New fix : KubeletEnsureSecretPulledImages, beta and on since v1.35.
Pod A (team-a) has a pull Secret Pod B (team-b) no Secret node cache private image pulls present : runs ! with verification (gate on) : kubelet checks pull record B pulls with its own creds 401

2.Pull records

  • Files in /var/lib/kubelet/image_manager/, keyed by the image name as the Pod wrote it.
  • A Secret hash matches from any namespace. An SA matches by UID.
  • Pulled anonymously, by node config.json or a non-token plugin : nodePodsAccessible, open to any Pod.
  • No record under Never : ErrImageNeverPull, same text as a missing image.
PolicyBy kubeletPreloaded
NeverVerifyno checkno check
NeverVerifyPreloadedImagescheckedno check
NeverVerifyAllowlistedImagesif not listedif not listed
AlwaysVerifycheckedchecked

Red = the default. The allowlist ignores how the image arrived.

3.Turning it on safely

  • Preloaded : on the node before the gate, from crictl pull --creds USER:PASS <img> or a node image. Clean private images off first.
  • Spell it NeverVerifyAllowlistedImages : the docs' AllowListed stops the kubelet.
  • Allowlist : no tag, only a trailing /*, spelled like the Pods (busybox is not docker.io/library/busybox).
  • Never delete image_manager/. Watch ..._mustpull_checks_total{result="mustAuthenticate"}.
Stop the leakAlwaysPullImagesKubelet checkImage source policy
Runs inadmissionkubeletadmission
Registry needed on every startyesnew credentials onlyno
kubectl debug containersnot coveredcoveredper policy
Beaten by a credential-less mirroryesyes (inference)no
registry.k8s.io/pause:3.10.2 certs.d/registry.k8s.io/hosts.toml ?ns=registry.k8s.io mirror 1 pull, resolve mirror 2 pull only server upstream, last timeout : "trying next host"

4.Mirrors and caches

  • registry.k8s.io : no uptime SLA, backends move (blobs from cdn.registry.k8s.io on 2026-10-07). Mirror what you run.
  • containerd hosts.toml, no restart. Untrusted mirror : capabilities = ["pull"]. No line = full trust.
  • CRI-O : pull-from-mirror = "digest-only", SIGHUP.
  • Digest pulls are safe via any mirror. A tag trusts its resolver.
  • A cache with no auth challenge (Spegel) serves anyone, past both checks above.

5.Image volumes take the same path

  • volumes[].image (GA v1.36, Volumes) : same pull path, credentials, back-off and verification. Mounted read-only.
  • A failed volume pull holds back only the containers that mount it. The Pod stays Pending (the docs say Failed).
  • ImageVolumeWithDigest : alpha, first shipped in v1.36, empty on v1.37.1. Pin the reference by digest instead.
In short : Verification stops one tenant running another's cached private image, mirrors live in the runtime and are safe only for digests, and image volumes share the pull path.

Supply chain security

Image Signing: Sigstore and Notation

1.What a signature proves

  • Binds an identity to one digest, never a tag.
  • Both tools resolve a tag and warn : use @sha256:.
  • Sigstore : cosign, an OIDC identity, a public log.
  • Notary Project : Notation, your own X.509 PKI.
Digest, not tag

2.cosign with a key

  • cosign generate-key-pair : encrypted key file, or a KMS.
  • Sign and verify IMAGE@sha256:... with --key.
  • Key mode still logs to public Rekor by default.
  • Private : signing config without rekorTlogUrls, verify with --insecure-ignore-tlog.
  • v3.0.6+ : --key plus identity flags is an error.
Key URIHolds
cosign.keyfile, plus cosign.pub
awskms:///alias/xAWS KMS, by alias
gcpkms://projects/P/...Google Cloud KMS
hashivault://, openbao://Vault transit (OpenBao v3.0.6)
k8s://ns/namea Secret : key, password, pub

3.Keyless, step by step

  • Fulcio : OIDC token to a 10-minute cert. Identity in the SAN, issuer in OID 1.3.6.1.4.1.57264.1.8.
  • Signed : a DSSE in-toto Statement, predicate type https://sigstore.dev/cosign/sign/v1.
CI job or a laptop TUF CDN tuf-repo-cdn.sigstore.dev 1 OIDC token (aud=sigstore) trusted_root.json, signing config cosign Fulcio CA, checks the token 2 token + public key 3 10-min cert + SCT precert CT log embeds the SCT cosign 4 signs DSSE(in-toto Statement{subject: digest}) 7 push 5 timestamp 6 log entry registry 7 bundle v0.3, a referrer TSA RFC 3161 timestamp Rekor v1 SET + inclusion proof

Bundle v0.3 = leaf cert + tlog entry + timestamp + DSSE envelope. The identity stays in public logs for good.

4.Reading the identity

  • A reusable workflow puts its own path in the SAN. cosign pins the caller only by deprecated OID .5 (--certificate-github-workflow-repository).
  • The caller sits in OIDs .12 (source repo) and .18 (build config) : that needs a policy engine.
Signer--certificate-oidc-issuerSAN (--certificate-identity)
GitHub Actionshttps://token.actions.githubusercontent.comhttps://github.com/ORG/REPO/.github/workflows/F.yml@refs/tags/T
GitLab CIhttps://gitlab.comhttps://gitlab.com/PROJECT//.gitlab-ci.yml@REF
Google SAhttps://accounts.google.comkrel-trust@k8s-releng-prod.iam.gserviceaccount.com
Kubernetes SAyour cluster issuerhttps://kubernetes.io/namespaces/NS/serviceaccounts/SA (RBAC)
SPIFFEyour configured issuerspiffe://... from sub (SPIFFE, SPIRE)

5.Trust roots, Rekor v1 and v2

  • Trust : TUF ships trusted_root.json. Its timestamp.json lives a week, so air-gapped verifiers pass --trusted-root.
  • Public default : Rekor v1, for the foreseeable future. v2 is opt-in (signing-config create --with-default-rekor-v2).
  • Rekor v2 has no SET, so a TSA is mandatory. Shards rotate : never hardcode log2025-1.
  • No revocation : short certs plus log monitoring. Private instance : cosign trusted-root create.
In short : Sign the digest, not the tag. Keyless signing turns a CI identity into a 10-minute certificate and a public log entry, so the identity you pin when verifying is the whole security story.

1.Where signatures live

  • cosign v3 : a bundle as an OCI 1.1 referrer, type ...sigstore.bundle.v0.3+json.
  • Legacy cosign (v2 default) : tag sha256-<hex>.sig, no subject, so the Referrers API never lists it.
  • .sig tag and fallback index sha256-<hex> differ. The index is a racy read-modify-write.
  • --registry-referrers-mode=oci-1-1 is not the v3 switch : experimental, legacy-format writes only.
FormatTagsubjectListedWritten by
Legacy cosignsha256-<hex>.signonov2 default; v3 with 2 legacy flags
cosign v3 bundlenoneyesyesv3 default; v2.6 with a flag
Notationnoneyesyesnotation
Fallback indexsha256-<hex>n/an/aboth, without the API; notation v1.x always

2.Verifying with cosign

  • Keyless verify requires an identity and an issuer flag (or their -regexp forms).
  • The regexps are unanchored : write ^...$. A .* accepted a stranger's bundle in the lab : any Sigstore user passes.
  • One identity per run. Always verify IMAGE@sha256:....
Anchored GitHub Actions verify
$ cosign verify ghcr.io/ORG/REPO@sha256:<d> \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github\.com/ORG/REPO/\.github/workflows/release\.yml@refs/tags/v[0-9.]+$'

3.What "verified" means

  • cosign runs eight checks, in the order on the right.
  • Key mode swaps cert, SCT and identity for the key, and needs no timestamp.
  • The predicate type is not checked. An SBOM attestation from the pinned identity passes cosign verify (see SBOMs, Provenance and Attestations).
  • --insecure-ignore-tlog still prints "verified offline". Never scrape stderr.
1 trusted root (TUF or file) 2 tlog inclusion proof 3 signing time (SET or TSA) 4 cert chains to Fulcio 5 SCT from the CT log 6 signature over payload 7 identity + issuer 8 subject digest = image not checked : predicateType --insecure-ignore-tlog skips key mode : no timestamp --insecure-ignore-sct skips key mode : 4, 5, 7 usethe key instead

4.Old and new verifiers

  • Old verifiers say Error: no signatures found (exit 10) on a v3 signature.
  • If any bundle exists, v3 ignores legacy .sig signatures.
  • Floor : cosign v3.1.3+ or v2.6.5+. GHSA-fx35-mq7g-6g98 has no CVE ID, so CVE scanners miss it.
  • CI lags : cosign-installer v4.1.2 defaults to v3.0.6, Alpine 3.20 ships 2.2.1. Pin it and print cosign version.
Signed with< v2.6.0v2.6.0 to v2.6.2v2.6.3 to v2.6.5v3.x
Legacy .sigyesyesyesyes, if no bundles
v3 bundle referrernoonly with --new-bundle-formatyes (auto-detect)yes
Both on one image.sig only.sig onlybundles onlybundles only

5.Commands

CommandWhat it shows
cosign sign --yes IMAGE@sha256:<d>Keyless sign, bundle pushed as a referrer
cosign sign --key cosign.key IMAGE@sha256:<d>
cosign verify --key cosign.pub IMAGE@sha256:<d>
Key mode, sign and verify
cosign tree IMAGE.sig and .att tags, plus referrers
cosign bundle inspect b.sigstore.jsonCert extensions, SCTs, tlog, timestamps (no crypto checks)
cosign versionClient version : check it in every CI image
In short : cosign v3 stores a bundle as a referrer of the digest, old verifiers cannot see it, and a verify is only as strong as an anchored identity plus issuer.

1.Notation : your own PKI

  • Keys via plugins : azure-kv v1.2.1, AWS Signer.
  • notation sign : JWS (default) or COSE envelope, optional RFC 3161 countersignature.
  • v1.3.2 writes a referrer with config.mediaType ...notary.signature (no artifactType) and the sha256-<hex> index.
signer verifier key in a KMS plugin + CA chain notation sign target descriptor JWS / COSE + optional TSA registry referrer subject = digest notation verify trustpolicy.json registryScopes trust store roots integrity, authenticity, expiry, timestamp, revocation (OCSP, CRL)

2.trustpolicy.json, trust stores

  • Stores hold root certs only, in ~/.config/notation/truststore/x509/.
  • Policy pick : exact repo, then one "*", else fail.
  • Identities need C, ST, O. No serverAuth EKU, unlike cert-manager TLS certs.
  • notation policy import trustpolicy.json, then notation verify -v by digest.
One strict statement
{"version": "1.0", "trustPolicies": [{
  "name": "app",
  "registryScopes": ["example.com/app"],
  "signatureVerification": {"level": "strict"},
  "trustStores": ["ca:acme", "tsa:my-tsa"],
  "trustedIdentities": [
    "x509.subject: C=US, ST=WA, O=acme.io"]
}]}

3.Verification levels

  • audit passes a wrong signer with exit 0.
  • skip with "*" : v1.3.2 accepts it, the spec forbids it.
  • No TSA : a signature dies with its leaf cert.
  • Unknown revocation fails strict (OCSP 2s, CRL 5s).

E enforce (fails), L log and go on, - skipped.

Checkstrictpermis.auditskip
IntegrityEEE-
AuthenticityEEL-
TimestampELL-
ExpiryELL-
RevocationELL-

4.cosign or Notation ?

Notation v1.3.2 (2025-04-27) is current (v2 : alpha). Flux checks both on OCI sources. Cluster : Image Policy.

Aspectcosign v3.1Notation v1.3
Root of trustTUF trusted_root.json or --trusted-rootRoot certs in trust stores
Identity checkSAN + issuer (OID .8), or a raw keytrustedIdentities x509 subject, or a plugin identity
Signing timeRekor v1 SET, or RFC 3161 (must with Rekor v2)Optional RFC 3161 countersignature
TransparencyRekor + CT log by defaultNone built in
RevocationNone : short certs, log monitoringOCSP first, CRL fallback

5.Copy, then verify again

  • oras cp -r (preview) misses legacy .sig. cosign copy (deprecated v3.0.5) misses v3 bundles.
  • regctl image copy --referrers --digest-tags SRC DST carries both. Keep the digest unchanged.
  • Re-verify at the destination. Notation also needs the new repository in registryScopes.

6.Worked example : Kubernetes v1.37.1

  • Images : krel-trust@k8s-releng-prod.iam.gserviceaccount.com, issuer Google, legacy .sig.
  • That .sig holds krel-staging and krel-trust signatures : either passes. Pin the digest (Deployments).
  • Binaries : krel-staging, with flags deprecated and due to go in v4. No .sigstore.json yet.
$ cosign verify registry.k8s.io/kubectl@sha256:b7cab618... --certificate-identity krel-trust@... --certificate-oidc-issuer https://accounts.google.com
$ for f in kubectl kubectl.sig kubectl.cert; do curl -sSfLO "https://dl.k8s.io/release/v1.37.1/bin/linux/amd64/$f"; done
$ cosign verify-blob kubectl --signature kubectl.sig --certificate kubectl.cert --certificate-identity krel-staging@... --certificate-oidc-issuer https://accounts.google.com
In short : Notation trades the public log for your own CA and a trust policy. Either way, copy signatures with the image, verify again, and check Kubernetes against the krel identities by digest.

Supply chain security

SBOMs, Provenance and Attestations

1.Signed is not the same as true

An attestation is a signed claim about one artifact digest. The signature tells you who said it, never that it is right. Signing itself is in Image Signing; this chapter is about what gets signed.

Every verify does four checks
  1. Signature + identity : a trusted signer and issuer, not "any signer".
  2. Subject : a subject[].digest equals the digest you hold.
  3. Type : the predicateType you expect (provenance, SBOM, VEX).
  4. Policy on the predicate : builder, source repo, workflow, ref.
Skip one and the claim is decoration. gh's own help says whoever controls the workflow can fake the predicate; only the cert and timestamps are unforgeable.

2.One claim, four layers

  • Statement v1 : _type, subject[] (name + digest), predicateType, predicate.
  • DSSE signs the PAE string, DSSEv1 28 application/vnd.in-toto+json <len> <body>, so payloadType is signed too.
  • keyid is only a hint, never a security decision.
  • Then use the verified bytes only. Never re-parse.
  • Referrers : Images and Pulls.
OCI referrer subject = image digest Sigstore bundle cert + tlog entry DSSE envelope payloadType, payload, sigs Statement subject, predicateType, predicate exactly one signature subject image @sha256:D

3.Predicate types

SBOMs, provenance, scans, VEX : all predicates. cosign v3.1.3 still writes Statement v0.1, actions/attest v1.

PredicatepredicateTypecosign --type
SLSA provenance v1https://slsa.dev/provenance/v1slsaprovenance1
SLSA provenance v0.2https://slsa.dev/provenance/v0.2slsaprovenance and slsaprovenance02 : the trap
SPDXhttps://spdx.dev/Documentspdx, spdxjson
CycloneDXhttps://cyclonedx.org/bomcyclonedx
OpenVEXhttps://openvex.dev/nsopenvex
cosign customhttps://cosign.sigstore.dev/attestation/v1custom : the default

4.SLSA 1.2 : Build and Source tracks

1.2 adds the Source track. The Build track and the predicate https://slsa.dev/provenance/v1 do not change.

LevelBuild trackSource track
L1Provenance exists. "Trivial to bypass or forge"Version controlled
L2Hosted platform signs it : stops tampering after the buildHistory and provenance per revision
L3Isolated runs, signing secrets out of reach : stops it duringTechnical controls on named refs
L4none (old v0.1 "SLSA 4" is gone)Two-party review
EvidenceProvenance v1, signed by the platformSource VSA

5.Provenance v1, field by field

  • externalParameters are untrusted inputs : check them, reject fields you do not know.
  • runDetails.builder.id is meant to decide the Build level on its own.
  • resolvedDependencies : sources with digests, here the Git commit.
  • materials or invocation mean v0.2 from an older generator, not a bad signature.

GitHub provenance v1, trimmed :

"buildType": ".../buildtypes/workflow/v1",
"externalParameters": {"workflow": {
  "ref": "refs/tags/v1.4.0",
  "path": ".github/workflows/release.yml"}},
"resolvedDependencies": [{"uri": "git+...",
  "digest": {"gitCommit": "<sha>"}}],
"builder": {"id": "https://github.com/acme/
  build-templates/.../oci-release.yml@..."}
In short : An attestation is a Statement about a digest, signed through DSSE, shipped in a bundle. Provenance says how it was built. Check signer, subject, type and content every time.

1.GitHub artifact attestations

  • actions/attest@v4 : one action for provenance, SBOM and custom. The older two are wrappers.
  • Write on id-token + attestations; on artifact-metadata only for storage records, packages for GHCR.
  • Alone : Build L2. Built and attested inside a reusable workflow : L3.
  • Private repos : no transparency log, GHE Cloud only.
build job actions/attest OIDC token id-token: write Fulcio cert short-lived bundle signed Statement attestations API push-to-registry registry referrer gh attestation verify reads

2.Verifying provenance

  • gh attestation verify oci://IMG@sha256:D -R org/repo : SLSA v1 + GitHub issuer by default.
  • A reusable workflow signed it ? Add --signer-workflow.
  • No TTY : success prints nothing. Gate on the exit code, keep --format json as evidence.
  • slsa-github-generator, slsa-verifier : unmaintained since 2026-08-07 (tags predate the notice).
ghcosignslsa-verifier
ReadsAPI, OCI, diskreferrers, .attgenerator, GCB
DefaultSLSA v1customby builder
Identity-R, signer flagsidentity + issuer--source-uri
Kept upyesyesno

3.BuildKit attestations

  • Push with --provenance=mode=max --sbom=true : one unknown/unknown manifest per platform.
  • SLSA v1 by default since v0.28.0 (Docker's docs say v0.2).
  • Layers are bare, unsigned Statements : sign the predicate yourself.
  • Read : docker buildx imagetools inspect IMG --format '{{ json .Provenance.SLSA }}'
  • mode=max records build-arg values. --load drops attestations.
image index : moby/buildkit:v0.33.1 6 platforms, each followed by its attestation linux/amd64 image manifest unknown/unknown attestation manifest subject layers : SPDX, SPDX, SLSA provenance v1 unsigned : bare Statements

4.What an SBOM lists

  • SPDX 2.3 is still the tools' default. SPDX 3.0.1 is JSON-LD and not field-compatible with it.
  • syft's SPDX and Kubernetes' SBOM miss the CISA 2026 minimum elements. The signature goes outside, in an attestation.
SPDX 2.3SPDX 3.0.1CycloneDX 1.7
ShapeJSON or tag-value documentJSON-LD graphJSON, XML, Protobuf
Default insyft, trivy, BuildKit, bomsyft opt-in (spdx-json@3.0)syft and trivy CycloneDX output
actions/attestaccepts JSON, not tag-valuerejectsaccepts, needs serialNumber
grype / trivyboth; cross-tool can read 0grype yes, trivy failsboth; cross-tool loses findings

5.Making and attaching SBOMs

  • syft scan registry:IMG@sha256:D -o spdx-json=s.json -o cyclonedx-json=c.json : two formats, no daemon.
  • trivy image --format cyclonedx is an SBOM, not a scan, without --scanners vuln.
  • Signed : cosign attest --type cyclonedx --predicate c.json IMG@sha256:D. Verify with cosign verify-attestation --type cyclonedx ; without --type it looks for custom.
  • cosign attest --statement is ignored for images. Only attest-blob --statement takes a full Statement.
  • Unsigned : oras attach --artifact-type application/vnd.cyclonedx+json IMG c.json. Fine to share, not evidence.
In short : GitHub and BuildKit make provenance and SBOMs, but only signed ones are evidence. Verify with signer and type pinned, then enforce it (Image Policy).

1.Rescan the SBOM, not the image

  • grype sbom:./x.json or trivy sbom x.json : stored SBOM vs today's DB. No pull, no rebuild.
  • Cross-tool rescans fail silently : Trivy found 0 in syft's SPDX, grype 0 in Trivy's SBOMs, against 48 to 52 direct. All exit 0 (2026-10-07).
  • Check for a detected OS and a sane package count.
  • grype --fail-on exits 2, not 1 (1 = the scan broke).
build SBOM attest registry months later verify + extract scanner + today's DB findings fix, or VEX VEX filters the next scan

2.VEX : saying "not affected"

  • OpenVEX v0.2.0 : vulnerability, products (digest PURL best), status. not_affected needs a justification or impact_statement.
  • VEX removes findings, so authenticate it. Trivy's experimental --vex oci let a VEX signed by a throwaway key drop a CVE (48 to 45).
  • Same with --sbom-sources oci : an unverified attached SBOM gave 9 findings, not 48. Who can push a referrer picks what is scanned (RBAC).
--vexTrivyGrype
FormatsOpenVEX, CycloneDX, CSAFOpenVEX, CSAF
Suppressesnot_affected, fixedsame, into ignoredMatches
Sig checknone on found VEXlocal files only
Two files--vex a --vex b or a,brepeat --vex only
$ vexctl create --product=pkg:oci/app@sha256:... \
   --vuln=CVE-... --status=not_affected \
   --justification=... --file=app.vex.json

3.Continuous scanning in the cluster

  • Trivy Operator v0.35.0 runs scan Jobs and writes a VulnerabilityReport and SbomReport per container, owned by the workload (GC).
  • scannerReportTTL 24h : reports are recreated daily against a fresh DB.
  • Standalone (default) : each Job downloads the DB. ClientServer : one shared server.
  • Its scanner image is pinned by tag (trivy:0.75.0) : mirror a verified digest.
$ kubectl get vulns -A -o wide
$ kubectl get clustervuln,clustersbom

4.Worked example : Kubernetes itself

  • sbom.k8s.io/v1.37.1/release : SPDX 2.3 tag-value, 812 packages (among them 24 images, 508 Debian packages) plus 100 files. Go modules are in /source.
  • Signed like the binaries (krel-staging, .sig + .cert) : verify as in Image Signing.
  • provenance.json : Statement v0.1, SLSA v0.2, unsigned, 404 for every 2026-09-23 patch. Only level claim : "Level 1" (v1.23).
  • Rescan 2026-10-07 : grype 1306 matches (155 CVEs, all Debian); Trivy 0 (os family none).
$ U=https://dl.k8s.io/release/v1.37.1
$ curl -sI $U/provenance.json    # 404
$ curl -Ls $U/kubernetes-release.spdx -o k.spdx
$ grype sbom:./k.spdx

5.Case study : the Trivy compromise (March 2026)

  • Stolen credentials published trivy v0.69.4 through the real release pipeline, and force-pushed 76 of 77 trivy-action tags and all 7 setup-trivy tags (CVE-2026-33634, in CISA KEV).
  • Pin actions by full SHA, transitively (trivy-action <= 0.30.0 called setup-trivy by tag). Pin the tool too : setup-trivy defaults to latest.
  • Verify the binary with a tight identity. The signer moved from private-trivy to trivy at v0.69.3.
late Febtoken stolen
03-01disclosed, rotation not atomic
03-19 ~17:43action tags force-pushed
03-19 18:22v0.69.4 released
03-22Docker Hub 0.69.5, 0.69.6
03-26in CISA KEV

Digest pins and full-SHA pins held, but only where every layer pinned.

In short : Rescan stored SBOMs with the same tool family, trust VEX and attached SBOMs only from a verified signer, and treat your scanner and its CI actions as supply chain too.

Supply chain security

Enforcing Image Policy

1.Where a check can run

A policy covers only the requests and fields it sees. Webhooks and VAP syntax : Admission Control.

  • CI checks manifests but only advises : anyone can kubectl apply around it.
  • Admission runs once per API write, before any pull.
  • The runtime checks every real pull, never cached images.

2.The gap between admit and pull

  • Admission verifies app:1.0 at digest A. The tag moves to B. A later pull gets B, never verified.
  • Fix : pin digests in Git, or let the verifier append one. All three keep the tag : podinfo:6.9.2@sha256:cab90e...
CI : kyverno apply, policy-tester advice only kubectl / GitOps mutating : tag -> tag@sha256 validating : VAP, verifiers API server, once per write etcd, then ReplicaSet controller Pod CREATE : admission again Pod-only rules bite here kubelet : image cached ? yes no no pull no node check runtime pull CRI-O policy.json / verifiers CRI-O re-checks at create if a policy path is set containerd : transfer service pulls only

3.Every field that holds an image

A Pod-only rule admits the Deployment, then denies its Pods as ReplicaSet FailedCreate, as in Pod Security. kubectl debug --copy-to makes a new Pod, so pods rules see it.

FieldReached byTrap
containers, initContainersPod CREATEExamples often check containers only
ephemeralContainerskubectl debug : UPDATEpods, * miss it : list pods/ephemeralcontainers
volumes[].image.referencePod CREATESkipped by every admission verifier by default
spec.template.specDeployment, StatefulSet, DaemonSet, JobUnchecked unless listed
spec.jobTemplate.spec.template.specCronJobOne more path for your CEL

4.What built-in CEL can do

variables:  # Pod form
- name: images
  expression: >-
    object.spec.?containers.orValue([]).map(c, c.image) +
    object.spec.?initContainers.orValue([]).map(c, c.image) +
    object.spec.?ephemeralContainers.orValue([]).map(c, c.image) +
    object.spec.?volumes.orValue([]).filter(v, has(v.image))
      .map(v, v.image.?reference.orValue(''))
validations:
- expression: variables.images.all(i, i.matches('@sha256:[a-f0-9]{64}$'))
  reason: Forbidden  # 403, not 422
  • Can : startsWith, matches, all / exists.
  • Cannot : parse a reference, normalize nginx, call a registry, resolve a tag.
  • End prefixes in / : blocks registry.example.com.evil.io.
  • Forbid :latest, even next to a digest.

5.Built-ins outside a webhook

  • ImagePolicyWebhook (off by default) : Pods and debug containers only. defaultAllow: false fails closed.
  • Static image VAP (manifest-based, beta v1.37) : no paramKind, so a hard-coded allow-list. Not deletable through the API. A broken file keeps the old one.
  • AlwaysPullImages : see Images, Registries and Pulls.

6.Commands

CommandShows
kubectl debug -it <pod> --image=<ref>@sha256:<d> --target=<c>A debug container : a separate request on the subresource
kubectl describe rs <rs>Late FailedCreate from a Pod-only rule
kubectl get validatingadmissionpolicy <name> -o jsonpath='{.status.typeChecking}'CEL type errors
kubectl get pod <p> -o jsonpath='{.spec.containers[*].image}'Whether digest pinning happened
In short : Place each check, cover every image field, and keep CEL to string hygiene (no signatures).

1.Kyverno ImageValidatingPolicy

policies.kyverno.io/v1, GA since 1.17 (v1 storage planned for 1.20). The Namespaced kind acts in its own namespace.

spec fieldWhat it does
matchConstraintsWebhook rules, copied verbatim : list pods/ephemeralcontainers yourself
matchImageReferencesA scope on the raw string : unmatched images are skipped, not denied
imagesExtra extractors (CEL). None built in for image volumes
attestors, attestationscosign (key, keyless, trustedRoot) or notary. Claims by intoto.type or referrer.type
validationsCEL : verifyImageSignatures, verifyAttestationSignatures (count > 0)
validationConfigurationsmutateDigest, verifyDigest, required : all default true

2.One Pod through Kyverno

  • Autogen covers controllers only when resources is exactly ["pods"]. Adding pods/ephemeralcontainers turns it off : use two policies.
API server ivpol.mutate mutateDigest ivpol.validate (validating webhook) extract built-ins + images match others skipped verifyDigest no digest : deny verify cache 60m required all kept images verified Deny / Warn / Audit Audit PolicyReport reports ctrl 1h, no cache API server ivpol.mutate mutateDigest ivpol.validate extract built-ins + images match others skipped verifyDigest no digest : deny verify cache 60m required all verified Deny / Warn / Audit Audit PolicyReport reports ctrl 1h, no cache

3.An IVPOL that pins a release workflow

matchImageReferences: [{glob: "ghcr.io/kyverno/*"}]
attestors:
- name: github
  cosign:
    keyless:
      identities:
      - issuer: https://token.actions.githubusercontent.com
        subjectRegExp: '^https://github\.com/kyverno/kyverno/\.github/workflows/.+@refs/tags/v.+$'
    source: {repository: ghcr.io/kyverno/signatures}
validations:
- expression: (images.containers + images.initContainers)
    .all(i, verifyImageSignatures(i, [attestors.github]) > 0)
  • Anchor subjectRegExp. Kyverno keeps its signatures in another repo.
  • cosign v3 bundles are auto-detected. cosign attestors check only in-toto.
  • trustedRoot sits next to keyless, not inside it (docs nest it).
  • IVPOL never reads the Pod's imagePullSecrets. spec.credentials (Kyverno namespace Secrets) replaces the helpers, no merge.

4.Kyverno version traps

  • 1.19.0 made all three real : upgrades can deny.
  • 1.19.1 fixes GHSA-5cjf-wwfg-pj4c : a one-image exception exempted the whole resource.
  • verifyImages (ClusterPolicy) : deprecated 1.19, removed in 1.20 (late 2026).
  • 1.19 is tested only on Kubernetes 1.33 to 1.35.

5.Sigstore policy-controller

  • Opt-in namespaces only. no-match-policy defaults to deny, so does a typo.
  • CIP : every matching one must pass, one authority in each is enough. ctlog = Rekor.
  • v0.15.1 cannot verify cosign v3 bundle image signatures : bundle is attestations only.
  • Chart 0.10.8 runs v0.13.1. No cache.

6.Commands

CommandShows
kubectl get ivpol <name> -o jsonpath='{.status.autogen}'Which controllers got variants
kubectl get ivpol <name> -o jsonpath='{.spec.validationConfigurations}'Stored defaults (all three true)
kubectl get policyreports -n <ns>Audit results
kubectl label namespace payments policy.sigstore.dev/include=trueOpts a namespace in
kubectl patch configmap/config-policy-controller -n cosign-system --type merge -p '{"data":{"no-match-policy":"warn"}}'Warn on unmatched images
In short : IVPOL verifies and pins matches. policy-controller cannot verify cosign v3 bundles.

1.Ratify behind Gatekeeper

  • External data : a Gatekeeper Provider, called from Rego with external_data. Responses cached 3m.
  • Ratify v1.4.6 is the stable line (config.ratify.deislabs.io/v1beta1).
  • v2.0.0-beta.2 (Executor) says "do not use in production" and renamed its API between betas.
  • The Gatekeeper chart fails open (Ignore, 3s). The sample constraint covers only Pods in default.
API server Gatekeeper mutating Assign : tag -> digest Gatekeeper validating RatifyVerification (Rego) external_data Ratify store -> notation, cosign -> threshold allow / deny timeout admitted Ignore (chart)

2.Checks on the node

  • CRI-O : policy.json per Pod namespace (/etc/crio/policies/<ns>.json) with sigstoreSigned.
  • containerd 2.x : bindir verifiers (exit 0 allows), only on transfer-service pulls, the CRI default since 2.1.
  • Legacy registry.mirrors / configs / auths or discard_unpacked_layers = true silently fall back to local pulls with no verifiers (kind's default did). Check : journalctl -u containerd | grep -i 'Falling back to local image pull'
  • ctr image pull --local skips verifiers too : test with crictl pull.
CRI-Ocontainerd
Keyless identityExact email SAN only : no GitHub workflow URIsWhatever your binary does
cosign v3 bundlesNo (container-libs issue 388)Your binary
Cached imagesRe-checked at create when a policy path is setNever checked
Pod showsSignatureValidationFailedErrImagePull, then ImagePullBackOff
Warn modeNone : test on a canary poolNone

3.Failure and caching

  • Fail : a registry or Rekor outage blocks admission.
  • Kyverno with Ignore still denied on registry errors (observed once). [Audit] still mutates.
  • Caches : Kyverno 60m, Gatekeeper 3m, policy-ctrl none.
  • Private Sigstore : Kyverno trustedRoot / tuf, policy-controller TrustRoot, Ratify verifier parameters.

4.Digest mutation and GitOps drift

  • Kyverno (autogen), policy-controller and Ratify rewrite templates to tag@sha256 : Argo CD and Flux see drift.
  • Pin digests in Git, so there is nothing to rewrite : the digest strategy of Argo CD Image Updater, or a Flux ImagePolicy with digestReflectionPolicy (Flux).
  • Or hide the diff : ignoreDifferences in Argo CD.

5.Check in CI too, as advice

  • kyverno apply ivpol.yaml --resource rendered.yaml runs only the validating path : tag-only images fail verifyDigest. A policy file without .yaml is skipped with exit 0.
  • policy-tester -policy cip.yaml -image REF checks one image against one CIP.

6.Which engine does what

VAPKyverno 1.19policy-ctrl 0.15.1Ratify + GKCRI-Ocontainerd
Signaturesnocosign, notarycosign .sig onlynotation, cosignsigstoreyour binary
Tag to digestnomutateDigestalwaysAssignn/an/a
kubectl debugif listedif listeddigest onlysent by defaultat pullat pull
Image volumesif CEL readsextractornonoyesyes
Pod templatesyour rulesautogenyesmutate onlyn/an/a
Unmatchedyour CELskippeddenyby scopesdefaultallowed
Trial mode[Warn, Audit][Audit] or [Warn]mode, no-match : warndryrun, then warnnonenone
On failureFailFailFailIgnorepull failspull fails
In short : Pin digests, cover every field, trial in warn mode, and back admission with node checks.

Scheduling

Advanced Scheduling

1.What does kube-scheduler do ?

It gives a node to each Pod with no spec.nodeName.

  • Filter : drop each node that breaks a hard rule.
  • Score : rank the rest on soft rules. The highest total wins, ties at random.
  • Bind : write spec.nodeName through the API server.
  • 0 nodes fit : try preemption, else Pending. The events name the failed rule.
Requests, not usage
  • A Pod fits if its request fits the node's allocatable minus what is already requested.
  • Effective request : the larger of the summed app + sidecar containers and the biggest init container, plus RuntimeClass overhead.
  • Usage and limits are ignored. No requests = fits anywhere. QoS : Quick Revision.

2.One Pod through the scheduler

The scheduling framework (stable v1.19) chains plugin hooks.

  • Scheduling cycle : serial, one Pod at a time, from PreFilter to Permit.
  • PostFilter runs only when 0 nodes fit. Preemption lives there.
  • Binding cycle : PreBind, Bind, PostBind. May overlap the next Pod.
  • A failure at Reserve or later runs Unreserve and sends the Pod back to a queue.
  • backoffQ : 1 s first, 10 s max. QueueingHints (GA v1.34) requeue only on an event that might help, like a new node.
New Podspec.nodeName empty PreEnqueuescheduling gates activeQhighest priority first PreFilter + Filterdrop infeasible nodes PreScore + Scoresoft rules, weighted Reserve + Permitapprove or deny PostFilterpreemption 0 fit backoffQor unschedulable pool denied hint matched or backoff over PreBind + Bindbinding cycle, async ok BoundScheduled event nodeName written

3.Hard rules filter, soft rules score

ControlHard (Filter)Soft (Score)Plugin, weight
nodeSelectorevery label must match-NodeAffinity, 2
Node affinityrequired...preferred..., weight 1 to 100NodeAffinity, 2
Pod (anti-)affinityrequired..., with a topologyKeypreferred..., weight 1 to 100InterPodAffinity, 2
TaintsNoSchedule, NoExecutePreferNoScheduleTaintToleration, 3
Topology spreadDoNotSchedule (default)ScheduleAnywayPodTopologySpread, 2
Requestsmust fit free allocatableLeastAllocated (default)NodeResourcesFit, 1
Priorityno filter : queue order, preemption-PrioritySort, DefaultPreemption
spec.nodeNameskips the scheduler-the kubelet admits or rejects it

Also scoring : BalancedAllocation 1, ImageLocality 1. A soft rule only adds to a weighted sum, so it can lose.

4.Node affinity

  • nodeSelector : the node must carry every listed label.
  • requiredDuringScheduling... is hard. preferred... is soft, it adds its weight.
  • Terms are ORed, matchExpressions in a term ANDed. With nodeSelector too, both must hold.
  • Operators : In, NotIn, Exists, DoesNotExist, Gt, Lt (integer labels).
  • IgnoredDuringExecution : relabeling a node never moves a running Pod.

5.Pod affinity and anti-affinity

  • Matches labels of Pods already running in a domain. topologyKey names it and is required.
  • kubernetes.io/hostname = per node, topology.kubernetes.io/zone = per zone.
  • Own namespace only, unless namespaces or namespaceSelector widen it.
  • Costly. Not recommended above several hundred nodes.
  • Trap : required anti-affinity on hostname with more replicas than nodes leaves the extras Pending.
In short : kube-scheduler filters nodes on hard rules, scores the rest on soft rules, and binds the winner. It counts requests, not usage.

1.Taints and tolerations

  • A taint on a node repels. A toleration permits, it never attracts.
  • NoSchedule blocks new Pods, PreferNoSchedule discourages, NoExecute also evicts.
  • tolerationSeconds : time allowed under NoExecute. Unset = forever.
  • operator : Equal (default) or Exists. Exists with no key matches all.
  • Dedicated nodes : taint and label them. Pods get the toleration and required affinity on the label.
Built-in taintEffectSet when
not-readyNoExecuteReady is False
unreachableNoExecuteReady is Unknown
memory-pressureNoScheduleMemoryPressure
disk-, pid-pressureNoScheduleDiskPressure, PIDPressure
network-unavailableNoSchedulenetwork not set up
unschedulableNoSchedulekubectl cordon, drain
out-of-serviceNoExecuteby hand, dead node
control-planeNoSchedulekubeadm control plane

Keys start with node.kubernetes.io/, control-plane with node-role.kubernetes.io/. Pods tolerate not-ready and unreachable for 300 s, so they leave a dead node after about 5 min.

2.Topology spread

  • maxSkew (> 0) : allowed gap to the least-loaded domain. topologyKey : the node label that makes domains.
  • whenUnsatisfiable : DoNotSchedule (default) or ScheduleAnyway.
  • labelSelector : which Pods count. Match the Pod's own labels.
  • None set ? Soft defaults : hostname skew 3, zone skew 5. Only checked at scheduling time.
new Pod, app=web maxSkew 1 DoNotSchedule zone a zone b zone c 3 - 1 = 2 > 1 blocked 3 - 1 = 2 > 1 blocked 2 - 2 = 0 placed here skew = Pods in the domain - fewest in any domain

3.Priority and preemption

  • PriorityClass (cluster-scoped) : yours go up to 1000000000. One may be globalDefault, else 0.
  • Admission writes spec.priority. Higher = earlier in activeQ. preemptionPolicy Never : jumps the queue, never evicts.
  • Preemption : delete lower-priority Pods on one node to make room, set status.nominatedNodeName (a hint).
  • Best effort : PDBs may break, QoS ignored, preemption never frees DRA devices.

4.Skip, hold or swap it

  • spec.nodeName : no filters, NoSchedule ignored. Pin with nodeSelector on kubernetes.io/hostname instead.
  • schedulingGates (GA v1.30) : the Pod waits at SchedulingGated until every gate is gone.
  • Profiles : several schedulerNames in one kube-scheduler. Pods pick with spec.schedulerName.
  • Descheduler : evicts Pods that now break a rule, kube-scheduler places them again.
  • DRA (GA v1.34) : a ResourceClaim asks for a GPU. Device and node are picked together.

5.Why is it Pending ?

FailedScheduling : 0/N nodes are available: <reason>, ...

Reason saysFix
Insufficient cpu / memorylower requests, add nodes
untolerated taint(s)unnamed since v1.35 : describe node
node affinity/selectorlabel nodes or fix the rule
pod anti-affinity rulespreferred, or spread instead
topology spread constraintsraise maxSkew, add domains
were unschedulablekubectl uncordon
unbound immediate PVCsfix the PVC (Storage)
SchedulingGated (status, no event)the gate owner removes it

6.Commands

CommandShows
kubectl events --for pod/<pod> --types=WarningThat Pod's warnings
kubectl get events -A --field-selector reason=FailedSchedulingAll scheduling failures
kubectl describe node <node>Taints, Allocated resources
kubectl get nodes -L topology.kubernetes.io/zoneZone as a column
kubectl taint nodes <node> k=v:NoScheduleAdd a taint, a trailing - removes it
kubectl get priorityclassesVALUE, GLOBAL-DEFAULT
In short : Taints repel, affinity attracts, spread balances, and priority decides who waits. A Pending Pod's FailedScheduling event names the rule to fix.

Scheduling

Dynamic Resource Allocation

1.Why DRA ?

  • Device plugin : only says how many, limits: nvidia.com/gpu: 2. No overcommit, no sharing between containers, no pick by attribute, config per node.
  • DRA : says which. The scheduler picks specific devices from the driver's ResourceSlices by CEL on attributes and capacity. The kubelet has the driver prepare them through CDI.
  • Status : GA since v1.34, gate locked on since v1.35. v1.37 serves only resource.k8s.io/v1 by default.
AspectDevice pluginDRA
Asks withinteger in limitsa ResourceClaim
Picks devicekubeletscheduler, by CEL
Inventorynode allocatableResourceSlices
Sharingnonecontainers, Pods
Configper nodeper claim
Node setupAllocate RPCPrepare + CDI
Partitionsstaticpartitionable (beta)
Healthdrops from countdevice taints (GA)

Both can run. One resource name on one node comes from the plugin or DRA, not both.

2.The object model

Like PVC and StorageClass (Storage) : the claim asks, the class categorizes, the slice is the inventory.

  • DeviceClass (cluster) : CEL selectors, optional config. Made by an admin or a driver chart.
  • ResourceSlice (cluster) : devices per pool. The driver owns it and overwrites manual edits.
  • ResourceClaim : the request. Status has allocation, reservedFor.
  • ResourceClaimTemplate : one claim per Pod, deleted with the Pod (Owners and GC).
  • DeviceTaintRule (cluster) : taints devices by driver, pool, device. In v1 since v1.37.
DRA driver Cluster admin publishes creates ResourceSlice cluster, per pool DeviceClass cluster, CEL selectors matches ResourceClaimTemplate namespaced ResourceClaim namespaced deviceClassName per Pod Pod spec.resourceClaims[] each entry : exactly one of the two resourceClaimTemplateName resourceClaimName Container resources.claims[] name (+ request)

3.One GPU, two containers

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata: { name: big-gpu }
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.example.com
          selectors:
          - cel:
              expression: device.capacity["gpu.example.com"]
                .memory.compareTo(quantity("40Gi")) >= 0
---
apiVersion: v1
kind: Pod
metadata: { name: trainer }
spec:
  resourceClaims:
  - name: gpu
    resourceClaimTemplateName: big-gpu
  containers:
  - name: main
    image: ubuntu:24.04
    command: [sleep, infinity]
    resources: { claims: [ { name: gpu } ] }
  - name: sidecar       # same GPU
    image: ubuntu:24.04
    command: [sleep, infinity]
    resources: { claims: [ { name: gpu } ] }

4.Wiring rules

  • Each spec.resourceClaims[] entry has a name and exactly one of resourceClaimName, resourceClaimTemplateName.
  • A named claim must already exist in the Pod's namespace, or the Pod stays Pending.
  • The Pod list only makes it available. A container gets the device only if its resources.claims names it.

5.Inside a request

  • devices.requests[] : a name plus exactly, or firstAvailable[] (ordered fallback subrequests).
  • exactly : deviceClassName, selectors, allocationMode ExactCount (default, count 1) or All, adminAccess, tolerations, capacity.
  • constraints[] : matchAttribute, distinctAttribute. config[].opaque : driver + parameters.
What CEL sees

device.driver, device.attributes["dom"].x, and device.capacity["dom"].x, a Quantity (use compareTo).

A missing key is an error that aborts allocation, not a skip. Guard with has(), or optional chaining with orValue().

In short : A Pod asks through ResourceClaims that name a DeviceClass. The scheduler picks real devices from ResourceSlices with CEL, and the driver prepares them on the node.

1.From claim to device

  1. The kubelet plugin publishes ResourceSlices.
  2. resourceclaim-controller creates claims from templates.
  3. DynamicResources runs CEL, picks devices first-fit.
  4. It writes allocation + reservedFor, then binds the Pod.
  5. The kubelet calls NodePrepareResources (gRPC).
  6. The driver returns CDI device IDs.
  7. The runtime injects them into the containers.
  8. NodeUnprepareResources after the Pod stops.
  9. Last Pod gone : deallocated, template claim deleted.

The rest of the cycle : Advanced Scheduling.

resourceclaim-controller in kube-controller-manager kube-scheduler DynamicResources plugin 2. claim from template 9. clean up 3. read, CEL, pick 4. allocate + bind kube-apiserver 1. ResourceSlices Pod bound DRA driver kubelet plugin, per node kubelet 5. Prepare 6. CDI IDs 8. Unprepare 7. start with CDI container runtime devices injected The plugin registers with the kubelet in /var/lib/kubelet/plugins_registry

2.Sharing a device

  • Template : each Pod gets its own claim and devices.
  • Shared claim : many Pods name one ResourceClaim, 256 at most (reservedFor).
  • Containers : one Pod claim listed in several containers.
  • How they share at runtime (time-slicing, MPS) is up to the driver and its opaque config.
Template own device each Pod A Pod B gpu-tpl claim A claim B gpu-0 gpu-1 Shared claim many Pods Pod C Pod D shared-gpu reservedFor C, D max 256 gpu-2 One Pod two containers ctr0 ctr1 Pod E claim gpu gpu-3

3.Features in v1.37

FeatureKey fieldsFeature gateMaturity
Core DRAv1 kinds, pod spec.resourceClaimsDynamicResourceAllocationGA 1.34, locked 1.35
Prioritized listrequests[].firstAvailable[]DRAPrioritizedListGA 1.36, locked 1.37
Admin accessexactly.adminAccess + namespace labelDRAAdminAccessGA 1.36
Device taintsdevices[].taints[], exactly.tolerations[]DRADeviceTaintsGA 1.37
DeviceTaintRuledeviceSelector {driver, pool, device}DRADeviceTaintRulesGA 1.37
Extended resourceextendedResourceName : limits served by DRADRAExtendedResourceGA 1.37
Device statusclaim status.devices[]DRAResourceClaimDeviceStatusGA 1.37
Partitionable devicessharedCounters[], consumesCounters[]DRAPartitionableDevicesBeta 1.36, on
Consumable capacityallowMultipleAllocations, capacity.requestsDRAConsumableCapacityBeta 1.36, on
Binding conditionsbindingConditions, bindsToNodeDRADeviceBindingConditionsBeta 1.36, on
Resource healthallocatedResourcesStatusResourceHealthStatusBeta 1.36, on
PodGroup claimsPodGroup spec.resourceClaimsDRAWorkloadResourceClaimsBeta 1.37, off

Alpha, off : list-type and derived attributes, node allocatable resources, optional node operations, compatibility groups, pool status.

4.Gotchas

  • Preemption never frees a DRA device. A higher-priority Pod stays Pending.
  • spec.nodeName skips the scheduler, so nothing allocates the claim. Pin with nodeSelector on kubernetes.io/hostname.
  • firstAvailable is decided per Pod. Replicas can get different subrequests.
  • Only v1 is on by default. A v1beta1 or v1beta2-only driver needs kube-apiserver --runtime-config.
  • The admin label is resource.kubernetes.io/admin-access="true", not resource.k8s.io.

5.Commands

CommandShows
kubectl get deviceclasses"No resources found" = on
kubectl get resourceslices --field-selector spec.nodeName=<node>One node's devices
kubectl get resourceclaims -Apending, allocated, reserved
kubectl describe resourceclaim <name> -n <ns>First stop, Pending Pod
kubectl get pod <pod> -o jsonpath='{.status.resourceClaimStatuses}'Generated claim names

Try it : dra-example-driver (mock GPUs, gpu.example.com). NVIDIA driver : gpu.nvidia.com, mig.nvidia.com.

In short : The scheduler allocates devices when it places the Pod, the kubelet has the driver prepare them through CDI, and the allocation is freed once no Pod reserves the claim.

Resources and autoscaling

Requests, Limits and QoS

1.Reservation and ceiling

Basics are in Quick Revision. Here : under the hood.

  • Scheduler : per resource, requests on the node plus the new Pod must fit node.status.allocatable. Usage is ignored. A miss : FailedScheduling, insufficient cpu (plugins).
  • For scoring only, a container with no request counts as 100m CPU and 200Mi, so zero-request Pods spread out. Fit still uses zero.
  • HPA utilization is a percentage of the request.
Pod spec : resources requests limits kube-scheduler fits allocatable ? cpu.weight share when contended kubelet + runtime cpu.max throttled memory.max OOM-killed, exit 137

2.Units and traps

CPUMemory
unitcores, m = millibytes, Mi, Gi, M
requestscheduling, cpu.weight sharescheduling, eviction ranking, Memory QoS hints
limitcpu.max quota per 100 msmemory.max hard cap
over limitthrottled, never killedOOM-killed, exit 137, restarted
kindcompressiblenot compressible
idleusable above request if no limitusable, but first to be evicted
Unit traps
  • memory: 400m is 0.4 bytes.
  • CPU is absolute, not a share of the node.
  • Limit only : the request copies it, even over a LimitRange defaultRequest.
  • Smallest useful CPU limit : 10m (1 ms quota).

3.What lands in cgroup v2

Fieldcgroup v2 fileExample
requests.cpucpu.weight, from shares = mCPU x 1024 / 1000 (2 to 262144)1 CPU = 100 (runc 1.3.2+, crun 1.23+), 39 before
limits.cpucpu.max = "<quota> 100000"500m : 50000 100000; none : max 100000
limits.memorymemory.maxmax when unset
requests.memorynothing hardmemory.min, .low, .high only with Memory QoS
  • The kubelet passes values through the CRI; the OCI runtime writes the files. The kubelet still writes Pod and QoS-tier cgroups with the old formula, so their cpu.weight uses another scale.
  • An OOM kills every process in the container together (memory.oom.group). cgroup v1 : the kubelet refuses it by default since 1.35 (failCgroupV1: true), removal not before 1.38.

4.Throttling below the average

  • Throttling shows while average usage looks far below the limit. Check nr_throttled in cpu.stat, not averages.
  • Runtimes read the limit : Go 1.25 (when go.mod says 1.25+) sets GOMAXPROCS from it (rounded up, minimum 2, never above the CPU count) and re-checks it after a resize.
limit 100m = 10 ms per 100 ms throttled throttled throttled run 10 ms : 30 ms of work needs 3 periods, about 210 ms 4 threads, limit 400m all stall all stall all stall 4 x 10 ms burns the 40 ms quota, all stall about 90 ms

5.Node allocatable

Allocatable = Capacity - kubeReserved - systemReserved - hard eviction threshold. By default both reserves are empty, so only memory.available<100Mi is held back. Docs example : 16 CPU, 32Gi minus 1 CPU / 2Gi kube, 500m / 1Gi system and <500Mi eviction gives 14.5 CPU, 28.5Gi. systemReserved should cover kernel memory, which is not charged to Pods.

In short : A request is a reservation counted against allocatable; a limit is a ceiling the kernel enforces with cpu.max and memory.max. Over it, CPU is throttled and memory is OOM-killed.

1.QoS beyond the rules

Rules in Quick Revision. The class is derived and fixed; an in-place resize that would change it is rejected.

Classoom_score_adjEviction position
Guaranteed-997last, by priority
Burstablemin(max(3, 1000 - 1000 x memRequest / nodeMemory), 999), so 3 to 999first if over its request, else late
BestEffort1000first : usage is always over a zero request
  • system-node-critical Pods also get -997. A sidecar's score is capped at that of the app container with the smallest memory request, so it is not killed first.

2.Two ways to die of memory

memory use rises hits its limit node runs short kernel OOM killer inside that cgroup OOMKilled, exit 137 restarted in the same Pod kubelet signal under threshold, every 10 s rank Pods, evict Pod Failed controller makes a new one PDBs ignored, hard : 0 s grace spike

3.Who is evicted first

  1. Usage above request ? (not for PID pressure)
  2. Then lower priority.
  3. Then usage relative to request.
  • memory.available = capacity minus working set. Page cache in active_file can fake pressure; the documented fix is memory request = limit.
  • Disk pressure : dead containers and unused images go first.
  • MemoryPressure, DiskPressure, PIDPressure become taints (scheduling).
  • HugepageAwareEviction (beta, on in 1.37) subtracts huge pages from memory.available.

4.Default hard thresholds (Linux)

memory.availablenodefs.availableimagefs.availablenodefs.inodesFreeimagefs.inodesFree
<100Mi<10%<15%<5%<5%
Setting one evictionHard value zeroes the others, unless mergeDefaultEvictionSettings: true. List every one you want. Soft thresholds need a grace period; evictionPressureTransitionPeriod is 5m.

5.Commands, and Memory QoS in v1.37

CommandWhat it shows
kubectl describe node <node>capacity, allocatable, allocated requests
kubectl get pod <p> -o jsonpath='{.status.qosClass}'the QoS class
kubectl describe pod <p>OOMKilled, exit 137, or FailedScheduling
kubectl top pod <p> --containerslive usage (needs metrics-server)
stat -fc %T /sys/fs/cgroup/prints cgroup2fs on cgroup v2
cat cpu.weight cpu.max memory.max cpu.statapplied values, in the container's cgroup
Memory QoS
  • Beta and on, but inert until configured (memoryThrottlingFactor defaults to null). With a factor, memory.high = request + factor x (limit - request) : 0.9, 256Mi, 1Gi gives about 947Mi. Never for Guaranteed or a container with request = limit.
  • memoryReservationPolicy: TieredReservation sets memory.min (Guaranteed) or memory.low (Burstable) to the request; a cache-heavy Guaranteed Pod may then OOM. Kernel 5.9+.
Seen in practice

A cache or JVM service crash-loops with OOMKilled. Pitfall : its own memory cap (-m, -Xmx, GOMEMLIMIT) equals the container limit, but buffers and off-heap memory live above that cap. Do better : keep the app cap clearly below the limit; raise the limit only once you know why memory grew.

In short : QoS sets the OOM score; the kubelet evicts by usage over request, then priority. A fast spike meets the OOM killer first, eviction ignores PDBs, one custom threshold zeroes the rest.

1.Guardrails at admission

Pod create LimitRanger : apply defaults LimitRanger : min, max, ratio ResourceQuota : sum vs hard stored, then scheduler 403 Forbidden
  • LimitRange defaulting : a missing default takes max; a missing defaultRequest takes default, then min.
  • Request only : gets the default limit. Limit only : request = its own limit.
  • default.cpu: 500m and a 700m request with no limit : rejected (must be less than or equal to cpu limit).
  • Two LimitRanges : the winning default is not deterministic. maxLimitRequestRatio >= 1. Admission only : running Pods never change (plugin order).

2.ResourceQuota

KeyWhat it sums
requests.cpu / .memorysum of requests
limits.cpu / .memorysum of limits
requests.ephemeral-storage, hugepages-<size>scratch, huge pages
requests.<extended>extended, requests. only
<class>.deviceclass.resource.k8s.io/devicesDRA device count
pods, count/<resource>.<group>, requests.storageobjects, PVC storage
  • Scopes : BestEffort, NotBestEffort, Terminating, NotTerminating, PriorityClass, CrossNamespacePodAffinity, VolumeAttributesClass.
  • Only non-terminal Pods count.
  • Quota on cpu or memory : Pods must set it (LimitRange).
  • Over quota : Deployment, no Pods.
  • Write access can delete it : RBAC.

3.Effective request : init, sidecars, Pod level

  • sidecar proxyindex 0 : 100m
  • init migrateindex 1 : 1 CPU + sidecar = 1.1
  • app500m; running sum 600m
  • effective1.1 CPU, for the Pod's whole life

Init and sidecars : Pods. The exact KEP-753 rule (short form : Advanced Scheduling). RuntimeClass overhead is added after.

Pod-level spec.resources (beta, on since 1.34) : only cpu, memory, hugepages-<size>. Pod values win for scheduling, the Pod cgroup and QoS. Container limits must be <= Pod limits; the rest share the budget. Not used by the CPU, memory or topology managers in 1.37, nor on Windows. An empty Pod-level block no longer changes QoS.

4.The CPU-limits question

PolicyGood forCost
request onlylatency-sensitive servicesneeds accurate requests; apps lean on spare CPU
request = limitbenchmarks, predictable capacityidle CPU wasted; throttled if sized low
request + headrooma middle groundstill throttles at the cap
whole-core limitgreedy batch Jobsthrottling is intended

Mainstream answer : CPU requests everywhere, sized from data (VPA); CPU limits only for a hard cap (multi-tenant, greedy batch, benchmarks); memory request = limit where it matters. The Kubernetes blog carries a dissent, for predictability.

Seen in practice

A gateway proxy's TLS throughput sat far below line rate, throttled at low average CPU. Pitfall : a small CPU limit on a data-path Pod. Do better : no CPU limit on proxies, a request from measured peak, a throttle-ratio alert.

5.Commands

CommandWhat it shows
kubectl describe quota -n <ns>used vs hard, per key
kubectl describe limitrange -n <ns>defaults, min, max, ratio
sum by (namespace,pod) (rate(container_cpu_cfs_throttled_periods_total[1h])) / sum by (namespace,pod) (rate(container_cpu_cfs_periods_total[1h]))throttle ratio by Pod (Prometheus). Aggregate by Pod, not container name
In short : LimitRange fills in and checks each container, ResourceQuota caps the namespace sum, both at admission. Init and sidecars count in the effective request; CPU limits only for hard caps.

Resources and autoscaling

In-Place Pod Resize

1.What is in-place resize ?

  • Change the CPU and memory requests and limits of a running Pod's containers without a new Pod : same node, same IP, warm caches.
  • KEP-1287, gate InPlacePodVerticalScaling : alpha 1.27, beta (on) 1.33, GA 1.35, locked on, due for removal in 1.38.
  • Do not set the gate : =true only logs a warning, =false stops the component at startup.
Two ways to change resources
  • Template change in a Deployment = new Pods (a rollout).
  • A /resize patch on the live Pod = the same Pod, new numbers.
  • The QoS class never changes : a resize that would change it is rejected.

2.Desired, allocated, actual

ViewFieldWritten by
desiredspec.containers[*].resourcesyou, through /resize
allocatedstatus.containerStatuses[*].allocatedResources (requests only)the kubelet, when it admits the resize
actualstatus.containerStatuses[*].resourcesthe kubelet, from what the runtime applied
progressstatus.conditions : PodResizePending, PodResizeInProgressthe kubelet

status.resize is deprecated and no longer set since 1.33. Scripts must read the two conditions instead. Pod spec and status basics are in Pods.

3.The resize subresource

  • Only --subresource resize on patch, edit or apply (kubectl 1.32+) changes resources. A plain update cannot.
  • RBAC resource : pods/resize (RBAC). Grant patch on it to whoever resizes.
  • It can change regular containers, sidecars, running non-sidecar init containers (GA 1.37), Pod-level spec.resources (beta 1.36), and with an alpha gate a memory-backed emptyDir sizeLimit.
$ kubectl patch pod resize-demo \
  --subresource resize --patch \
  '{"spec":{"containers":[{"name":"app","resources":
    {"requests":{"cpu":"800m"},"limits":{"cpu":"800m"}}}]}}'

4.One resize, end to end

patch --subresource resize API server validation + admission 403 Forbidden spec = desired kubelet : does it fit ? others' allocated + new <= allocatable not now : Deferred, retried never here : Infeasible yes allocated + InProgress event ResizeStarted CRI update : cgroups failed : reason Error status = actual, Completed

Admission : PodResizeValidator, LimitRanger, ResourceQuota. One resize is atomic; patches sent before the kubelet acts merge.

5.Conditions and events

  • PodResizePending, reason Deferred : fits the node's allocatable but not its free room. Retried.
  • PodResizePending, reason Infeasible : this node can never do it. Not retried.
  • PodResizeInProgress : allocated, being applied. Reason Error when applying failed.
  • Both at once : a new resize arrived while the last one is still applying.
  • Events : ResizeStarted, ResizeCompleted, ResizeDeferred, ResizeInfeasible, and ResizeError as a Warning.
conditions:  # a resize that waits
- type: PodResizePending
  status: "True"
  reason: Deferred
  message: "... cpu, requested: 6000,
    used: 3950, capacity: 8000"
In short : Desired lives in spec, actual in status. You change desired only through /resize; the kubelet catches up in the same Pod and reports progress in PodResizePending and PodResizeInProgress.

1.resizePolicy

  • Per container and resource : NotRequired (resize live if possible) or RestartContainer (restartCount goes up). Only these two; never PreferNoRestart.
  • No entry = NotRequired. An entry with an empty restartPolicy fails validation.
  • Both resources change and either is RestartContainer : the container restarts.
  • Pods with restartPolicy: Never must use NotRequired.
  • Fixed after creation (recreate the Pod). Since 1.36 non-sidecar init containers may not use RestartContainer.
containers:  # Pod resize-demo
- name: app
  image: registry.k8s.io/pause:3.8
  resizePolicy:
  - resourceName: cpu
    restartPolicy: NotRequired
  - resourceName: memory
    restartPolicy: RestartContainer
  resources:  # Guaranteed : change
    # request and limit together
    requests: {cpu: 500m, memory: 128Mi}
    limits:   {cpu: 500m, memory: 128Mi}

2.How the kubelet applies it

Node-side admission : other Pods' allocated requests + this Pod's new requests <= allocatable. Then the cgroups are written (cpu.weight, cpu.max, memory.max; see Requests, Limits and QoS).

Growing : pod cgroup first, then containers. Shrinking : containers first, pod cgroup last. Decreases before increases. Memory before CPU. On any error the kubelet stops and retries on a later sync.

node pod cgroup container A container B growing 1. pod cgroup 2. containers shrinking 1. containers 2. pod cgroup memory first, then CPU; decreases before increases

3.Shrinking memory

  • Allowed with NotRequired since 1.34.
  • Usage at or above the new limit : the kubelet refuses, sets PodResizeInProgress reason Error with a Warning ResizeError, and retries later.
  • A race remains : usage can rise between check and write. On cgroup v2 the kernel then reclaims, or OOM-kills.
  • Common mistake : expecting a shrink below usage to "stay pending". It is an Error, not Pending.

4.Waiting and accounting

  • Retry order : no request increase, then higher PriorityClass, then Guaranteed before Burstable, then the one waiting longest.
  • Scheduler and quota count max(desired, allocated, actual) (not desired if Infeasible).
  • Race : a new Pod placed from cached data can be rejected OutOfCPU (open issue).
  • 1.37 alpha (...SchedulerPreemption on API server, scheduler, kubelet) : preempt lower-priority Pods for a Deferred resize, same node only.

5.What happens when

SituationOutcome
fits the node nowPodResizeInProgress, then done
fits allocatable, but other Pods use the roomDeferred, retried
above the node's allocatable403 at admission since 1.36 (PodResizeValidator; it ignores other Pods); was Infeasible
static or Windows Pod, QoS change, removing a set value, not CPU or memory, resizePolicy changerejected by API validation (403/422 at the API server); QoS case : "Pod QOS Class may not change as a result of resizing"
memory change with swap and NotRequiredInfeasible
Guaranteed Pod on a static CPU (Memory) manager node, CPU (memory) changeInfeasible unless an alpha gate; the other resource resizes
memory limit at or below usagePodResizeInProgress Error, retried

Burstable Pods on static manager nodes are not blocked. Ephemeral containers are never resizable.

In short : resizePolicy sets restart per resource, fixed at creation. The kubelet admits against allocatable, writes cgroups in a safe order and refuses memory shrinks below usage.

1.Who resizes in place ?

kubectl --subresource VPA updater (in place) operator calling it pods/resize same Pod same IP not in place : edit in Git template new Pods
  • A resources change in a Deployment, StatefulSet or DaemonSet template still rolls new Pods. Job template resources cannot change.
  • So a template's resizePolicy matters only when something patches the live Pod. Workload-level in-place resize is only a design sketch.
  • A manual patch is out of band from Git : when the Pod is replaced, the template value returns.
  • Operators resize in place only if they call pods/resize. CloudNativePG recreates its Pods instead.

2.Pod-level resize

  • Beta and on since 1.36, still beta in 1.37. Patch Pod-level spec.resources (the API) through /resize.
  • No Pod-level resizePolicy. A container without its own limit inherits the Pod limit, so it resizes too, and restarts if its own policy says RestartContainer.
  • Rules : Pod requests >= sum of container requests; each container limit <= Pod limit.
  • Needs cgroup v2, Linux, containerd 2.0+ or CRI-O.

3.VPA uses it

  • updateMode: InPlaceOrRecreate (GA in VPA 1.6.0) resizes all containers through pods/resize and falls back to eviction. InPlace (alpha, VPA 1.7.0) never evicts. Auto is deprecated.
  • The updater needs patch on pods/resize. VPA does not support Pods with Pod-level resources. Fallback rules : VPA.

4.Runtimes

RuntimeCPU live ?Memory live ?
Go, Rust, Cyesyes, more room is usable at once
JVM (HotSpot)mostly : availableProcessors() re-reads cgroups, but GC and pool threads are sized at startno : max heap is fixed at start
Node.jsyesheap capped by --max-old-space-size, set at start
Pythonyesthe GA blog says it needs a restart

Rule of thumb : CPU NotRequired; memory RestartContainer for runtimes with a fixed heap.

Seen in practice
  • Pitfall : memory NotRequired in the Git-managed template of a fixed-heap JVM service, "so future bumps land live". A Git edit still rolls the Pod, and a live memory raise adds only off-heap room, no heap.
  • Do better : treat resizePolicy as the contract for /resize callers (VPA, kubectl), use memory RestartContainer for fixed heaps, and write manual resizes back to Git.

5.Commands

CommandWhat it does
kubectl get pod POD -o jsonpath='{.status.containerStatuses[0].allocatedResources.cpu}'The CPU request the kubelet allocated
kubectl get pod POD -o jsonpath='{.status.conditions[?(@.type=="PodResizePending")]}'Why a resize waits (reason and message)
kubectl get events --field-selector involvedObject.name=PODThe five Resize* events
kubectl exec POD -- cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/memory.maxApplied cgroup v2 values (max 100000 = no CPU limit)
kubectl auth can-i patch pods/resizeRBAC check for the subresource

Kubelet metrics (alpha) for Prometheus : kubelet_pod_pending_resizes, kubelet_pod_infeasible_resizes_total{reason_detail}, kubelet_pod_resize_duration_milliseconds.

In short : Templates still roll. Only a /resize caller (kubectl, VPA, an operator) resizes in place, until the next rollout. CPU resizes live almost everywhere; fixed heaps need a restart.

Resources and autoscaling

Vertical Pod Autoscaler

1.What is VPA ?

A feedback loop that moves a Pod's requests (and limits in proportion) toward its real usage. It never changes replicas. An add-on from kubernetes/autoscaler, not core : two CRDs (VerticalPodAutoscaler, ...Checkpoint) and three controllers you install. It reads usage from metrics-server. Beside it, HPA adds Pods and Karpenter adds nodes.

VPA changes Pods, not your DeploymentThe template keeps the old requests, so look at the Pod. Remove VPA and new Pods get the template values back.

VPA 1.8.0 runs on Kubernetes 1.36 to 1.38. API autoscaling.k8s.io/v1 only (v1beta2 not served since 1.3.0).

2.Three components, one loop

metrics-server usage, every 1m recommender Checkpoint history status.recommendation VPA object both read the target admission controller (webhook) updater patch on CREATE evict resize Pods
ComponentDoes
recommenderevery 1m reads metrics.k8s.io, writes status and checkpoints. One active, metrics :8942
updaterevery 1m evicts or resizes out-of-date Pods. Evicted Pods get new values from the webhook. One active, metrics :8943
admission controllerwebhook on Pod CREATE, failurePolicy: Ignore. Patches requests, limits. Several active, metrics :8944

3.The VPA object

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata: {name: my-app-vpa}
spec:
  targetRef:
    {apiVersion: apps/v1, kind: Deployment, name: my-app}
  updatePolicy: {updateMode: "Recreate"} # not "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: "*"
      minAllowed: {cpu: 100m, memory: 50Mi}
      maxAllowed: {cpu: 1, memory: 500Mi}
      controlledResources: ["cpu", "memory"]
      controlledValues: RequestsAndLimits
  • targetRef : any controller with a /scale subresource, or a well-known kind (Deployment, StatefulSet, DaemonSet, Job...). Its Pods must be owned directly by it, so operator Pods need a CR with /scale.
  • Per container mode: Off skips a container (its default, Auto, is not the deprecated one).
  • controlledValues: RequestsOnly leaves limits alone.
  • updatePolicy.minReplicas overrides the updater's floor. evictionRequirements filter by direction, for example only evict to scale up.

4.Update modes

ModeAt creationRunning PodsStatus
Offnonorecommend only
Initialyesnostable
Recreateyesevicts (PDBs apply)default when unset
InPlaceOrRecreateyesin place first, eviction as fallbackGA in VPA 1.6.0
InPlaceyesin place only, never evictsalpha in 1.7.0, gate InPlace
Autoyesevictsdeprecated in 1.5.0, same as Recreate

5.Install

  • Script : install metrics-server first, then ./hack/vpa-up.sh from a clone of kubernetes/autoscaler. It installs into kube-system and needs OpenSSL 1.1.1+. vpa-down.sh removes it.
  • Helm : chart 0.13.0, "not ready for production" in its README. It sets --in-place-skip-disruption-budget=true, unlike the binary. Recommender only : --set updater.enabled=false --set admissionController.enabled=false.
In short : VPA moves requests toward usage. The recommender only writes numbers; the updater and the webhook apply them, by updateMode and by which of them you installed.

1.Reading a VPA

kubectl get vpa : MODE, CPU, MEM (first container's target), PROVIDED; -o wide adds MinReplicas, OOMSeconds. Status has four numbers per container :

  • targetwhat the webhook applies.
  • lowerBound"not guaranteed to be sufficient".
  • upperBoundabove it is likely waste.
  • uncappedTargetbefore minAllowed / maxAllowed.
Conditions
  • RecommendationProvided
  • LowConfidence
  • NoPodsMatched
  • FetchingHistory

2.The recommender recipe

  1. Grouped per VPA and container name, not Pod.
  2. CPU : a decaying histogram, 24h half-life.
  3. Memory : one peak per 24h interval, 8 intervals.
  4. Target p90, lower p50, upper p95, then +15%.
  5. Short history widens bounds : upper x3 at 12h, x2 at 24h, x1.14 at a week; lower x0 with none.
  6. Per-Pod floor 25m / 250Mi, split across containers.
  7. Caps : min/maxAllowed, LimitRange.

An OOMKill adds a synthetic sample of max(used + 100Mi, used x 1.2). The floor unit is MiB, even though the flag says -mb.

Recommender flagDefaultEffect
--recommender-interval1mloop
--cpu-histogram-decay-half-life24hday-old weighs half
--memory-aggregation-interval x -count24h x 8a peak a day
--target-cpu-percentile, memory0.9target
--recommendation-margin-fraction0.15+15% on all
--pod-recommendation-min-cpu-millicores, -memory-mb25 / 250floor, in MiB

3.When the updater acts

lowerBound target upperBound belowact insideno action aboveact also : running 12h+ and a change of 10%+, or OOMKilled after running under 10 minutes
One replica is never evicted with defaults. Set updatePolicy.minReplicas: 1 if you mean it.
  • Candidate : a request outside [lowerBound, upperBound] (a missing request counts as outside), or one of the two rules in the box.
  • The largest summed change goes first.
  • Safety : at least minReplicas (2) live, at most half evicted at once (tolerance 0.5), PDBs through the Eviction API, an optional rate limit, and no evictions if the admission controller looks dead.

4.In place, and its fallback

InPlaceOrRecreate resizes all containers at once through pods/resize (In-Place Resize). It evicts instead :

  • at oncefailed PATCH (a QoS change is rejected), Infeasible, or reason Error.
  • later5 min Deferred, or 1 hour in progress (fixed, not flags).

The fallback still obeys minReplicas and tolerance. The kubelet refuses a memory-limit cut below current usage (reason Error, see In-Place Resize), so VPA evicts instead. InPlace (alpha) never evicts; it retries an infeasible size once the recommendation drops.

5.Limits and LimitRange

  • RequestsAndLimits keeps the request:limit ratio. Request 10m with limit 1 CPU (ratio 100) : at request 300m the limit becomes 30 CPU. Use RequestsOnly, or keep them close (Requests and Limits).
  • A limit of 0 is not "unlimited" : with a request set, it is rejected. Omit the limit.
  • Over a LimitRange max, the limit drops to max and the request in proportion.
  • If the VPA policy and a LimitRange conflict, VPA's docs say VPA wins.
In short : Targets are p90 of decayed usage plus 15%; bounds widen on short history. The updater acts outside the bounds, after 12h with a 10% change, or on a quick OOM, and by default never below two replicas.

1.VPA with HPA

Both on CPU : a loop VPA raisesCPU request HPA utilizationdrops HPA scales in per-Pod usagerises Safe : split by metric VPA : memory HPA : CPU Deployment requests replicas
PatternSafe ?Why
VPA on CPU, HPA on CPU utilizationnothe loop
VPA controlledResources: ["memory"], HPA on CPUyesupstream pattern
HPA on custom or external metrics (RPS, queue depth, KEDA), VPA on bothyesdifferent signals
VPA Off, any HPAyesnothing applied

HPA utilization is usage divided by request, and VPA moves the request. More in HPA in Depth.

2.Recommender only

  • Install only the recommender, give every workload a VPA with updateMode: "Off", and change requests by hand in Git : no evictions, no webhook.
  • Goldilocks : a dashboard on top of it.
  • kube-state-metrics removed its VPA metrics in v2.9.0 : export status.recommendation with CustomResourceState.
max_over_time(container_memory_rss
  {namespace="NS",container="C"}[7d])
max_over_time(container_memory_working_set_bytes
  {namespace="NS",container="C"}[7d])

PromQL : close together, the target is honest. RSS far below : page cache inflates it.

3.Where the numbers mislead

  • Memory follows working set, page cache included : file-heavy workloads look far too big.
  • Bursty, short-lived workloads get undersized : p90 + 15% misses rare peaks (Jobs).
  • A recommender restart forgets the current interval's peak so far.
  • Identical tiny values are the floor, not data.
  • Labels matching stray Pods pollute the aggregate. Two VPAs on one Pod : undefined.
  • Pod-level resources are not supported.
  • Bigger than any node or quota : Pods stay Pending. Set maxAllowed (Karpenter).

4.Operating it

  1. Read release notes first : the 1.7.0 recommender reset recommendations from checkpoints (fixed in 1.7.1).
  2. Back up verticalpodautoscalercheckpoints first.
  3. After the roll, prove history loaded : log line Fetching VPA checkpoints ... count=N. PROVIDED=True proves nothing (old status stays).
  4. Do not resize from numbers within 24h of a recommender restart.

The webhook's failurePolicy: Ignore silently skips mutation while it is down.

Seen in practice
  • Memory limits copied from VPA on a file-heavy service came out many times its resident memory. Bursty proxies sized from VPA came out too small : VPA saw only steady state.
  • Pitfall : treating the VPA target as the size.
  • Do better : use VPA as one input. Check memory against peak RSS, size bursty and data-path Pods from peaks. Keep a CPU limit off the recommender : its own throttling skews every number.

5.Commands

CommandWhat it shows
kubectl get vpa -AMODE, CPU, MEM, PROVIDED for every VPA
kubectl describe vpa web -n shoprecommendation, conditions, eviction events
kubectl get vpa web -n shop -o jsonpath='{.status.recommendation.containerRecommendations[*].target}'just the targets
kubectl get pod <p> -o jsonpath='{.metadata.annotations}'what the webhook applied (vpaUpdates)
kubectl get verticalpodautoscalercheckpoints -A -o json > checkpoints.jsonback up before an upgrade
kubectl -n kube-system logs deploy/vpa-recommender | grep -E 'Fetching VPA checkpoints|Error while loading checkpoint'did history load
In short : Keep VPA and HPA on different signals, start with recommender-only and updateMode: "Off", and check memory targets against resident memory before you trust them.

Resources and autoscaling

Horizontal Pod Autoscaler in Depth

1.Beyond the basics

  • The formula : desired = ceil(current x value / target); a ratio of 0.9 to 1.1 changes nothing (tolerance 0.1). Deployments and Scaling works an example. Everything else here stops that formula from flapping.
  • autoscaling/v2 is GA since 1.23 (v2beta2 not served since 1.26). autoscaling/v1 is still served, CPU only; v2-only fields survive as annotations.
  • It scales anything with a scale subresource : Deployment, StatefulSet, ReplicaSet, CRDs. Never a DaemonSet.
  • A controller loop in kube-controller-manager, every 15s. In 1.37 a new HPA, or one whose spec changed (HPAGeneration, beta, on), is reconciled at once.

2.One sync

  1. Read the target's /scale : replicas and selector.
  2. Out of min..max today ? Clamp, without reading metrics.
  3. current in the formula is scale.spec.replicas, the desired count, not the Pods running now.
  4. Every metric failed, or some failed and the rest say "fewer" : an error, no change.
  5. Some failed and the rest say "more" : scale up, and still report the error.
HPA object controller, every 15s read /scale : replicas, selector list Pods by selector metrics APIs ratio per metric, largest wins behavior : window, policies write scale.spec.replicas Deployment splits across ReplicaSets 0 replicas and not ScaledToZero : stop (paused) some failed : scale up only then clamp to min and max

3.Which Pods count

  1. Pods being deleted are ignored, Failed Pods discarded (phases : Pods).
  2. Pending Pods (every metric type) and Pods with no sample are set aside.
  3. CPU only : a not-yet-ready Pod is set aside. In its first 5 minutes if NotReady or its last sample predates its readiness change; after that only if it was never Ready.
  4. The ratio comes from the rest. Inside tolerance : stop.
  5. Recompute conservatively. If that flips direction or lands inside tolerance : no change (status still shows the plain average).
No metric ?

On a scale down a missing Pod counts as 100% of its request (or the target if higher), on a scale up as 0%. Set-aside Pods : 0% on a scale up. Pods metrics : the target, going down.

Object, External : a Value target multiplies by Running and Ready Pods; AverageValue gives ceil(total / target), checked against status.replicas.

4.kube-controller-manager flags

Flag ("..." = --horizontal-pod-autoscaler)Default
...-sync-period15s
...-tolerance0.1
...-downscale-stabilization5m
...-cpu-initialization-period5m
...-initial-readiness-delay30s
--concurrent-horizontal-pod-autoscaler-syncs5

Cluster-wide; a per-HPA tolerance overrides it. -upscale-delay and -downscale-delay are gone since 1.31 : set one and the process exits (unknown flag).

5.Utilization needs requests

  • Utilization is usage as a percent of the request (see Requests, Limits and QoS).
  • If any container in a Pod (often a sidecar) has no request for that resource, the HPA takes no action on that metric.
  • Fix : request on every container, or ContainerResource for one named container (next sheet).
  • With Pod-level requests set (PodLevelResources, beta, on) and no container named, the Pod-level request is the denominator.
In short : Every 15s the HPA reads one number per metric, divides it by the target, multiplies by the current replicas and rounds up. It ignores or sets aside Pods it cannot trust and recomputes conservatively, so it moves only when the evidence is clear.

1.Five metric types

TypeReadsTargetsFrom
Resourcecpu or memory per Pod, summed over containersUtilization, AverageValuemetrics.k8s.io
ContainerResourceone named container (GA 1.30)Utilization, AverageValuemetrics.k8s.io
Podsa per-Pod custom metric, averagedAverageValue onlycustom.metrics.k8s.io
Objectone object in the namespace (describedObject)Value, AverageValuecustom.metrics.k8s.io
Externalnot tied to an object; matching series are summedValue, AverageValueexternal.metrics.k8s.io

No metric set : CPU Utilization 80%. Several metrics : each one proposes a count and the largest wins.

2.Utilization, AverageValue, Value

TargetComparesReplicas
Utilizationmean usage / request, in percent, over the counted Podsceil(current x mean% / target%)
AverageValuemean per Pod (Object, External : total / status.replicas)ceil(current x mean / target), or ceil(total / target)
Valueone raw series value, as isceil(readyPods x value / target)

AverageValue on CPU or memory needs no request (raw usage, 500m). 1500m means 1.5.

3.Where the numbers come from

metrics-server metrics.k8s.io prometheus-adapter custom.metrics.k8s.io KEDA (or an adapter) external.metrics.k8s.io one server per cluster kube-apiserver aggregation HPA controller
  • metrics.k8s.io/v1 is GA in 1.37, yet the 1.37 HPA reads v1beta1 of both the resource and external APIs : keep serving it.
  • custom.metrics.k8s.io : the version the server prefers (discovery).
  • Installing KEDA takes external.metrics.k8s.io from prometheus-adapter. See metrics-server, Prometheus.
  • The docs prefer custom over external : the external API can expose anything.

4.The behavior field

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300 # kcm flag
    policies:
    - {type: Percent, value: 100, periodSeconds: 15}
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
    - {type: Percent, value: 100, periodSeconds: 15}
    - {type: Pods, value: 4, periodSeconds: 15}
    selectPolicy: Max
  • Each sync records a recommendation. Scale down takes the highest one in the window (0 to 3600s) : a rolling max.
  • A policy caps change over its periodSeconds (1 to 1800). selectPolicy : Max = most change, Min = least, Disabled = off.
  • Per-direction tolerance, GA in 1.37 : 0.05 down and 0.01 up on a 100Mi target scale down below 95Mi, up above 101Mi.
  • No behavior at all (every autoscaling/v1 HPA) : the legacy path, scale up capped at max(2 x current, 4) per sync (ScaleUpLimit).

5.A policy at work

From 80 replicas, scale down with Pods 4 / 60s and Percent 10 / 60s (Max) : above 40, 10% allows more, so it removes 8, then 8 (7.2 rounded up); below 40, 4 at a time.

"Scale down feels slow" : that is the 300s window, by design. Change it per HPA or with the kcm (kube-controller-manager) flag.

In short : Pick the metric type by what you can measure, the target type by what it means; the largest proposal wins. behavior : no wait up, a 5-minute rolling max down, policies cap each step.

1.Reading describe hpa

ConditionTrue reasonsFalse reasons
AbleToScaleReadyForNewScale, ScaleUpStabilized, ScaleDownStabilized, SucceededRescale ...FailedGetScale, FailedUpdateScale
ScalingActiveValidMetricFoundScalingDisabled, FailedGet...Metric ...
ScalingLimitedTooManyReplicas, TooFewReplicas, ScaleUpLimit, ScaleDownLimitDesiredWithinRange
ScaledToZeroScaledToZeroNotScaledToZero

Events : SuccessfulRescale, FailedComputeMetricsReplicas, AmbiguousSelector (two HPAs, same Pods : no scaling).

2.Rollouts and GitOps

  • The HPA owns replicas; the Deployment splits them across ReplicaSets, surge Pods included.
  • Drop spec.replicas from the manifest, or every apply resets it. Naively it drops to 1 once : use kubectl apply edit-last-applied (client-side) or transfer field ownership (server-side, API Machinery, Flux).
  • Readiness that passes only when warm keeps startup CPU from scaling up (Rollouts and Probes). CPU target with headroom : (1 - buffer) / (1 + growth); 10% and 30% give about 70%.

3.Scale to zero in 1.37

N replicas HPA scales down : minReplicas 0 + Object / External metric rises : ceil(usage ratio) 0, ScaledToZero=True someone scales to 0 0, paused : ScalingActive False, ScalingDisabled the HPA will not wake it
  • Gate HPAScaleToZero : beta, on in 1.37, in kube-apiserver and kube-controller-manager. Upgrade both first; minReplicas >= 1 before a downgrade.
  • minReplicas: 0 needs an Object or External metric; CPU or memory alone is rejected.
  • ScaledToZero came in 1.36. Raising minReplicas wakes an HPA-held zero. From zero : no tolerance or ready-Pod count; on the way to zero the 300s window applies.

4.Beside VPA, KEDA and Karpenter

ToolChangesSignalSee
HPAreplica countresource, custom, external metricsthis chapter, Deployments and Scaling
VPAcontainer requestsusage historyVertical Pod Autoscaler
KEDAcreates and drives an HPA, adds 0 to 1event sourcesKEDA
Karpenternodesunschedulable PodsKarpenter
  • Never VPA and HPA on the same resource metric (VPA). KEDA owns keda-hpa-<name> : add no HPA of your own to that workload. min = max : a no-op.
  • CPU limit = request : utilization tops out near 100%, so a sync grows replicas by at most about 100 / target (1.25x at 80%); throttling hides demand (Requests and Limits).
Seen in practice

KEDA-scaled services that never go to zero, with a tuned cooldownPeriod. Pitfall : it only governs scaling to zero; N to 1 follows the HPA's 300s window. Do better : set behavior in advanced.horizontalPodAutoscalerConfig; check the metric with kubectl get --raw.

5.Commands

CommandShows
kubectl get hpa -ATARGETS as current / target
kubectl describe hpa webmetrics, conditions, events
kubectl get events --field-selector involvedObject.kind=HorizontalPodAutoscaler,involvedObject.name=webrescale history and failures
kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/<ns>/podsthe numbers the HPA reads
In short : Read the conditions first, let the HPA own replicas, and give scale to zero an Object or External metric. Never let two autoscalers drive the same workload on the same signal.

Networking internals

Pod Networking and CNI

1.What Kubernetes asks for

  • Kubernetes states the rules (one IP per Pod, Pod to Pod with no NAT). A plugin builds them.
  • Observed on k3s v1.37.1 : Pod IPs crossed the tunnel intact (10.42.1.2 > 10.42.0.5).
Three address owners
  • Pod IPsthe network plugin (CNI)
  • Service IPskube-apiserver
  • Node IPskubelet or cloud controller

2.Who calls CNI

  • The kubelet only calls CRI. containerd makes the netns (none for hostNetwork) and runs CNI.
  • The kubelet sorts the IPs into status.podIPs, then sets PodReadyToStartContainers (GA v1.37).
  • Errors : kubectl get events -A --field-selector reason=FailedCreatePodSandBox
kubelet containerd (CRI) netns cni-<uuid> RunPodSandbox create status.podIPs sorted by node family libcni first conflist only CNI ADD : env + stdin stdout : ips, routes flannel main plugin bridge cni0 + veth exec host-local IPAM exec portmap hostPort then bandwidth tc shaping CNI agent DaemonSet flanneld, calico-node, cilium-agent hostNetwork, runs before any Pod writes the conflist + binaries, programs routes between nodes

3.The CNI contract

  • A plugin is a binary : CNI_* env in, JSON config on stdin, a result (ips, routes) or an error code on stdout.
  • Spec 1.1.0 added GC and STATUS. containerd sends STATUS (1.1.0 configs only), never GC.
  • DEL runs the chain in reverse, must be repeatable, and follows every ADD, even a failed one.
VariableADDDELGCSTATUS
CNI_COMMANDreqreqreqsent
CNI_CONTAINERIDsandboxreq--
CNI_NETNSreqopt--
CNI_IFNAMEeth0req--
CNI_ARGSK8S_POD_*opt--
CNI_PATHoptoptreqopt

4.A conflist, chained

  • containerd loads only the lexically first file (max_conf_num = 1) : a runtime rule, not the spec.
  • k3s uses /var/lib/rancher/k3s/agent/etc/cni/net.d.
  • Later plugins get prevResult; the main one runs its own IPAM plugin.
  • bandwidth : Pod egress is shaped on an ifb named bwp<hash>.
k3s 10-flannel.conflist
{ "name": "cbr0", "cniVersion": "1.0.0", "plugins": [
  { "type": "flannel", "delegate": { "hairpinMode": true,
      "forceAddress": true, "isDefaultGateway": true } },
  { "type": "portmap", "capabilities": { "portMappings": true } },
  { "type": "bandwidth", "capabilities": { "bandwidth": true } } ] }

5.A node with no network

  • Empty conf dir : cni plugin not initialized, Node Ready=False, not-ready taints (evict after 300s).
  • Only hostNetwork Pods start, so CNI agents are hostNetwork DaemonSets.
  • A conflist mv'd in is ignored (create event). Check crictl info | jq '.lastCNILoadStatus'.
  • NetworkUnavailable is a separate signal with its own taint.
WriterNetworkUnavailable reason
Route controllerRouteCreated, NoRouteCreated
Upstream flannelFlannelIsUp
calico-nodeCalicoIsUp, CalicoIsDown
Cilium operatorCiliumIsUp
kubelet, k3s flannelnever
In short : Kubernetes only states the rules. The runtime makes a netns and runs the first conflist's plugin chain for the Pod's IPs; with no conflist, only hostNetwork Pods run.

1.Inside the Pod, on the host

  • Pod : eth0@if5, 10.42.1.2/24, default via 10.42.1.1 (the cni0 address).
  • Host : veth71d39cee@if2 ... master cni0. if5 and index 5: are one veth pair.
  • cni0 appears with the first non-hostNetwork Pod. Pod MTU 1450 from FLANNEL_MTU.

Observed on k3s v1.37.1+k3s1, flannel VXLAN.

Pod A netns eth0 10.42.1.2 Pod B netns eth0 10.42.1.3 veth veth cni0 bridge 10.42.1.1 same node : L2 over cni0 node routes 10.42.0.0/24 flannel.1 VTEP eth0 uplink other node

2.Bridge or routed

  • flannel : Pods share an L2 on cni0, so a Pod with NET_RAW can ARP-spoof. k3s says use restricted.
  • Calico : a cali* host side with proxy ARP, one /32 route per local Pod.
  • Calico v3.33 has Felix, not BIRD, program IPIP routes by default. Its docs page still says BGP.
flannel + bridgeCalico routed
Host sideport on cni0cali*, /32 route
Pod gatewaycni0 (.1)169.254.1.1
Neighbourslocal Pods, one L2MAC ee:ee:ee:ee:ee:ee
ARP spoofingyes, with NET_RAWno shared L2

3.One packet across nodes (flannel VXLAN, observed)

  • Per peer : one route, one neighbour, one FDB entry (ip route, ip neigh, bridge fdb), from Node annotations.
  • tcpdump -nni <uplink> udp port 8472 : VXLAN ... vni 1 around IP 10.42.1.2 > 10.42.0.5.
  • 50 bytes each : ping -M do -s 1422 passes, 1423 fails. A missed MTU : pings work, big transfers hang.
Node 2 : 172.31.250.11, holds 10.42.1.0/24 Pod B 10.42.1.2 cni0 gateway .1 route via flannel.1 neighbour to VtepMAC FDB MAC to .250.10 flannel.1 wrap, UDP 8472 outer 172.31.250.11 > .10, vni 1, inner Pod IPs untouched Node 1 : 172.31.250.10, holds 10.42.0.0/24 flannel.1 unwrap cni0 bridge Pod A 10.42.0.5

4.Overlay, routed or VPC-native

  • Overlay : only UDP between nodes. Routed : one L2 (host-gw) or BGP. VPC-native : real VPC IPs.
  • Cilium defaults to routingMode: tunnel (VXLAN).
  • 8472 (kernel) vs 4789 (IANA) : a firewall for one breaks the other. Never expose 8472.
On a 1500 linkPortCostPod MTU
flannel vxlanUDP 8472501450
flannel host-gwnone01500
wireguard-nativeUDP 51820801420
Calico VXLANUDP 4789501450
Cilium VXLAN, Geneve8472, 6081501450

5.k3s defaults and multi-NIC traps

  • Pods 10.42.0.0/16, Services 10.43.0.0/16, DNS 10.43.0.10.
  • Backends : vxlan, host-gw, wireguard-native, none. Others fail.
  • flannel binds the default-route NIC, not the node-ip one : nodes Ready, 100% Pod loss.
  • flannel.1 dies with its uplink; k3s v1.36.0 to v1.37.1 (IPv4-only) then panic and restart.
/etc/rancher/k3s/config.yaml
node-ip: 172.31.254.11
flannel-iface: fabric0  # on every node
  • ip -d link show flannel.1 : local <ip> dev <nic>
  • flannel.alpha.coreos.com/public-ip must equal the InternalIP.
  • Alert on a missing flannel.1, k3s-agent restarts.
In short : Pods hang off veths on their node. flannel wraps cross-node packets in UDP 8472 (MTU 1450), routed designs skip the tunnel, and multi-NIC k3s needs flannel-iface.

1.Node Pod ranges

  • KCM --allocate-node-cidrs (off by default) carves /24 and /64 blocks.
  • spec.podCIDRs is set once : no change, not even a second family.
  • A /16 of /24s holds 256 nodes; node 257 gets CIDRNotAvailable, no range.
  • IPv6 : /64 per node needs a /48 or longer cluster CIDR.
  • kubectl get nodes -o custom-columns=N:.metadata.name,CIDR:.spec.podCIDRs
--cluster-cidr 10.42.0.0/16 node IPAM controller RangeAllocator in KCM Node A 10.42.0.0/24 Node B 10.42.1.0/24 node 257 CIDRNotAvailable podCIDRs flanneld writes subnet.env host-local .2 to .254 = 253 Pod IPs per Pod ADD

2.When the CNI owns IPAM

  • In-tree ClusterCIDR was removed in v1.29. node-ipam-controller v0.3.0 brings it back as a CRD.
  • Calico and Cilium ignore podCIDRs; block sizes are permanent.
  • Cilium's default 10.0.0.0/8 can hit node networks.
IPAM modelPicked byPer node
RangeAllocatorKCM/24, 253 IPs
ClusterCIDRnode-ipam-ctrlperNodeHostBits
calico-ipamCalico/26 blocks
cluster-poolcilium-operatormask 24

3.Service ranges are objects

  • ServiceCIDR and IPAddress : GA v1.33, v1beta1 gone in v1.37. See kubectl get servicecidr, ipaddress.
  • Random picks use the upper band : hand-picked IPs (DNS .10) go low.
  • Full : failed to allocate a serviceIP ... range is full.
  • v1.37 : no IPv4 size limit. IPv6 shorter than /64 shows Ready, allocates nothing.
Grow it live
apiVersion: networking.k8s.io/v1
kind: ServiceCIDR
metadata:
  name: newcidr1
spec:
  cidrs: ["10.96.0.0/24"]

The failed Service then got 10.96.0.173. Guard it : Admission.

4.Dual-stack end to end

  • Both families on every component : apiserver, KCM, kubelet --node-ip (first = primary), CNI.
  • status.podIPs follows the node's primary family, not the CNI (10.42.1.2, then fd00:42:0:1::2). k3s : dual-stack only at creation.
  • ipFamilyPolicy : SingleStack (default), PreferDualStack, RequireDualStack. Primary never changes.
  • get svc hides the second IP : kubectl describe svc <s> | grep -E '^IP'.

5.Symptoms, and second networks

  • portmap : skip KUBE-MARK-MASQ (absent in nftables mode); its nftables backend before v1.9.0 has CVE-2025-67499.
  • Multus sorts first (00-multus.conf) and adds net1. DRA reports networkData (GA v1.37).
SymptomCheckCause
no IP addresses availablefiles in /var/lib/cni/networks/<net>leaks (no CNI GC) or too many Pods
cni0 already has an IPdid the node subnet change ?forceAddress: false
subnet doesn't contain podCIDRflannel Network vs KCMkubeadm range not 10.244.0.0/16
hostPort refused, no eventsconflist portMappings ?no portmap in the chain
incompatible CNI versionsCNI_COMMAND=VERSION on the binaryplugins older than the conflist
In short : Plan addresses once : node ranges never change, ServiceCIDRs grow live, and dual-stack needs both families on every component from day one.

Networking internals

The Service Dataplane

1.A controller, not a proxy

A Service is kernel NAT rules, not a process. This picks up where the watch loops in Services stop (v1.37).

  • Informers : Services (never headless ones), EndpointSlices, ServiceCIDRs and its own Node.
  • Runner : syncs at least 1s apart, retries within 30s, full sync at least hourly.
  • Partial sync : changed Services only. Over 1000 endpoints, iptables skips the hourly full sync.
informers Services, Slices, Node, CIDRs change trackers old + new state sync runner 1s min, 30s retry, 1h max iptables-restore or one nft transaction client socket Pod or host process 1st packet kernel rules DNAT, SNAT mark records conntrack entry keeps the pick later : replayed

2.Ports, health and metrics

  • :10256/healthz : 200 unless an update waited over 2 x syncPeriod (60s) or the Node is being deleted. /livez ignores the Node.
  • 127.0.0.1:10249 (loopback) : /metrics, /configz, /proxyMode. Try curl -s 127.0.0.1:10249/proxyMode.
  • k3s binds healthz to 127.0.0.1, so LB probes on <nodeIP>:10256 fail.
kubeproxy_...Tells you
sync_full_ / sync_partial_proxy_rules_duration_secondsSync cost
network_programming_duration_secondsSlice change to rules
sync_proxy_rules_last_timestamp_secondsStaleness
..._iptables_restore_failures_total, ..._nftables_sync_failures_totalFailed syncs

3.The config that matters

  • Unset mode = iptables plus an error log line. The KubeProxyDefaultMode event never arrives : grep the logs for No kube-proxy mode specified.
  • Sync settings come from the section matching mode. Flags lose to --config, and an unknown field only logs. Lint with --write-config-to.
Pin the mode (validated, v1.37.1)
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: nftables        # never leave it unset
clusterCIDR: 10.244.0.0/16
nftables:             # not iptables:
  syncPeriod: 30s     # retry, not resync

4.iptables mode, chain by chain

  • KUBE-SERVICES : one rule per Service port, walked in order. O(Services) per new connection.
  • KUBE-SVC : rule i jumps with --probability 1/(N-i), the last always. Each endpoint gets 1/N.
  • KUBE-SEP DNATs to PodIP:port. Marked packets (0x4000) are masqueraded on the way out.
HookJumps to (verdict)
nat PREROUTING, OUTPUTKUBE-SERVICES (DNAT)
nat POSTROUTINGKUBE-POSTROUTING (SNAT)
filter FORWARD, OUTPUTKUBE-SERVICES (REJECT)
filter INPUT, FORWARDKUBE-EXTERNAL-SERVICES (DROP)
-d 10.96.144.12 --dport 80 nat PREROUTING or OUTPUT (host) KUBE-SERVICES 1 rule per port KUBE-SVC-CDGG... off-cluster : mark p=0.5 else KUBE-SEP-7U6M... DNAT 10.244.0.5:8080 KUBE-SEP-UIQK... DNAT 10.244.1.2:8080 last rule (local dst) KUBE-NODEPORTS by --dport KUBE-EXT-<hash> external dst rules KUBE-SVC or SVL SVL : local endpoints KUBE-MARK-MASQ set mark 0x4000 nat POSTROUTING : KUBE-POSTROUTING marked : MASQUERADE --random-fully SVC, SEP and EXT jump to the mark nat PREROUTING or OUTPUT (host) KUBE-SERVICES 1 rule per port -d 10.96.144.12:80 KUBE-SVC-CDGG... off-cluster : mark p=0.5 else KUBE-SEP-7U6M... DNAT 10.244.0.5:8080 KUBE-SEP-UIQK... DNAT 10.244.1.2:8080 KUBE-NODEPORTS last SERVICES rule KUBE-EXT-<hash> then SVC or SVL KUBE-MARK-MASQ set mark 0x4000 KUBE-POSTROUTING MASQUERADE

Real chain names from iptables-save on a kind node (v1.37.1, observed 2026-10-07).

5.When rules vanish

  • Only the mangle KUBE-PROXY-CANARY is polled. If it goes, a full resync follows (observed : 18s).
  • Flushing only nat or filter goes unnoticed (observed : broken after 80s). Restart kube-proxy.
iptables-save is not the input

The mark prints as --set-xmark 0x4000/0x4000, probabilities as 0.50000000000. Grep names : iptables-save -t nat | grep -E 'KUBE-(SERVICES|SVC|SVL|EXT|SEP)'

In short : kube-proxy turns Services and EndpointSlices into node rules. In iptables mode a new connection walks KUBE-SERVICES, takes a random jump and is DNATed.

1.nftables mode

  • Owns table ip kube-proxy and table ip6 kube-proxy (no inet table). Base chains : nat-prerouting, nat-output, nat-postrouting, filter chains.
  • services marks off-cluster ClusterIP traffic, then does one lookup : nft list map ip kube-proxy service-ips.
  • A Service chain picks the endpoint in one numgen random mod N rule (since v1.36). One endpoint : a plain dnat to. Endpoint chains remain only for ClientIP affinity.
  • The no-endpoint-* maps reject (no endpoints) or drop (none local). Never edit this table or the mark.
nft list table ip kube-proxy (kind, v1.37.1, observed)
chain services {
  ip daddr @cluster-ips ip saddr != 10.244.0.0/16 meta mark set meta mark | 0x00004000 ...
  ip daddr . meta l4proto . th dport vmap @service-ips
  ip daddr @nodeport-ips meta l4proto . th dport vmap @service-nodeports }
chain service-SUSTUHLJ-default/web/tcp/http {
  meta l4proto tcp dnat ip to numgen random mod 2 map { 0 : 10.244.0.5 . 8080, 1 : 10.244.1.2 . 8080 } }

2.Why it scales, and the switch

  • iptables : O(Services) per new connection and update, global lock. nftables : one lookup, updates O(changes).
  • Changes : NodePorts on the primary IP only, none on localhost (v1.37 alpha gate). Open the NodePort range yourself. Kernel 5.13+.
  • Switch : check the kubeproxy_iptables_* counters, set mode: nftables. Old rules go away; mixed nodes are fine.
Releaseiptablesnftablesipvs
v1.37defaultGAdeprecated, gated
v1.39kubeadm rejects 5.4 and 5.10 kernels
v1.40not defaultdefaultoff
v1.43okokremoved

From v1.39 on : plans (KEP-5343 is provisional).

3.ipvs, while it lasts

  • Service IPs sit as /32s on the dummy kube-ipvs0. One IPVS virtual server per IP and port, scheduler rr, NAT.
  • Masquerade, NodePorts and filtering still use iptables, matched against 17 ipsets.
  • Leaving ipvs for iptables keeps kube-ipvs0 : stop kube-proxy, then run kube-proxy --cleanup.

4.conntrack, the hidden half

  • Later packets skip the nat table, so only new connections re-balance. gRPC and HTTP/2 stay put. Pins : conntrack -L -d <clusterIP>.
  • UDP : stale entries are deleted after a sync (v1.37 : also with zero endpoints). TCP : never.
  • Max = 32768 x CPUs, capped at 1,048,576 (v1.36), at least 131072. Check conntrack -S.
  • Read-only sysctl (LXC) : v1.36 and v1.37.1 can fail to start. Set maxPerCore: 0.
  • TCP established 24h, close-wait 1h. k3s leaves both to the kernel.
Pod 10.244.0.6:48986 SYN to 10.96.144.12:80 first packet nat rules pick 10.244.1.2:8080, DNAT conntrack entry orig : dst 10.96.144.12:80 reply : src 10.244.1.2:8080 later packets and replies : no rule walk UDP endpoints changed in a sync kube-proxy deletes stale entries

5.SNAT and the client IP

  • SNAT : masqueradeAll, off-cluster or host sources to a ClusterIP, hairpin, and all external traffic under eTP=Cluster.
  • MASQUERADE uses the egress address. "Local Pod" comes from detectLocalMode (default ClusterCIDR; no clusterCIDR = no-op).
PathBackend sees (observed)
Pod to ClusterIPthe client Pod IP
Pod to its own ServiceSNAT, 10.244.0.1
host to ClusterIPnode IP if remote, 10.244.0.1 if local
external, eTP=Clustera node address, even if local

Test : wget -qO- http://<svc>/clientip (agnhost). eTP=Local : Services.

In short : nftables swaps the rule walk for one map lookup (planned default in v1.40), ipvs is leaving, conntrack makes the first pick stick, and SNAT decides who the backend sees.

1.What kube-proxy reads

  • serving mirrors Pod Ready, terminating comes with deletion, ready = serving and not terminating.
  • Local = nodeName equals kube-proxy's node name, never by IP. A wrong hostnameOverride makes all endpoints remote.
  • 100 endpoints per slice by default (Controllers).
Field or labelEffect
kubernetes.io/service-nameRequired link
addresses[0]The only IP used
conditionsready, serving, terminating
nodeNameLocality
zone, hintsTopology

2.Which endpoints a connection can reach

  • Per policy : Ready endpoints, else serving and terminating ones. This holds for Cluster policy too (docs say Local only).
  • Observed : a Pod with a 120s preStop sleep, scaled to 0, kept taking new connections.
  • kube-proxy uses hints only if every Ready endpoint has one. Local policies win over hints.
  • PreferSameZone has no overload guard : spread replicas (Scheduling). Values : Services.
Local traffic policy? yes no (Cluster) local Ready this node's endpoints Ready node hint, zone hint, or all none none local serving + terminating serving + terminating none none DROP client times out REJECT connection refused

3.Health checks for Local policies

  • Past the eTP panel in Services : one listener per Service on healthCheckNodePort, 200 with local Ready endpoints, else 503, plus header X-Load-Balancing-Endpoint-Weight.
  • Probe : curl -si http://<nodeIP>:<port>/healthz. Terminating local Pods fail it, so the LB drains the node.
  • iTP=Local with no local endpoint DROPs (timeout), not the REJECT the docs imply.

4.eBPF : Services without netfilter

  • Verified programs run at hooks (XDP, tc, cgroup connect4) and keep state in maps.
  • Cilium kubeProxyReplacement: true : socket LB at connect(), NodePorts per packet (snat, dsr, hybrid).
  • Calico eBPF : connect-time LB for TCP only. NodePorts keep the client IP, even with eTP=Cluster.
  • Traps : socket LB breaks Istio sidecars, KubeVirt, Kata, gVisor. tcpdump misses XDP.
in-cluster : socket LB, once per connection connect() to 10.96.0.50:80 cgroup hook connect4 service map pick a backend socket to 10.244.1.7:8080 on the wire no DNAT north-south : per packet external client NodePort or LB IP XDP or tc at the NIC Maglev pick consistent hash SNAT or DSR loadBalancer.mode backend node DSR : replies direct in-cluster north-south connect() to 10.96.0.50:80 cgroup hook connect4 service map pick a backend socket to 10.244.1.7:8080 on the wire no DNAT external client NodePort or LB IP XDP or tc at the NIC Maglev pick consistent hash SNAT or DSR loadBalancer.mode backend node DSR : replies direct
iptablesnftablesipvsCilium KPRCalico eBPF
v1.37defaultGAdeprecatedopt-inopt-in
Dispatchrule walkone mapIPVS tableBPF mapBPF map
Stateconntrackconntrack+ IPVSBPF mapsBPF maps
Kernel-5.13-5.105.10

5.Removing kube-proxy

  • kubeadm : --skip-phases=addon/kube-proxy. Live cluster : delete the DaemonSet and ConfigMap, run kube-proxy --cleanup, set k8sServiceHost.
  • Cilium breaks open Service connections in the switch. Calico keeps TCP flows, drops VXLAN ones, wants ipvs gone first. k3s : --disable-kube-proxy, or --kube-proxy-arg=proxy-mode=nftables.
  • Check : cilium-dbg status --verbose | grep -A12 'KubeProxyReplacement Details'. CNI side : Pod Networking.
In short : Slice conditions and hints pick the reachable Pods, health checks steer LBs to nodes with one, and eBPF can do it all at connect().

Networking internals

Cluster DNS in Depth

1.Two halves

The kubelet writes each Pod's resolver file once, at sandbox creation (GetPodDNS), from clusterDNS, clusterDomain and resolvConf. CoreDNS (Deployment coredns, label k8s-app=kube-dns) answers from API watches behind the Service still named kube-dns : the 10th Service IP on kubeadm (10.96.0.10), 10.43.0.10 on k3s.

Pod (ns shop) getaddrinfo() /etc/resolv.conf nameserver 10.96.0.10, options ndots:5 search shop.svc.cluster.local svc.cluster.local cluster.local lan A, AAAA : one query per suffix 10.96.0.10:53 Service kube-dns kube-proxy DNAT CoreDNS Pod Deployment coredns kubernetes plugin cluster.local, *.arpa cluster names, answered from informers forward . /etc/resolv.conf everything else node's upstream resolvers CoreDNS runs dnsPolicy Default

2.What the kubelet writes

  • ClusterFirst options are exactly ndots:5. Node edns0, timeout, attempts are dropped.
  • hostNetwork : silently Default (kind), use ClusterFirstWithHostNet. No clusterDNS : Default + MissingClusterDNS.
  • dnsConfig is appended, options merged by name (Pod wins). None : dnsConfig alone.
dnsPolicyserversearchopts
ClusterFirstclusterDNS3 cluster + nodendots:5
+ hostNetworknode filenode filenode
...WithHostNetclusterDNS3 cluster + nodendots:5
DefaultresolvConfsame filefile

Over 3 servers, 32 searches or 2048 chars : the API rejects, the kubelet trims the merged file (DNSConfigForming). See it : kubectl exec <pod> -- cat /etc/resolv.conf. Basics : Namespaces.

3.The ndots:5 walk

  • Under 5 dots : all suffixes, then as typed. A 4-dot FQDN too.
  • musl always sends A and AAAA.
  • glibc drops AAAA only with no IPv6 but ::1. Link-local fe80:: counts : 8 queries for 3 suffixes on kind.
  • Plan for 2 x (suffixes + 1) queries.
api.stripe.com.shop.svc.cluster.local NXDOMAIN, A+AAAA api.stripe.com.svc.cluster.local NXDOMAIN, A+AAAA api.stripe.com.cluster.local NXDOMAIN, A+AAAA api.stripe.com.lan NXDOMAIN, A+AAAA api.stripe.com. NOERROR, A+AAAA 10 queries, 8 NXDOMAIN (example names, lab counts)

4.glibc vs musl

Behaviour (seen in the lab)glibcmusl (Alpine)
name with dots >= ndotsas typed, then searchas typed only
NODATA or SERVFAIL on a suffixkeeps walkingstops and fails
. in the search listas typed at that spotskipped
search line over about 255 charsusedwhole line ignored
TCP fallback on truncationyesmusl 1.2.4+ (Alpine 3.18+)

The docs' dnsutils Pod runs musl 1.1.24 (no TCP fallback). dig and nslookup use their own resolver, and dig skips the search list without +search. Test in the app's image : kubectl exec <pod> -- getent ahosts <name>.

5.Cutting the cost

  • Trailing dot : one name, any libc.
  • ndots: "2" : names with 2+ dots go upstream first (seen on kind). musl never searches them.
  • autopath : 1 round trip. Needs pods verified, Pod IP.
  • Searches may hold _ and a lone . (GA v1.34).
Per-Pod tuning (ran on kind)
dnsConfig:
  options:
  - { name: ndots, value: "2" }
  - name: single-request-reopen
  - { name: timeout, value: "2" }
# options timeout:2 ndots:2 single-request-reopen
In short : The kubelet writes a resolv.conf once per Pod, and ndots:5 walks every suffix, often twice per name. glibc and musl differ : test with the app's libc, and cut the walk with trailing dots or lower ndots.

1.Records beyond the basics

Query (kind)Answer
TXT dns-version.<zone>"1.1.0", the spec version
AAAA, IPv4-only ServiceNOERROR, no answer
dig -x 10.96.0.10kube-dns.kube-system.svc.cluster.local.
headless, none readyNXDOMAIN
ExternalNameCNAME + the target's A records

Basic records : Services, StatefulSets.

Per-Pod names
  • hostname + subdomain = headless Service, same ns.
  • Ready Pods only. publishNotReadyAddresses adds unready and terminating ones.
  • setHostnameAsFQDN : 64 chars max. hostnameOverride (GA v1.37) : hostname only, not DNS.

2.pods modes, and a captured name

  • disabled (default) : a-b-c-d.<ns>.pod NXDOMAIN.
  • insecure (kubeadm, k3s) : no Pod check, 1-2-3-4.default.pod gave 1.2.3.4. No *.pod certs.
  • verified : Pod must exist. Watches all Pods (~4x memory, old data).
  • Capture : Service github in namespace com answers github.com (derived). Guard namespaces.

3.A query through CoreDNS

  • Order is compiled (plugin.cfg), not the Corefile's.
  • ttl (plugin 5, kubeadm 30) is also the negative TTL.
  • k3s caches cluster names : new headless names NXDOMAIN up to 30s.
  • Over max_concurrent : REFUSED. SERVFAIL retried only with failover.
query server block .:53 cache kubeadm skips cluster.local hosts k3s NodeHosts kubernetes cluster.local, in-addr, ip6.arpa answer or NXDOMAIN in zone fallthrough, other zone forward . /etc/resolv.conf random policy, max 15 upstreams upstream node's resolvers loadbalance shuffles A / AAAA on the way out. The order is compiled, so moving lines does nothing.

4.The default Corefile

.:53 {                   # kubeadm v1.37.1
  health {               # k3s: plain health
    lameduck 5s
  }
  kubernetes cluster.local in-addr.arpa ip6.arpa {
    pods insecure
    fallthrough in-addr.arpa ip6.arpa
    ttl 30               # k3s: no ttl (5)
  }
  forward . /etc/resolv.conf { max_concurrent 1000 }
  cache 30 {             # k3s: plain cache 30
    disable success cluster.local
    disable denial cluster.local
  }
}  # + errors ready prometheus :9153 loop reload loadbalance
Lifecycle plugins
  • health:8080. lameduck : stays 200, exit delayed 5s
  • ready:8181. 503 until plugins sync, Shutting down in lameduck
  • loopHINFO self-probe for 30s
  • reloadSHA512 check every 30s +/- 15s

The debug page's Corefile fails : 1.14.7 rejects upstream. k3s adds hosts NodeHosts and import *.override. kubeadm : 2 replicas, CoreDNS v1.14.6. k3s v1.37.1+k3s1 : 1 replica, 1.14.7.

5.Changing it safely

  • kubeadm : edit ConfigMap coredns, allow 2 minutes (kind : 59s and 84s).
  • A bad config keeps the old one, but restarts crash. Check : coredns -conf ./Corefile.
  • Now : rollout restart deploy/coredns (kube-system).
  • Stub domain : own block, or forward consul.local first.
k3s : coredns-custom
  • coredns is rewritten every start. *.server adds blocks.
  • A 2nd hosts, cache, errors, prometheus fails. Lameduck health is fine.
kind: ConfigMap  # coredns-custom, kube-system
data:
  example-com.override: |  # inside .:53
    forward example.com 10.0.0.1
In short : CoreDNS answers from API watches in a fixed plugin order. kubeadm and k3s differ on cache, TTL, lameduck and replicas : change the right ConfigMap and allow two minutes.

1.Enough CoreDNS

  • cluster-proportional-autoscaler (not in kubeadm) : max(ceil(cores/256), ceil(nodes/16)) with the docs' values. 40 nodes x 8 cores : 3.
  • Re-reads its ConfigMap every 10s. Tag :v1.11.0, not :1.11.0.
  • Rollouts : 1 replica without lameduck lost 1 of 450 queries per rollout (kind). With lameduck 5s : none.
  • k3s ships 1 replica, no lameduck, no PodDisruptionBudget.

2.NodeLocal DNSCache

  • hostNetwork DaemonSet node-local-dns. Its dummy nodelocaldns holds 169.254.20.10 and the kube-dns IP.
  • 12 rules per IP : 8 raw NOTRACK (6 DNS, 2 health port), 4 filter ACCEPT.
  • External names skip CoreDNS : no stub domains, rewrites or k3s overrides.
  • Create kube-dns-upstream first. An OOM kill leaves the rules : no DNS there.
  • Moved to kubernetes-sigs/node-local-dns. v1.37.1 pins 1.26.8.
Pod UDP to 10.96.0.10:53 raw PREROUTING : NOTRACK no conntrack, so no DNAT node-cache (hostNetwork) nodelocaldns : 169.254.20.10 + 10.96.0.10 kube-dns-upstream cluster.local : cache, then TCP node's resolvers "." : CoreDNS skipped kube-proxy DNAT CoreDNS

3.NodeLocal by dataplane, and DNS egress

ModeHow Pods reach itKubelet changeIf the cache dies
kube-proxy iptablesNOTRACK on the bound kube-dns IPnoneclean exit falls back
ipvs (deprecated)Pods use 169.254.20.10clusterDNS: [169.254.20.10]no fallback
nftablesundocumented; iptables install worked (1 kind node)nonefell back on clean exit
CiliumCiliumLocalRedirectPolicy on kube-dnsnonekube-dns backends
Default-deny egress : packets keep the kube-dns ClusterIP, so allowing CoreDNS Pods (or 169.254.20.10/32) stopped matching on kindnet. ipBlock: 10.96.0.10/32 fixed it. Test your CNI. See Network Policy.

4.Two failures up close

  • 5s race : parallel A and AAAA both pass DNAT unconfirmed, one is dropped at confirm.
  • Fixed by kernels v4.19 to v5.9. IPVS is hit too. See conntrack -S insert_failed. Basics : Service Dataplane.
  • Loop : node 127.0.0.53 makes CoreDNS forward to itself. kubeadm fixes resolvConf only if unset (leaked into kubelet-config before v1.37.1).
node resolv.conf nameserver 127.0.0.53 kubelet resolvConf copied into Default Pods CoreDNS dnsPolicy Default forward . /etc/resolv.conf 127.0.0.53 inside CoreDNS's netns loop plugin : exit CrashLoopBackOff

5.Other failures, and the playbook

SymptomCauseFix
External SERVFAIL, or REFUSED under loadupstreams down, or max_concurrentfix upstreams; scale
New headless name NXDOMAIN for ~30scluster domain cacheddisable success/denial
Cluster names SERVFAILno EndpointSlice RBAC, or first 5sdescribe clusterrole system:coredns
One node has no DNS after a NodeLocal OOMrules left behindraise its memory limit

Playbook : Pod resolv.conf, events, getent ahosts, CoreDNS Pods, EndpointSlice and logs, system:coredns, log on (2 min), metrics, conntrack -S.

Metrics : coredns_dns_responses_total{rcode}, coredns_proxy_request_duration_seconds{proxy_name="forward"} (old coredns_forward_request* gone since 1.11), hit ratio = cache_hits_total / cache_requests_total.

In short : Scale CoreDNS with replicas and lameduck, add NodeLocal DNSCache to skip conntrack (and fix DNS egress policies), and debug from the Pod outward with the app's libc.

Networking internals

Network Policy in Depth

1.A request, not a firewall

  • No status : the API server defaults, validates, stores. Alpha status left in v1.28.
  • Not in conformance : the 49 enforcement e2e tests are feature-tagged.
  • No enforcer (plain flannel) : every policy is stored and silently ignored.
  • Agents : kube-router (k3s), Calico Felix, cilium-agent, antrea-agent, kube-network-policies.
kubectl apply kube-apiserver defaults, validates etcd no status written back watch policy agent on every node per-node rules iptables / nftables eBPF / OVS programs Pod A Service DNAT ClusterIP:80 -> Pod B:8080 egress check A's node rewritten before any check ingress check B's node wire Pod B policy sees Pod B's IP and port 8080, never the ClusterIP

2.Defaulting traps

  • Egress-only, no policyTypes : all ingress is cut too.
  • kubectl describe netpol : "isolated for ingress".
  • No podSelector (optional, v1.34+) : every Pod.

v1.37. Basics : Namespaces and Ingress, Architecture.

You wroteStored (seen on v1.37.0)
no policyTypes, no rules[Ingress]
egress rules only[Ingress, Egress]
no podSelectorpodSelector: {}
a port, no protocolprotocol: TCP

3.After DNAT : Pod ports

  • Calico, Cilium, Antrea, kube-network-policies check after Service DNAT. Docs are silent.
  • Name the Pod's targetPort or port name, and the backend Pods.
  • Named ports resolve per destination Pod. endPort needs a numeric port.
  • Only TCP, UDP, SCTP. ICMP is rejected.
# Service api : port 80 to containerPort "http" (8080)
ports: [{name: http, port: 80, targetPort: http}]
# NetworkPolicy api-from-frontend
ingress:
- from: [{podSelector: {matchLabels: {app: frontend}}}]
  ports: [{protocol: TCP, port: http}]
# or 8080. 'port: 80' never matches

4.Where NAT and IP rules blur

  • ipBlock is for external CIDRs. Pod IPs in it : e2e expects a match, as kind does. Cilium ignores them by default.
  • Strict CIDRs since v1.36 (Beta) : 10.0.0.5/24, leading zeros, IPv4-mapped IPv6 fail on create.
  • Stored values ratchet with a warning, so a fresh copy fails. Test with kubectl apply --dry-run=server --validate=strict.
TrafficThe policy sees
NodePort, LB with eTP=Clustera node IP (SNAT). Use eTP=Local or allow node IPs
probes, local host processesthe local node : ingress allowed (kind : root's sockets only)
a hostNetwork Podthe node IP : undefined
NodeLocal DNSCachehostNetwork : only an ipBlock matches

See Service Dataplane, Cluster DNS, Pod Security.

5.The edges the API leaves open

EdgeThe API saysImplementations
ICMPundefinedkube-router accepts echo, unreachable, time-exceeded. Calico can name ICMP. Cilium blocks unrelated ICMP under L4 rules
Pod to own nodegodoc : ingress from it. Docs : "to and from"kube-network-policies enforces Pod-to-host. Felix default DROP, calico.yaml ACCEPT
Pod to itself via its Servicecannot block itselfthe upstream e2e test expects it blocked
  • Undefined too : open connections after a change, new Pods before programming (windows). Not expressible : explicit deny, logging, Service or FQDN peers, L7, a cluster-wide default.
In short : A NetworkPolicy is stored; a per-node agent enforces it after DNAT. Name Pods and Pod ports, mind policyTypes defaulting, and treat ICMP and node traffic as undefined.

1.ClusterNetworkPolicy (v1alpha2)

  • network-policy-api v0.2.0 (2026-04-21) : one cluster-scoped CRD, cnp.
  • Replaces ANP, BANP. Allow is now Accept, ports now protocols. v1beta1 PR #411 open.
  • tier Admin or Baseline. priority 0 to 1000, lower wins, ties ambiguous.
  • tcp: {} is rejected. Any port : range: {start: 1, end: 65535}.
PeerIngressEgressChan.
namespacesyesyesstd
podsyesyesstd
networksnoyes, hits Pods toostd
nodes, domain­Namesnoyesexp
  • Subject : namespaces or pods, not hostNetwork. 25 rules max.
  • Install from the v0.2.0 tag : main changed the schema.
  • kubectl get cnp : Tier, Priority. (channels)

2.Evaluation order

new connection Admin CNPs priority, then rule order Accept : allow, final Deny : drop, final Pass or no match NetworkPolicy selects the Pod for this direction ? its rules decide yes no Baseline CNPs Accept / Deny decide no match : allow
  • Accept, Deny are final. Pass skips the rest of Admin.
  • Accept covers replies and related ICMP. An egress Accept keeps the far end's ingress rules.
  • DNS guarantee : an Admin Accept to k8s-app: kube-dns Pods, port 53. Upstream user-story-2 uses app: kube-dns, a wrong label.

3.Who implements what (checked 2026-10-07)

ImplementationCNP v1alpha2Notes
kube-network-policies v1.1.2yes, v1.0.0+reference. nftables + NFQUEUE, fails open. kind v0.33.0 embeds an older copy : no CNP
Calico v3.33.0yes, v3.32+tiers kube-admin, kube-baseline. ANP, BANP dropped. A bad peer fails the rule
Cilium 1.20.2opt-ink8sClusterNetworkPolicy.enabled. Conformance needs [pods, nodes] CIDR match
Antrea v2.7.0alpha gate, offgate ClusterNetworkPolicy. v1alpha1 deprecated
OVN-K8s, Kube-OVNOVN-K8s v1alpha1Kube-OVN : pre-v0.2.0 build, v0.2.0 on master

The upstream list (21 April 2026) omits Cilium 1.20 and Antrea 2.7. No conformance reports from them or Calico.

4.k3s : kube-router, one packet

  • k3s v1.37.1+k3s1 : fork v2.6.3-k3s1, iptables + ipset, no CNP.
  • --disable-network-policy leaves its rules behind.
  • Unmarked : NFLOG group 100, then REJECT. Ping works.
  • New Pods fail open : no --netpol-default-deny.
  • Chains : iptables -t filter -S | grep KUBE-.
FORWARD KUBE-ROUTER-FORWARD RETURN ClusterIP, NodePort (pre-DNAT) KUBE-POD-FW-<hash> accept node-local sources KUBE-NWPLCY-COMMON drop INVALID, accept ESTABLISHED, ICMP KUBE-NWPLCY-<hash> : ipset match, mark 0x10000 mark 0x20000 : ACCEPT no mark : NFLOG 100, REJECT

5.Calico and Cilium

Calico
  • Tiers by order. NetworkPolicies sit in default as knp.default.<name>.
  • Allow, Deny, Pass, Log. Follow Log with Allow, or the tier's end drop wins.
  • cali-pi- / cali-po- chains, or a tc hook in eBPF mode.
Cilium
  • Labels become identities. Per endpoint : a BPF map of (identity, port, proto), 16k.
  • ingressDeny / egressDeny beat every allow. toFQDNs needs a rules.dns allow.
  • Dump : cilium-dbg bpf policy get <ep-id>.

Cilium 1.20 may balance NodePort at the client (Service Dataplane). Antrea has two CNP kinds.

In short : CNP adds Admin and Baseline tiers around NetworkPolicy. Each dataplane enforces it its own way : an iptables REJECT on k3s, a missing eBPF map entry on Cilium.

1.Stored is not enforced

  • Only a probe seen to fail proves a deny.
  • Counts lie : KUBE-POD-FW can read 0 on a node without iptables; nft showed 70.
  • An untested deny is a guess : probe it from a Pod that should fail.

The official task expects wget: download timed out; k3s refuses at once.

DataplaneClient seesagnhost
kube-router, k3srefused at onceREFUSED
Calico, kube-network-policiestimeoutTIMEOUT
Cilium L3/L4timeout (opt-in ICMP on egress)TIMEOUT
Cilium L7HTTP 403, DNS REFUSEDconnects

2.A per-node probe for your cluster

  • Server Pods on each node. Default-deny ingress, only label allowed passes.
  • A client DaemonSet probes from every node : all must fail.
  • Control 1 : one allowed client must pass, or the network is just broken.
  • Control 2 : agent off on one node, FAIL there only.
  • No ping. Self-health on 127.0.0.1 : own-Service calls may be blocked.
client, node 1 client, node 2 client, node 3 server Service server Pods one per node must fail control 1 : allowed label must pass on every node control 2 : agent off on node 2 FAIL there, and only there
# a sketch, not run. Namespace np-probe : Service server, client DaemonSet (agnhost)
for p in $(kubectl -n np-probe get pods -l app=client -o name); do
  n=$(kubectl -n np-probe get "$p" -o jsonpath='{.spec.nodeName}')
  if kubectl -n np-probe exec "$p" -- /agnhost connect server.np-probe.svc.cluster.local:80 \
       --timeout=3s --protocol=tcp 2>err.txt
  then echo "FAIL $n: denied connection worked"; else echo "ok $n: $(cat err.txt)"; fi
done   # ok lines read TIMEOUT or REFUSED

Upstream e2e diffs an expected x/y/z truth table against agnhost results. No expected X, no proof.

3.Watching verdicts

DataplaneWhereRead it with
k3s kube-routerNFLOG 100. Empty podSelector not logged (k3s #8008)tcpdump -ni nflog:100
CiliumHubble (relay off). 133 no allow, 181 deny rulehubble observe --verdict DROPPED
CalicoLog rules : calico-packet. Whisker previewjournalctl -k | grep calico-packet

Antrea : np.log. kube-network-policies : -v=2 (not on kind : no flags). policy-assistant simulates from YAML (v1, v1alpha1 only).

4.Start-up and change windows

ImplementationNew Pod before programmingOpen connection after a new deny
kube-router, k3sfails open (no --netpol-default-deny)survives
Calicoopt-in hold, linuxPolicySetupTimeoutSeconds (default 0)survives until reconnect
Ciliumreserved:init, allowed unless selectednot documented
kube-network-policiesNRI learns Pod IPs earlycut (--strict-mode, default)
kind (its built-in copy)open until its node lists it (0.7 s)survives (no strict mode)

The docs suggest an init container that waits for its dependencies, and advise against changing policies under live connections.

5.Troubleshooting checklist

  1. Anything enforcing ? flannel alone is not.
  2. Pod selected ? kubectl describe netpol.
  3. Both ends allow ? DNS (Cluster DNS) ?
  1. Pod port named ? Source NATed ?
  2. Higher tier ? kubectl get cnp, Calico tiers.
  3. Read the verdict, then probe every node.
In short : Prove every deny with a probe that must fail on every node, read the verdict where your dataplane logs it, and expect start-up and change windows to differ by plugin.

Storage internals

CSI Drivers and the Volume Lifecycle

1.What a CSI driver is

  • A gRPC server on a UNIX socket (CSI_ENDPOINT, ends in .sock).
  • Identity (always), Controller (provision, attach, expand, snapshot), Node (stage, publish, stats). Also GroupController and alpha SnapshotMetadata.
  • Capabilities switch steps off, down to NodePublish and NodeUnpublish only. No capability says "block".
CapabilityWithout it
CREATE_DELETE_VOLUMEprovisioner refuses
PUBLISH_UNPUBLISH_VOLUME"trivial handler"
STAGE_UNSTAGE_VOLUMEkubelet skips NodeStage
VOLUME_MOUNT_GROUPkubelet applies fsGroup
EXPAND, MODIFY, CLONE ...no resize, VAC, copies

Plus plugin CONTROLLER_SERVICE to provision. v1.37 vendors a commit just before spec v1.13.0.

2.Two halves, two callers

  • Controller plugin : Deployment or StatefulSet, leader election. More replicas add availability, not throughput. Sidecars share the socket in an emptyDir.
  • Node plugin : privileged DaemonSet, Bidirectional propagation on /var/lib/kubelet/pods and .../plugins (staging). Privileged Pod Security level.

3.The sidecars

SidecarCallsNewest
external-provisionerCreate / Deletev6.3.0
external-attacherPublish / Unpublishv4.13.0
external-resizerExpand / Modifyv2.3.0
csi-snapshotterCreateSnapshotv8.6.0
external-health-monitorVolumeHealthv0.19.0
node-driver-registrarGetPluginInfo onlyv2.18.0
livenessprobeProbev2.20.0
PVC VolumeAttachment PVC size / VAC SnapshotContent watch : provisionerCreateVolume attacherControllerPublish resizerControllerExpand snapshotterCreateSnapshot csi.sock in a shared emptyDir livenessprobe driver (controller) storage API kubelet registrar-reg.sock GetInfo NodeStageNodePublish csi.sock (hostPath) driver (node) : mounts node DaemonSet Pod

Each controller sidecar is a reconcile loop with leader election. Checked 2026-10-07 : provisioner --timeout is 10s in code (README : 15s), set it for slow backends. Resizer image tops out at v2.2.1. The health monitor exits unless the driver advertises health.

4.Registration on a node

  • Registrar : GetPluginInfo only, then a socket in plugins_registry/. Kubelet (inotify) : GetInfo, NodeGetInfo (2m), then the csi.volume.kubernetes.io/nodeid annotation.
  • --kubelet-registration-path = host path of the driver socket. Wrong : not found in the list of registered....
  • Same watcher as DRA, no maxSurge upgrades (all send "1.0.0").
registrar plugins_registry/ <driver>-reg.sock 2. creates 1. GetPluginInfo kubelet 3. GetInfo 6. Notify status driver (node) 4. NodeGetInfo 5. writes CSINode Node labels nodeid annot. lab : hostpath.csi.k8s.io key topology.hostpath.csi/node

5.Three API objects

  • CSIDriver : 11 fields in v1.37, two immutable (path misspelled spec.attachedRequired). No object is not neutral : attach stays on, no Pod info, tokens or inline.
  • CSINode : by the kubelet, never the driver. nodeID, topologyKeys (keys only), allocatable.count.
  • VolumeAttachment : csi-<sha256(handle+driver+node)>. spec immutable, status only by the attacher.
CSIDriver fieldDefault
attachRequiredtrue, immutable
volumeLifecycleModes[Persistent], immutable
podInfoOnMountfalse
tokenRequestsnone
requiresRepublishfalse
serviceAccountTokenInSecretsunset

The other five : Volume Topology, Volume Operations.

In short : Sidecars turn PVCs, VolumeAttachments and snapshots into calls on the controller half; the kubelet calls the node half itself once the registrar announces its socket.

1.Provision : PVC to PV

  • PV controller : ExternalProvisioning. WaitForFirstConsumer waits for selected-node.
  • external-provisioner : CreateVolume pvc-<UID>, minus reserved keys; writes the PV, finalizer.
  • No secret : provisioning fails, deletion does not.
Reserved keyBecomesRead by
fstypePV fsTypekubelet
provisioner-secret-*Create, Deleteprovisioner
*-publish-, node-stage-*PV *SecretRefattacher, kubelet
*-expand-, *-modify-*refs, annotationresizer, kubelet

Prefix csi.storage.k8s.io/. Secrets : -name / -namespace, templates like ${pvc.namespace}.

2.Attach : the VA handshake

  • The attach-detach controller (ADC) creates VA csi-<sha256>, waits 2m for status.attached.
  • external-attacher : finalizer, node ID from CSINode, ControllerPublishVolume, attachmentMetadata.
  • Immutable. v1.36+ : flipping to false cleans old VAs.
attachRequiredtrue (default)false
VolumeAttachmentper volume per nodenever
driver can publishControllerPublishskipped (wrong)
driver cannottrivial handler, no RPCnothing
attacher missingtimed out after 2mfine
publish_contextVA attachmentMetadatanil

3.The whole lifecycle

control plane driver PV controller external-provisioner writes PV {driver, volumeHandle} attach-detach controller 2. VA csi-<sha256> external-attacher writes status.attached; ADC : volumesAttached controller CreateVolume, Publish 1. Create Publish driver node volumesAttached : ADC volumesInUse : kubelet container runtime 5. bind mount kubelet node Stage, Publish 3. NodeStage 4. NodePublish stage once per node, publish per Pod teardown : NodeUnpublish, NodeUnstage, volumesInUse cleared, VA deleted, ControllerUnpublish, DeleteVolume

4.Stage and publish on the node

  • MountDevice : globalmount + vol_data.json, NodeStage. No staging : directory only.
  • SetUp : Pod directory + vol_data.json, driver makes mount, then fsGroup.
  • Canceled, DeadlineExceeded, Unavailable, ResourceExhausted, Aborted : uncertain, kept. Others : removed.
Path (/var/lib/kubelet/...)Written by
plugins_registry/<driver>-reg.sockregistrar
plugins/<driver>/csi.sockdriver
plugins/kubernetes.io/csi/<driver>/<sha256(handle)>/globalmountkubelet
pods/<podUID>/volumes/kubernetes.io~csi/<pv>/mountkubelet, driver

Reconcile every 100ms. Only the first access mode is sent (RWO as 7 if advertised). Before v1.24 : .../csi/pv/<pv>/globalmount.

5.Teardown and kubelet restart

  • Unpublish, Unstage (last Pod on the node), volumesInUse cleared (0 to 10s), VA deleted, ControllerUnpublish (observed : 34.327, 34.428, 35.118). Why : Volume Operations.
  • DeleteVolume is retried : idempotent. A VA left : VolumeFailedDelete ... still attached. Reconstruction (GA v1.30) : a restarted kubelet rebuilds from vol_data.json and re-issues Stage and Publish.

6.Commands

CommandShows
VA=csi-$(printf '%s%s%s' "$HANDLE" "$DRIVER" "$NODE" | sha256sum | cut -d' ' -f1)VA name (a hash)
printf '%s' "$HANDLE" | sha256sumStaging dir (a hash)
kubectl get volumeattachment <va> -o jsonpath='{.status.attachError}{"\n"}{.metadata.finalizers}'Error, finalizers
findmnt -o TARGET,SOURCE,FSTYPE,PROPAGATION | grep kubernetes.ioReal mounts (hostpath : no globalmount)
journalctl -u kubelet | grep -E 'MountVolume.(WaitForAttach|MountDevice|SetUp)|UnmountDevice'Node steps
In short : Provision makes the PV, attach is a VolumeAttachment the attacher answers, stage mounts once per node, publish once per Pod, and teardown runs backwards, gated by volumesInUse.

1.Volumes that skip steps

Raw block : volumeMode: Block, devicePath, no fsType; paths under .../csi/volumeDevices/, held by a loop device.

CSI persistentCSI inlineGeneric ephemeralImage volumeIn-tree, migrated
API fieldPV spec.csivolumes[].csivolumes[].ephemeralvolumes[].imageold PV fields
ParametersadminPod authoradminPod authoradmin
CSI callsallPublish onlyallnoneall, translated
VAif attachRequiredneverif attachRequiredneverif needed
Capacity-awareyesnoyesn/ayes

2.Inline, generic and image volumes

  • Inline : CSIDriver lists Ephemeral (Volumes). volume_id = csi-<sha256(podUID+name)>. podInfoOnMount : five keys (pod name, namespace, uid, serviceAccount.name, ephemeral) to NodePublish only.
  • Pod authors set volumeAttributes; restricted Pod Security allows csi. Guard with admission.
  • Generic : PVC <pod>-<volume>, no PVC create right needed. Names collide : pod is not owner.
csi kubelet NodePublish only driver makes, deletes ephemeral ephemeral ctrl PVC <pod>-<vol> owner Pod, full cycle image kubelet, runtime read-only, no CSI PullImage via CRI

3.Credentials into drivers

  • tokenRequests : Pod-bound token per audience, 600s minimum. Refresh : RBAC; vs projected.
  • Node audience restriction (beta, on) : only audiences used on that node.
  • requiresRepublish : republish on pod syncs (1m to 1.5m, labs 17s to 80s), not 0.1s. Contents only.
Tokens involume_contextsecrets
Whendefaultfield true
Logsprintedstripped
API warningyes, if unsetno
Real leaksCVE-2023-2878, CVE-2024-3744

serviceAccountTokenInSecrets (GA v1.36) without tokenRequests, even false : rejected. Real drivers : SPIFFE CSI, Secrets Store CSI (ESO).

4.Migration and what comes next

  • Ten in-tree plugins were removed (1.26 to 1.36). Seven types need their CSI driver (Volumes). CephFS, RBD, GlusterFS have no translator : move by hand first. FlexVolume : deprecated, still there.
  • Horizon : all-in-one sidecars (KEP-4958, alpha target v1.38); COSI v1alpha2 only on main (tag v0.2.2).

5.Debugging and commands

Attach and mount symptoms : Volume Operations. -v=5 driver logs are sanitized (not hostpath's gRPCCall).

SymptomLook atCause and fix
PVC Pending, only ExternalProvisioningprovisioner, leaderprovisioner down, or another driver named
does not support snapshot restoredataSourceno CREATE_DELETE_SNAPSHOT (clones: CLONE_VOLUME), no RPC
timed out waiting for external-attacherVAs, attacher podsdeploy one, or recreate with attachRequired false
not found in the list of registeredregistrar logsno node plugin, wrong path
VA CSINode ... does not contain driverkubectl get csinodenot registered on that node
volume mode "Ephemeral" not supportedvolumeLifecycleModesinline volume, Persistent-only driver
CommandShows
kubectl logs <controller-pod> -c csi-attacherTrivial sync started
kubectl logs <node-pod> -c node-driver-registrarGetInfo, NotifyRegistrationStatus
ls -l /var/lib/kubelet/plugins_registry/ /var/lib/kubelet/plugins/Both sockets
find /var/lib/kubelet -name vol_data.json -exec sh -c 'echo {}; cat {}' \;Kubelet bookkeeping
kubectl get --raw /api/v1/nodes/<n>/proxy/metrics | grep -E '^(csi_operations|reconstruct_volume)'Kubelet CSI metrics
kubectl delete csidriver <d> && kubectl apply -f csidriver.yamlOnly way to change immutables
In short : Variants skip or swap lifecycle steps, tokens belong in the secrets field, and a stuck volume's message names the part to read : provisioner, attacher or registrar.

Storage internals

Volume Topology and Scheduling

1.Where a volume can live

  • Topology = the PV spec.nodeAffinity. Terms are ORed, expressions in a term ANDed.
  • CSINode topologyKeys are key names. The values are Node labels.
  • Immediate is the default. The mode is immutable.
ModeImmediateWFFC
Volume madePVC createdPod scheduled
Zone fromprovisionerscheduler's node
Capacityneverif storageCapacity
Pod, unboundPendingnormal
Riskfull zonestale selected-node

2.VolumeBinding in the framework

  • A missing class or a pre-bound claim counts as immediate.
  • bindTimeoutSeconds: 0 fails at once, not "no wait". The waiting node shows as NOMINATED NODE.
  • UnschedulableAndUnresolvable : evicting cannot help, so no preemption (Advanced Scheduling).
PreFilter : bound / delayed / immediate Filter per node : PV affinity, smallest static PV, allowedTopologies + capacity Score : free capacity (weight 1) Reserve : assume, cache only PreBind : selected-node, poll 1s, 600s Pod bound Unschedulable AndUnresolvable = no preemption

3.The handshake, end to end

PVC events : WaitForFirstConsumer, WaitForPodScheduled, ExternalProvisioning. Basics : Volumes and Storage. A nodeName pin skips the scheduler : a bound PV fails MountVolume.NodeAffinity check failed, a WFFC claim never binds.

kubelet node-b attach, mount kube-scheduler VolumeBinding PV nodeAffinity PVC data-0 WFFC class CSINode node-b topologyKeys external-provisioner Node node-b labels (values) CSI driver 1 selected-node 2 watch 3 keys 3 values 4 CreateVolume (requisite, preferred) 5 accessible_topology [{zone: b}] 6 zone In [b] 7 bind 8 Pod bound 9 affinity re-check

4.What CreateVolume receives

  • Plain WFFC : requisite is the aggregated topology, every Node with the key labels, driver or not.
  • allowedTopologies keys must be driver keys, or ... is not in requisite.
  • Node-local storage : --strict-topology.
Node in zone brequisitepreferred
WFFC, plaina, b, cb, c, a
WFFC, strictbb
WFFC, allowed a, ba, bb, a
Immediatea, b, chash(set) + ordinal
Immediate, topology offnonenone

5.Provisioning fails on the node

  • Only RESOURCE_EXHAUSTED (or an uncached CSINode NotFound) frees the Pod.
  • v6.0.0 and v6.1.0 could retry in the wrong zone : use v6.0.1, v6.1.1, v6.2.0+.
CreateVolumeProvisionerPod
RESOURCE_EXHAUSTEDdrops the annotationrescheduled
INVALID_ARGUMENT5m slow retrypinned
anything elsesame node, backoffpinned, 600s

6.Commands

CommandShows
kubectl get pvc <pvc> -o jsonpath='{.metadata.annotations.volume\.kubernetes\.io/selected-node}'Node picked (empty if static)
kubectl get pod <pod> -o jsonpath='{.status.conditions[?(@.type=="PodScheduled")].message}'Current verdict (events lag)
kubectl get events --field-selector involvedObject.name=<pvc>PVC events, ProvisioningFailed
kubectl describe pv <pv>Node Affinity, Term 0
In short : With WaitForFirstConsumer the scheduler picks the node and the driver's answer becomes the PV's nodeAffinity. With Immediate the volume picks first and the Pod must follow.

1.Storage capacity tracking

  • Opt in : CSIDriver storageCapacity: true and provisioner --enable-capacity. Objects land in the provisioner's namespace.
  • Only new volumes of WFFC classes are checked. No match : did not have enough free storage.
CSI driver external-provisioner GetCapacity per segment create / update, every 1m and after create, delete CSIStorageCapacity class, nodeTopology, capacity reads kube-scheduler Filter + Score

A hint only, no bookkeeping, unlike Dynamic Resource Allocation.

2.Why the numbers are only hints

  • Each claim checked alone, new volumes not subtracted, data up to 1m old.
  • Flag true before any object exists blocks every node : ship false, then flip.
  • Matched by class name only : write access to csistoragecapacities steers scheduling (RBAC).
FieldRuleUsed by
nodeTopologynil = none, {} = allFilter
storageClassNamename onlyFilter
capacityavailable_capacityFilter, Score
maximumVolumeSizeoptionalFilter first

The first two are immutable. Flag turned off : PVC Pods wait for the 5 min flush.

3.Storage scoring in v1.37

  • StorageCapacityScoring : beta and on.
  • Dynamic : most free space wins. Static : the node whose smallest fitting PV is largest wins.
  • Weight 1. Gate off with a shape set : kube-scheduler will not start.
Claim, space%Def.Inv.
10Gi, 100Gi109010
10Gi, 50Gi208020
50Gi, fit 200Gi2575-
50Gi, fit 100Gi5050-
50Gi, PVs 60 + 500Gi8317-

4.Attach slots per node

  • NodeVolumeLimits checks unique volumes against CSINode allocatable.count (in-tree EBSLimits and co : removed v1.32). Over : exceed max volume count, preemption may run.
  • nodeAllocatableUpdatePeriodSeconds (>= 10, GA v1.36) and external-attacher v4.10.0+ : a code 8 attach fails the Pod for a new one.
CSI driver max_volumes_per_node kubelet NodeGetInfo every period CSINode allocatable.count scheduler NodeVolumeLimits external-attacher ControllerPublish RESOURCE_ EXHAUSTED code 8 kubelet : refresh count Pod Failed VolumeAttachmentLimitExceeded controller makes a new Pod

5.Nodes without the driver

  • VolumeLimitScaling (beta, on) : the flag rejects driverless nodes : <driver> CSI driver is not installed on the node.
  • Only with CA --enable-csi-node-aware-scheduling=true (1.35.0+, default false) or no autoscaler. Karpenter : out of scope.
kind: CSIDriver   # name: ebs.csi.aws.com
spec:
  attachRequired: true
  nodeAllocatableUpdatePeriodSeconds: 60
  preventPodSchedulingIfMissing: true
# CA flag first, or no autoscaler

6.Commands

CommandShows
kubectl describe csinode <node>Node ID, count, topology keys
kubectl get csidriver <d> -o jsonpath='{.spec.preventPodSchedulingIfMissing} {.spec.nodeAllocatableUpdatePeriodSeconds}'The two fields get hides
kubectl get csistoragecapacities -A -o custom-columns='SC:.storageClassName,CAP:.capacity,MAX:.maximumVolumeSize,TOPO:.nodeTopology.matchLabels'What the scheduler matches
kubectl get volumeattachments -o custom-columns='NODE:.spec.nodeName,CODE:.status.attachError.errorCode'Code 8 = RESOURCE_EXHAUSTED
In short : The scheduler counts two scarce things per node : free space, from CSIStorageCapacity hints, and attach slots, from CSINode counts. v1.37 scores space by default.

1.Who says no, and can preemption help ?

  • Each node reports only its first failing filter, so one reason can hide another. Alerts on had volume node affinity conflict (renamed v1.33) are silent.
PluginReason, node(s) ...StatusPreempt ?
VolumeBindingdidn't match PersistentVolume's node affinity, didn't find available persistent volumes to bind, did not have enough free storage. PreFilter : pod has unbound immediate PersistentVolumeClaims...Unresolvablenever
VolumeZonehad no available volume zone (legacy PV zone labels only)...Unresolvablenever
NodeVolumeLimitsexceed max volume count, <driver> CSI driver is not installedUnschedulablepossible
VolumeRestrictionsRWOP claim already in-use by another podUnschedulablepossible

...Unresolvable = UnschedulableAndUnresolvable, also given by the last two for a missing PVC.

2.Reading one FailedScheduling message

Format : Advanced Scheduling. A PreFilter failure gives one reason. A trailing . not found in v1.37 is DRA noise.

0/3 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 2 node(s) had untolerated taint(s). preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. all nodes, control plane too first failing filter per node volume reasons never preempt

3.Node-local storage

  • local-static-provisioner v2.9.0 : one PV per disk, can drop a startup taint once disks are found.
  • Node-local CSI (TopoLVM, host-path) : strict topology, provisioner central or per node.
  • Node deleted : PV stays, Pod Pending. Same name reused : wrong disk.
local-staticNode-local CSI
Volumespre-made diskscarved per claim
Keykubernetes.io/hostnamedriver key
Classno-provisioner + WFFCdriver + WFFC
Scheduler seesPV sizesCSIStorageCapacity
v1.37 scorelargest best fitmost free space
Node lossnode-cleanup (opt-in)force-delete PV

4.A zone dies under a StatefulSet

  • A bound replica fails PV affinity everywhere, spare PVs or not. Dropping its PVC still hits zone spread unless nodeTaintsPolicy: Honor (GA v1.33) skips tainted nodes. Old Pod gone first : Volume Operations.
  • Or restore (Snapshots, Clones and Backup), or go regional, widen PV affinity (alpha MutablePVNodeAffinity).
created : db-0 zone-b, db-1 zone-a zone-a lost (NoExecute taint) db-1 Pending : PV node affinity delete PVC data-db-1 and Pod db-1 nodeTaintsPolicy unset : didn't match pod topology spread constraints Honor : binds spare PV in zone-b, runs observed, v1.37.0

5.Autoscalers and volume zones

  • cluster-autoscaler : group per zone. Per-node CSIStorageCapacity blocks scale-up : add one for template-node=true.
  • Karpenter : adds a bound PV's zone, picks one allowedTopologies zone at random, no region topology (Karpenter).
  • Attach limits : CA --enable-csi-node-aware-scheduling. Karpenter is out of KEP-5030's scope.

6.Commands

CommandShows
kubectl get events --field-selector involvedObject.name=<pod>,reason=FailedSchedulingReason history
kubectl get pv -o custom-columns='PV:.metadata.name,KEYS:.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].key'PV topology keys
kubectl delete pv <pv> --wait=false --grace-period=0 --force, then clear its finalizersDrop a dead node's PV
kubectl -n kube-system logs <scheduler> | grep 'Plugin scored node for pod'Scores (--v=10)
In short : Binding and zone rejections never preempt, attach-limit and RWOP ones can. Plan for node and zone loss : nodeTaintsPolicy: Honor, a node group per zone, a restore path.

Storage internals

Volume Operations and Failure Modes

1.Who writes what

  • Basics : Volumes and Storage. Gate : PersistentVolumeClaimResize admission (on). Its only dynamically provisioned pvc can be resized misleads : static PVs on an expandable class grow.
  • CSI : external-resizer does the controller step and status, the kubelet the node step.
user edits PVC spec size or VAC name kube-apiserver validation, Resize, Quota kcm expander ExternalExpanding, hands off watch external-resizer controller step calls CSI controller ControllerExpandVolume ControllerModifyVolume writes PVC status allocatedResources, statuses, conditions (+ PV size, VAC) NodeResizePending status.capacity kubelet online or at next mount calls CSI node plugin NodeExpandVolume

2.Expansion, step by step

  • Resizer : allocatedResources, condition Resizing, then ControllerExpandVolume (10s timeout). Node step needed : condition FileSystemResizePending.
  • Offline-only driver : FAILED_PRECONDITION while published. The resizer waits for zero Pods.
  • An unused PVC never finishes : Waiting for user to (re-)start a pod.
(no key) resizer ControllerResizeInProgress resizer NodeResizePending kubelet NodeResizeInProgress kubelet key removed, capacity updated ControllerResizeInfeasible NodeResizeInfeasible lower the request red = driver refused

3.Status values

Exactly five are accepted. The API reference still names ControllerResizeFailed and NodeResizeFailed : both are rejected.

ValueByMeans
ControllerResizeInProgressresizerstarted
ControllerResizeInfeasibleresizerdriver refused
NodeResizePendingresizergrow the FS
NodeResizeInProgresskubeletnode call running
NodeResizeInfeasiblekubeletnode refused

Infeasible : InvalidArgument, OutOfRange, NotFound. Timeouts : still running.

4.Undo a typo (GA v1.34)

  • Lower to anything strictly above status.capacity. Quota (max of spec and allocated) drops with it.
  • No help while ControllerResizeInProgress holds, or once only the node step failed.
  • Editing a PV's capacity cancels the resize. Raising the PVC to match then fakes a grown backend.
$ kubectl patch pvc data ... 1000Gi
1000Gi ControllerResizeInfeasible 10Gi
$ kubectl patch pvc data ... 100Gi  # ok
$ ... 10Gi  # Forbidden: can not be less

5.VolumeAttributesClass

  • Edit IOPS of a Bound PVC. GA v1.34, v1beta1 fails on v1.37.
  • Immutable classes, no default, vac-protection on each.
  • Cancel = old class back. get pvc : desired, get pv : applied.
  • A VAC quota checks create and spec. Status can pass hard.
ExpansionVAC modify
Spec fieldrequests.storageVAC name
Allowed onexpandable classMODIFY_VOLUME
ProgressallocatedResource*modifyVolumeStatus
ControllerControllerExpandControllerModify
Node stepNodeExpandVolumenone
Infeasible3 codesINVALID_ARGUMENT
Undolower, > capacityprevious class

6.Commands

CommandShows
kubectl get pvc data -o jsonpath='{.spec.resources.requests.storage}{" alloc="}{.status.allocatedResources.storage}{" state="}{.status.allocatedResourceStatuses.storage}{" cap="}{.status.capacity.storage}{"\n"}'State in one line. Read it first
kubectl events --for pvc/dataResize and VolumeModify events
kubectl get pvc data -o jsonpath='{.status.currentVolumeAttributesClassName}{" "}{.status.modifyVolumeStatus}{"\n"}'Applied class, change in flight
kubectl -n <ns> get resourcequota <q> -o jsonpath='{.status.used}{" / "}{.status.hard}{"\n"}'VAC quota over hard
In short : external-resizer runs both changes. Infeasible = driver refused : change the spec back.

1.Two hidden walks

  • A mount can walk the whole volume twice : kubelet fsGroup chown, runtime SELinux relabel. Both grow with file count.
  • VOLUME_MOUNT_GROUP driver : gets volume_mount_group, no walk, both policy fields ignored.
VOLUME_MOUNT_GROUP driver ? read-only mount ? CSIDriver fsGroupPolicy default : fsType, RWO(P) ? OnRootMismatch, root matches ? recursive walk no no yes no File driver does it yes skip the walk yes None no yes

2.What the fsGroup walk does

  • Lchown to the GID, OR in 0660 (0440 ro), setgid on dirs. Never removes bits.
  • OnRootMismatch skips if the root has the GID, 0770 (0550 ro) and setgid.
  • 30s : VolumePermissionChangeInProgress, then every 60s (CSI only).
  • RWX skipped by default : NFS needs File.
LeverWhereEffect
OnRootMismatchPodskip if root matches
Always, or unsetPodwalk every mount
fsGroupPolicy: NoneCSIDrivernever apply
fsGroupPolicy: FileCSIDriveralways, even RWX
VOLUME_MOUNT_GROUPdriverdriver does it
readOnly: truePodskipped

3.SELinux : a mount option

  • PV, seLinuxOptions.level, driver seLinuxMount: true : mount -o context. Else (secret, configMap ...) a full relabel.
  • RWOP and change-policy gates locked v1.36. SELinuxMount GA v1.37, unlocked : =false in v1.37 only.
MountOptionRecursive
Who labelskubelet, -o contextruntime walk
Costconstantgrows with files
One nodesame label onlyany labels
Needslabel, opt-innothing

4.Label conflicts from v1.37

  • Two labels (or privileged + not) on one volume and node : the 2nd Pod hangs. Fix : Recursive or one label.
  • Audit first : selinux-warning-controller (off, Controllers). Its metric names Pods : admin-only. Kubelet : volume_manager_selinux_*.
Pod A label s0:c1 mounted on node-1 -o context=s0:c1 Pod B, same node label s0:c2 ContainerCreating conflicting SELinux labels both Pods set policy Recursive both run runtime relabels, slower start selinux-warning-controller flags the pair before they meet

5.Access modes, enforced where

  • See Volumes, Topology. RWO is one node at attach unless the PV lists RWX, ROX or nothing.
  • recursiveReadOnly: IfPossible may fall back silently. Use Enabled.
ModeSchedulerA/D ctrlKubelet
RWOno checkone nodeothers may write
RWOPpreemptsone noderefuses a 2nd Pod
RWX, ROXnonemany nodesnone

6.Commands

CommandShows
kubectl get csidriver <d> -o jsonpath='{.spec}'fsGroupPolicy, seLinuxMount
kubectl patch csidriver <d> --type=merge -p '{"spec":{"fsGroupPolicy":"None","seLinuxMount":false}}'Mutable. attachRequired is not
kubectl get events -A --field-selector reason=SELinuxLabelConflictPotential conflicts (Normal)
kubectl get --raw /metrics | grep kubernetes_feature_enabledGate states (SELinuxMount 1)
kubectl get pod <p> -o jsonpath='{range .status.containerStatuses[*].volumeMounts[*]}{.name}{" rro="}{.recursiveReadOnly}{"\n"}{end}'Recursive read-only took effect ?
In short : OnRootMismatch, fsGroupPolicy and -o context skip the walks. v1.37 : one label per volume per node.

1.Attach and detach, the normal path

  • A second node is refused unless the PV lists RWX or ROX : Waiting for detach (was Multi-Attach error before v1.37.0, v1.36.4, v1.35.9). Reason is still FailedAttachVolume. In a rollout it is usually a race.
  • The 6m clock (hard-coded) starts once the Pod is gone or terminated. Call order : CSI Drivers.
Pod gone or terminated attach-detach ctrl not desired, 6m clock yes : wait in volumesInUse ? kubelet still mounted no VA deleted ControllerUnpublish, then finalizer off not Ready, over 6m force detach (timeout) out-of-service detach now red : skip volumesInUse, NodeUnpublish and NodeUnstage

2.A node dies under RWO

  • Terminating Pods look running : no clock. Deleting the Node fails too.
  • Node off : out-of-service NoExecute taint (StatefulSets). Instant detach, no health check : never on a rebooting node. Untaint after.
  • Or force-delete the Pod : detach after 6m unless --disable-force-detach-on-timeout.
PathTriggerSkipsRisk
Normalunmountednothingnone
Timeout6m, NotReadyin-use checkcorruption
Tainttaint key+ no waitsame
VA finalizerby handUnpublishstill attached

3.New signals

  • Unused : Bound PVCs, Pending Pods count. Nothing acts.
  • Health : redesigned, off, writes dropped.
  • Stats need GET_VOLUME_STATS (not KSM).
SignalByv1.37
Unused conditionkcmbeta, on
3 health fieldssidecar, kubeletalpha, off
kubelet_volume_stats_*kubelet, 1malpha
..._forced_detacheskcmno _total

4.Symptom, cause, fix

FailedAttachVolume = attach side. FailedMount (FailedMapVolume for Block) = node side. A plain timeout posts no summary event since v1.28 : read the kubelet log. FileSystemResizeFailed is read-only mounts only.

Event reasonMessage startsCauseFix
FailedAttachVolumeWaiting for detachRWO in use elsewherewait, Recreate, or taint a dead node
FailedAttachVolumeAttachVolume.Attach failedControllerPublish errorVA attachError, attacher logs
FailedMountMountVolume.MountDevice / SetUp failedNodeStage / NodePublishnode plugin logs
FailedMount... failed while expanding volumenode expansionnode logs, lower if infeasible
FailedMount... conflicting SELinux labelstwo labels, one nodeRecursive, or one label
ExternalExpandingnothing after itno resizer for the driverdeploy csi-resizer
VolumeResizeFailedonly supports offline / OutOfRangeoffline driver / infeasiblestop Pods / lower request
VolumeModifyFailedInvalidArgumentmodify Infeasibleset previous class

Missing attacher or driver : CSI Drivers. VolumeAttachmentLimitExceeded : Volume Topology. Restore as the way back : Snapshots.

5.Commands

CommandShows
kubectl get volumeattachment -o custom-columns=PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,ATTACHED:.status.attached,DEL:.metadata.deletionTimestamp,FIN:.metadata.finalizers,ERR:.status.detachError.messageStuck detaches : fix the driver, keep the finalizer
kubectl get node <n> -o jsonpath='{.status.volumesInUse}{"\n"}{.status.volumesAttached}{"\n"}'Kubelet view, controller view
kubectl taint nodes <n> node.kubernetes.io/out-of-service=nodeshutdown:NoExecuteNode is off. Add - to remove
kubectl get events -A --field-selector reason=FailedAttachVolumeAny wording
kubectl get pvc <p> -o jsonpath='{.status.conditions[?(@.type=="Unused")]}'Unused since
kubectl get --raw "/api/v1/nodes/<n>/proxy/metrics" | grep '^kubelet_volume_stats_available_bytes'Fill per PVC (nodes/proxy)
In short : A volume leaves when the kubelet lets go or the node is proven gone. Alert on reasons.

Storage internals

Snapshots, Clones and Backup

1.Not core : CRDs plus two controllers

  • snapshot.storage.k8s.io/v1 CRDs (GA v1.20), CSI only : VolumeSnapshot (vs, namespaced), VolumeSnapshotContent (vsc), VolumeSnapshotClass (vsclass).
  • The distro installs them and one snapshot-controller; each driver runs a csi-snapshotter (v8.6.0, CSI Drivers). CEL rules in the CRDs replaced the webhook (gone in v8.2.0); VolumeSnapshotClass has none, so edits pass.
kubectl or Velero create VolumeSnapshot new PVC dataSourceRef -> snapshot kube-apiserver CEL rules in the CRDs watch external-provisioner checks binding, driver, ready, size, volume mode watch snapshot-controller one per cluster : default class, finalizers, binding CreateVolume CSI driver creates CreateSnapshot VolumeSnapshotContent snapcontent-<VS uid> csi-snapshotter watches its driver's contents handle, ready, size into content status

2.One dynamic snapshot

  • No class : the driver's one class with is-default-class: "true". Zero or two fail (unlike StorageClass).
  • Driver returns at the cut; readyToUse may lag hours.
FieldVolumeSnapshotContent
linkboundVolumeSnapshot­ContentNamevolumeSnapshotRef
handlenonesnapshotHandle
creationTimedate-timeint64 ns
restoreSizequantityint64 bytes

3.Pre-provisioned, deletionPolicy

  • Import : a content with snapshotHandle, deletionPolicy, volumeSnapshotRef, ideally sourceVolumeMode.
  • Trust it if both point at each other. Delete the VS, not the content (finalizers).
WhenDeleteRetain
VS deletedstorage snapshot goesboth stay
namespace gonestorage snapshot goescontent orphaned
importedset by handset by hand
Veleroonce backup expireswhile backup exists

4.Restore : always a new PVC

  • dataSourceRef names the VolumeSnapshot. No revert in place.
  • Mode guard (GA v1.30) : Block <-> Filesystem needs allow-volume-mode-change: "true" on the content, so content write access is powerful (RBAC).
  • Errors : ProvisioningFailed on the PVC.
Check, in orderError says
CREATE_DELETE_SNAPSHOTcapability is not reported
snapshot not deletingis being deleted
bound, content points backbound to a different snapshot
content driver = provisionernot handled by CSI driver of ...
readyToUse and a handleis not Ready
size >= restoreSizeis less than the size
volume mode guarddoes not have permission

5.Clone a PVC

  • Source PVC in dataSource : Bound, storageClassName on both, size >= source, same driver (any class, v3.2.0+), same volumeMode, CLONE_VOLUME.
  • Same namespace (cross-namespace is alpha). Errors say not bound or invalid : see the provisioner log.

6.Commands

CommandShows
kubectl get vs -n <ns>RESTORESIZE 1Gi
kubectl get vscin bytes
kubectl get vsclass -o custom-columns='NAME:.metadata.name,DEFAULT:.metadata.annotations.snapshot\.storage\.kubernetes\.io/is-default-class'Two defaults ?
kubectl patch vsc <c> --type=merge -p '{"spec":{"deletionPolicy": "Retain"}}'Keep the data
In short : A snapshot is a namespaced request plus a cluster-scoped content holding the storage handle, and they must point at each other. Restores and clones always make a new PVC.

1.dataSource vs dataSourceRef

  • A custom kind in dataSource is dropped on create, no warning : an empty volume. Use dataSourceRef.
  • AnyVolumeDataSource : GA v1.33, gate removed in v1.37. Passing it stops the component.
dataSourcedataSourceRef
kindsPVC, snapshotany non-core kind, PVC
other kinddropped silentlyrejected
synccopied to the Refcopied back if no namespace
both setmust match; both immutable
namespacemust stay emptyalpha gate only

2.Volume populators

  • Unknown kind : the provisioner assumes a populator, stops.
  • lib-volume-populator v3.4.0 : waits for selected-node (WFFC), fills prime-<uid>, moves the PV over.
  • A VolumePopulator (populator.storage.k8s.io/v1beta1) registers each kind. Populators rebind PVs : control-plane code (Operators).
PVC, Pending dataSourceRef -> Hello + target-protection provisioner : assumes a populator, stops populator namespace PVC' prime-<uid> PV-x binds Pod populate-<uid> writes claimRef patched PVC' Lost, Pod and PVC' deleted PVC Bound to PV-x finalizer removed

3.Across namespaces (alpha, stalled)

  • CrossNamespaceVolumeDataSource : alpha, off since v1.26, KEP-3294 closed "not planned". Gate on kube-apiserver, controller-manager and csi-provisioner, plus the ReferenceGrant CRD and RBAC.
  • The grant sits in the source namespace; an empty to.name opens every object of that kind. Gate off : a namespaced ref is dropped silently.

4.Group snapshots : one cut

  • GA v1.36. Snapshotter v8.6.0 serves v1, stores v1beta2, gate Beta and off. Driver needs GroupController.
  • Label the PVCs. Members are owned by the group : none deleted alone, one driver.
  • Use v1 : conversion is None, so v1beta1 readers lose members and its writes skip CEL (CRDs).
SingleGroup
storedv1v1beta2 (serves v1)
turn onCRDs, controller+ CSIVolumeGroupSnapshot gate
picksone PVC namelabel selector, resolved once
consistencyone volumecrash-consistent across all
restorePVC from itone PVC per member
CSI callCreateSnapshotCreateVolumeGroupSnapshot

5.Changed block tracking

  • KEP-3314, beta v1.36, out of tree : the driver advertises SNAPSHOT_METADATA_SERVICE and runs external-snapshot-metadata v1.1.0. Its SnapshotMetadataService gives address, audience, CA.
  • GetMetadataDelta (base = an older snapshotHandle) returns block ranges, not data. FAILED_PRECONDITION : full backup. Velero 1.18 skips it.
backup app audience-scoped token 5. restore to a Block PVC, read those ranges kube-apiserver metadata sidecar TLS, per driver CSI driver 1. TokenRequest 2. gRPC + token block ranges 3. TokenReview, SAR 4. GetMetadataDelta

6.Commands

CommandShows
kubectl get volumepopulatorsRegistered populator kinds
kubectl get crd volumegroupsnapshots.groupsnapshot.storage.k8s.io -o jsonpath='{range .spec.versions[*]}{.name} served={.served} storage={.storage}{"\n"}{end}'v1 served, v1beta2 stored
kubectl get vs -n <ns> -o jsonpath='{range .items[?(@.metadata.ownerReferences[0].name=="<vgs>")]}{.metadata.name}{"\n"}{end}'Members of a group
kubectl get smsCBT drivers (cbt.storage.k8s.io/v1beta1)

Group snapshots ignored ? In v8.6.0 support is a flag on both snapshot components : check their args before the driver.

In short : Populators fill a PVC from any kind, group snapshots cut many volumes at one crash-consistent point, and changed block tracking makes backups incremental.

1.Why a snapshot is not a backup

  • It sits on the same storage as the volume and goes with its namespace under Delete. Record handles.
FailureCSI snapshotVelero, no data moveVelero data mover, fs-backup
namespace deletedgone too under Deletekept (Retain), YAML in bucketdata in the bucket
storage pool lostlost if kept beside the volumelost, only the YAML survivessurvives if the bucket is elsewhere
whole cluster lostre-import by snapshotHandlere-imported, same driver nameany cluster with CSI and the bucket
ransomware, bad writeonly if nobody deleted itsameseparate bucket; guard the key

2.Consistency and hooks

  • One snapshotcrash-consistent
  • Groupcrash-consistent, all volumes
  • + fsfreeze hooksapp-consistent (frozen FS)
  • fs-backupnot point in time

KEP-3476 : quiesce weekly, group snapshot nightly.

3.Velero v1.18 : three ways

  • v1.18.4 (now velero-io/velero) needs --features=EnableCSI, though CSI is built in since v1.14. Hooks : no shell; first container, Fail, 30s.
  • Class pick : PVC annotation, Backup _<driver> annotation, class label velero.io/csi-volumesnapshot-class, the default, the only class.
CSI snapshotCSI snapshot data movementfs-backup (kopia)
turn on--features=EnableCSI+ --use-node-agent, --snapshot-move-data--use-node-agent + pod annotation
data goes tostays on the storagekopia repo in the BackupStorageLocationkopia repo
works forCSI volumesCSI volumesmounted volumes, not hostPath
runs asVelero serverdata-mover Pods, root, BestEffortnode-agent, kubelet hostPath
one volumesnapshot or fs-backup, never both. Restic restores stop working in Velero 1.19

4.One backup with data movement

  • csiSnapshotTimeout (10m) waits for the handle (the cut); itemOperationTimeout (4h) cancels uploads.
  • All repos share one static key : set repository-password first. Groups : velero.io/volume-group label (1.18.x talks v1beta2).
velero backup --snapshot-move-data Velero server VS per PVC, hooks snapshot-controller + csi-snapshotter DataUpload one per snapshot handle (cut) node-agent PVC from snapshot data-mover Pod kopia uploader bucket (BSL) .../kopia/<ns> snapshot deleted after the upload

5.Restore in order, and its pitfalls

  • Storage before workloads, like Flux.
  • A snapshot returns as an imported content (Retain) plus a VS; the PVC loses selected-node, so WFFC picks again. Same driver name.
  • Map classes with resource modifiers. ignoreDelayBinding risks the wrong zone (Topology).
1. CRDs, then refresh discovery 2. namespaces, StorageClasses 3. VS classes, contents, VS, DataUploads 4. PVs, PVCs 5. RBAC, SAs, Secrets, ConfigMaps 6. Pods, ReplicaSets, Services 7. the rest, A to Z (Deployments ...) low priority : Cluster API, Carvel

6.Commands

CommandShows
velero install --features=EnableCSI --use-node-agent --plugins=<object-store-plugin> ...Velero with CSI and the node-agent
velero backup create NAME --snapshot-move-dataSnapshot, then copy the data off
kubectl -n velero get datauploads -o wide -l velero.io/backup-name=NAME -wWatch these, not the backup phase
velero restore create --from-backup NAME --resource-modifier-configmap <cm>Restore with patches (new class). Postgres : CNPG
In short : Snapshots guard against bad writes, not against losing the storage, namespace or cluster. Velero copies data to object storage and restores storage before workloads.

Cluster management

Cluster API

1.What is Cluster API ?

A SIG Cluster Lifecycle sub-project. Whole clusters become custom resources : create, scale, upgrade and delete them declaratively, on cloud or on-prem.

  • One CRD API instead of a different tool per installer and cloud.
  • Controllers reconcile it, like any operator.
Two kinds of cluster
  • Management : runs the CAPI and provider controllers, and holds the Cluster and Machine objects.
  • Workload : the cluster those objects describe. Your apps run there, CAPI does not.

v1.14.2 (2026-09-08), API v1beta2. Workload clusters v1.31 to v1.37.

2.Bootstrap and pivot

A throwaway kind cluster builds the real management cluster, then hands over.

  1. Create a temporary kind cluster.
  2. clusterctl init : kind can now manage clusters.
  3. Apply the target cluster's YAML. Wait for it.
  4. Get its kubeconfig, clusterctl init there.
  5. clusterctl move : pause, copy, resume.
  6. Delete kind. The target manages itself.

move is not backup and restore.

Your laptop clusterctl, kubectl kind cluster temporary 1-2 create, init 6 delete at the end 4 kubeconfig, init again 5 clusterctl move 3 provisions the target Management cluster CAPI core + providers Infrastructure API AWS, Azure, vSphere VMs, LBs networks Workload A Workload B Machines become Nodes, + a kubeconfig Secret

3.Core objects

  • Cluster : pod and service CIDRs, serviceDomain. Refs to its InfraCluster and its control plane.
  • KCP (KubeadmControlPlane) : control plane Machines, etcd membership, certificates.
  • Machine : one host, one Node (status.nodeRef).
  • Templates : each new Machine gets its own clones of an InfraMachineTemplate and a KubeadmConfigTemplate. Treat them as immutable : copy, then repoint the ref.
AWSCluster InfraCluster Cluster CIDRs, domain KCP like StatefulSet infrastructureRef controlPlaneRef MachineDeployment like Deployment MHC health check old + new selects, deletes MachineSet like ReplicaSet Machine like Pod owns N owns CP Machines infrastructureRef configRef nodeRef AWSMachine InfraMachine KubeadmConfig bootstrap Secret Node workload

4.Same pattern as workloads

CAPI objectLikeJob
MachinePodOne host, one Node. Never edited : its owner replaces it
MachineSetReplicaSetKeeps N Machines. Not used directly
MachineDeploymentDeploymentRolls a template change across an old and a new MachineSet
KubeadmControlPlaneStatefulSetControl plane Machines and etcd. Upgrade with spec.version
MachinePoolASG, VMSSHands workers to a provider-native group. Beta, on by default
MachineHealthCheckauto-repairDeletes unhealthy Machines owned by a MachineSet or KCP

5.Providers

  • Infrastructure : VMs, networks, load balancers.
  • Bootstrap : CABPK, kubeadm and cloud-init.
  • Control plane : KCP. Also IPAM and add-on (Helm) providers.
  • clusterctl init adds core, kubeadm bootstrap and KCP by default. The CAPI Operator uses CRDs instead.
Infra providerFlagNotes
CAPA v2.13.1awsEKS too. IAM via clusterawsadm
CAPZ v1.27.0azureAKS too. Azure Service Operator
CAPV v1.17.0vsphereOVA images v1.30 to v1.37
CAPM3 v1.14.1metal3Bare metal, bring Ironic
CAPDdockerDevelopment and tests only

Flag = clusterctl init --infrastructure <name>.

In short : A management cluster reconciles Cluster and Machine objects into real clusters through providers, the way Deployments turn into Pods.

1.ClusterClass and managed topology

A ClusterClass bundles templates, variables and patches. A Cluster then sets only spec.topology. The ClusterTopology gate is alpha and off in v1.14 : export CLUSTER_TOPOLOGY=true before clusterctl init.

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata: { name: capi-quickstart }
spec:
  topology:
    classRef: { name: my-class-v1 }
    version: v1.37.0   # bump to upgrade
    controlPlane: { replicas: 3 }  # odd for etcd
    workers:
      machineDeployments:
      - class: default-worker
        name: md-0
        replicas: 3
Cluster spec.topology ClusterClass variables + patches classRef templateRef KCP template InfraCluster template InfraMachine templates KubeadmConfig template version, replicas Topology controller ClusterTopology gate clones + patches creates and owns KCP + InfraCluster MachineDeployment one per workers entry MHCs from healthCheck Edit the Cluster, never the generated objects.

2.Upgrades replace Machines

A new Machine joins, then the old one is drained and deleted. Control plane first, then workers.

  • Topology : change only spec.topology.version. Since v1.12 it can chain minors from ClusterClass spec.kubernetesVersions.
  • Rollout : RollingUpdate (maxSurge 1, maxUnavailable 0) or OnDelete (MD only). spec.rollout.after forces one.
  • In-place (alpha, v1.12) : an update extension changes Machines without replacing them.
Bump the version spec.topology.version 2+ minors Upgrade plan chained, topology only each minor KCP rollout control plane first may skip minors Worker rollout MachineDeployments Machine image must match the version Without a topology 1. KCP spec.version 2. each MachineDeployment one minor at a time, no upgrade plan New MachineSet up by maxSurge (1) Old MachineSet down, maxUnavailable 0 Old Machine drain, delete infra, Node

3.Commands

CommandDoes
clusterctl init --infrastructure dockerMake this a management cluster. Export gates first
clusterctl generate cluster capi-quickstart --flavor development --kubernetes-version v1.37.0 > capi-quickstart.yamlRender a template from the provider repo. --list-variables shows what it needs
clusterctl describe cluster capi-quickstartTree of the Cluster and its Machines, surfacing problems
clusterctl get kubeconfig capi-quickstart > capi.kubeconfigAdmin kubeconfig, from the <name>-kubeconfig Secret
clusterctl move --to-kubeconfig=mgmt.kubeconfig --dry-runPivot. The target needs the same or newer providers
kubectl patch cluster capi-quickstart --type json -p '[{"op":"replace", "path":"/spec/topology/version", "value":"v1.37.1"}]'Upgrade a topology cluster, control plane first
kubectl delete cluster capi-quickstartTeardown. Cascades through the infrastructure in order
Gotchas
  • Never kubectl delete -f cluster.yaml : infrastructure can be left behind.
  • clusterctl upgrade upgrades providers and CRDs, never a cluster's Kubernetes.
  • No CNI, no Ready nodes. CAPI installs none.
  • nodeDrainTimeoutSeconds 0 = no limit. A PDB can stall a rollout.
Good to know
  • Add-ons : ClusterResourceSet (GA) applies a CNI by label. CAAPH installs Helm charts.
  • GitOps : Flux applies the Cluster YAML. A Kustomization with kubeConfig.secretRef <cluster>-kubeconfig deploys into the workload cluster.
  • v1beta2 Cluster and Machine refs use apiGroup, not apiVersion. v1beta1 stops being served in v1.16.
In short : A ClusterClass holds the shape, the Cluster holds version and replicas, and upgrades replace Machines.

GitOps

GitOps with Flux: Deep Dive

1.Repo layout

Running Flux for many apps and clusters, at v2.9.6. The basics are in Flux CD v2.

  • apps/ : a base plus one overlay per env. infrastructure/ : controllers, then configs.
  • clusters/<env>/ : only Flux objects, such as Kustomizations that point at that env's overlays.
  • Env differences : Kustomize overlays, or .spec.patches on a Flux Kustomization.
One fleet repo
  • fleet/
    • apps/ : base/, staging/, production/
    • infrastructure/ : controllers/, configs/
    • clusters/staging/, clusters/production/
      • flux-system/ : written by bootstrap
      • artifacts.yaml, infrastructure.yaml, apps.yaml

2.Promotion

LayoutHow a change moves on
MonorepoOverlay bump. Big PRs can hide prod
Repo per envCross-repo PRs. Hardest for infra
Repo per teamPlatform onboards each team repo
Repo per appConfig repo points at it by semver
Trunk-based flow
  • Staging gets new app versions from image automation.
  • Production gets them by pull request.
  • Infra goes to staging first, then on after e2e tests.
  • Helm : a passing staging upgrade fires an Alert to a githubdispatch Provider. CI reads metadata.revision and opens the prod PR.

3.Layers with dependsOn

  • Controllers and CRDs, then their custom resources, then apps. All with prune: true.
  • infra-configs has no wait : apps starts once configs are applied, healthy or not.
  • ArtifactGenerator slices the monorepo. Its revision is a content hash, so a commit outside apps/ leaves apps idle.
  • Needs source-watcher. ExternalArtifact gate off : Stalled AccessDenied.
GitRepository flux-system, monorepo ArtifactGenerator source-watcher infrastructure ExternalArtifact apps ExternalArtifact ./controllers infra-controllers wait: true ./configs infra-configs no wait dependsOn apps wait: true red arrow = dependsOn (waits for Ready)

4.What Ready means

Dependency setsIts Ready means
nothingapplied, no more
healthCheckslisted objects healthy
wait: trueevery object healthy
only CEL exprsapplied (CEL unused)

Only the dependency's wait counts, not the dependent's.

The dependsOn check
  • Reads the API server, not the cache : generation observed and Ready=True.
  • Same source : the dependency must have applied that revision too.
  • No watch : a blocked dependent re-checks every 30s (--requeue-dependency).
  • readyExpr replaces the built-in check. Repeat the Ready test in it.
  • Health : kstatus or a CEL expr, polled every 5s until timeout. Failed ends it early.

5.Interval is not the delay

A run starts at once on a new source revision, a spec change, flux reconcile or a Receiver. Also on an edit to a ConfigMap or Secret it uses in substituteFrom, decryption or kubeConfig, if labelled reconcile.fluxcd.io/watch: Enabled. Labelling anything else does nothing.

The three timers
  • intervalfull cycle that fixes drift. Shorter = more API load, not faster delivery
  • retryIntervaldefaults to interval. Flat, no backoff
  • timeoutinterval minus 30s (1h : 59m30s)
In short : Keep apps, infrastructure and clusters apart, promote by pull request, chain layers with dependsOn, and remember Ready means healthy only when the dependency waits.

1.Apply in stages, then prune

  • An SSA dry-run diff first. Only changed objects are sent, as manager kustomize-controller.
  • Stages put CRDs and Namespaces first.
  • Prune deletes what left the build (.status.inventory). Since 2.9 failed prunes retry.
  • Moving a pruned Kustomization recreates its workloads. Set prune: false first.
1. CRDs, Namespaces, ClusterRoles cluster-wide first wait 2. kinds ending in Class StorageClass, GatewayClass ... wait 3. custom stage kinds --custom-apply-stage-kinds (2.8+) no wait 4. everything else Deployments, Services ... prune health stage waits poll every 2s, up to 60s only with wait or healthChecks

2.Why kubectl edits come back

  • Before each apply Flux takes over the fields of every field manager whose name starts with kubectl. Then Git wins : extra fields go, Git's values return.
  • Spared : /status and /scale fields, ssa: Merge objects, other managers.
  • rollout restart rolls twice. Add --field-manager=flux-client-side-apply.

Table keys live under kustomize.toolkit.fluxcd.io/.

Object keyEffect
ssa: OverrideDefault. Git wins
ssa: MergeKeeps others' fields, not atomic lists
ssa: IfNotPresentCreate once, never patch
ssa: IgnoreSkipped, health too
prune: DisabledKept when it leaves Git
reconcile: disabledNo apply, no prune
force: EnabledRecreate if immutable

3.Ignore rules (2.9)

  • .spec.ignore : JSON pointer paths plus an optional target. No target means every object.
  • kubectl edit and the HPA (via /scale) are Update-type, so their values are adopted.
  • HPA : commit a starting replicas and ignore /spec/replicas.
  • An ignored path keeps hand edits forever. Scope it.
ignored path drifted found during a run another Apply-type manager owns the field ? yes no Strip drop it, Flux lets go Adopt copy the live value in on create Git's value seeds the field

4.SOPS and variables

  • Encrypt only data and stringData.
  • secretRef keys : *.agekey, *.asc, sops.*. It replaces the global age key.
  • No secretRef : global age key or KMS identity.
  • No decryption : ENC[...] is applied as is. See Secrets in Git with SOPS or External Secrets.

Test : kustomize build . | flux envsubst --strict. flux debug ks <n> --show-vars prints secrets.

decryption:
  provider: sops
  secretRef: {name: sops-age}
postBuild:
  substitute: {env: prod}  # wins
  substituteFrom:  # later wins
  - {kind: ConfigMap, name: vars}

2.9 strict : a ${var} with no value and no default fails the build, when substitution runs. ${var:=x} is fine.

5.Tenant lockdown

  • --no-cross-namespace-refs=true : a sourceRef to another namespace stalls (AccessDenied). dependsOn is not checked.
  • --no-remote-bases=true, and --default-service-account=default on kustomize and helm (source-watcher from 2.10).
  • flux-system keeps cluster-admin through serviceAccountName: kustomize-controller. See RBAC.
flux-system SA kustomize-controller applies Tenant layer Namespace apps, SA flux RoleBinding : cluster-admin Tenant repo GitRepository in apps Kustomization in apps serviceAccountName: flux dependsOn impersonates SA flux objects in apps namespace only Refused CRDs, cluster-scoped, other-namespace sourceRef
In short : Flux applies in stages, prunes by inventory, takes kubectl-made fields back unless you opt out or ignore them, decrypts SOPS at build time, and fences tenants with impersonation.

1.OCI artifacts, signed

  • Gitless : CI pushes manifests to a registry. Controllers pull only from there.
  • --revision=<branch>@sha1:<sha> returns as lastAppliedOriginRevision, so commit status works.
  • Verify with cosign (keyless matchOIDCIdentity) or notation. Prod refs : semver or digest.
CI flux push artifact cosign sign, flux tag OCI registry artifact + tag + signature poll OCIRepository verify : cosign / notation verified Kustomization lastAppliedOriginRevision VerificationError nothing is pulled CI can also call flux trigger receiver

2.HelmRelease defaults that bite

FieldDefaultChange it to
*.strategyRemediateOnFailure, 0 retriesRetryOnFailure : 5m, no rollback
upgrade.crdsCreate : never updates CRDsCreateReplace
driftDetection.modedisabledenabled
postRenderStrategycombined (2.9) : sees hooksnohooks
upgrade.serverSideApplyauto : keeps the old methodenabled
chartRef (OCI)version+digest : re-push upgradesDisableChartDigestTracking

3.Receivers and latency

  • No webhooks : scan interval + Git poll interval + push and rollout. 5m + 1m is about 6 min.
  • Receivers on both edges : seconds plus rollout.
  • Never on the path : Kustomization, HelmRelease and automation intervals.
  • An automation sees policies in its namespace only, and never pokes its GitRepository.
registry scan interval ImageRepository watch ImagePolicy watch, same ns ImageUpdateAutomation push Git poll interval GitRepository watch Kustomization apply rollout + health a Receiver replaces either red timer
Receivers (v1)
  • Patch requestedAt on the sources they list. Path in .status.webhookPath.
  • 2.9 generic-oidc : no Secret. Pin the token claims.
  • Rate-limit the public Ingress.
Alerts (still v1beta3)
  • Commit status : a github or gitlab Provider + an Alert.
  • Same event dropped for 5m. Big fleets : DisableCommitStatusEvent.
  • Chat : eventSeverity: error.

4."My change is not deployed"

Ready reasonLikely causeLook with
ArtifactFailedSource not Ready : auth, ref, verifyflux get sources all -A
BuildFailed (strict mode)A ${var} with no valueflux debug ks <n> --show-vars
DependencyNotReadyDependency not Ready, or behindflux get ks <dependency>
HealthCheckFailedRollout slower than timeout, or brokenkubectl describe the object
Stalled AccessDeniedCross-namespace ref, or gate offflux debug ks <n> --show-status
HelmRelease RetriesExceeded0 retries by default : one failure stallsfix, then flux reconcile hr <n> --reset

Start with flux get all -A --status-selector ready=false. Owner : flux trace -n <ns> deployment <n>. Preview : flux diff ks <n> --path ./dir.

5.Scale

  • One work queue per kind, sized by --concurrent.
  • Shards : label a source and its users sharding.fluxcd.io/key.
  • Upgrades : FluxInstance "2.9.x" floats patches.
cluster.sizeWorkersLimitsDep. requeue
small51 CPU, 512Mi10s
medium102 CPU, 1Gi5s
large203 CPU, 3Gi5s

Rows set kustomize and helm. Plain defaults : kustomize 4, helm 4, source 2.

In short : Ship signed OCI artifacts, set Helm remediation on purpose, put Receivers on both edges, read the Ready reason first, then scale with workers and shards.

GitOps

Secrets in Git with SOPS

1.Envelope encryption

  • SOPS encrypts values, not files. The YAML shape stays readable.
  • One random 256-bit data key per file seals each selected value : AES256-GCM, its own IV, its key path as authenticated data.
  • The data key is wrapped once per master key (age, PGP, cloud KMS, Vault or OpenBao, HuaweiCloud KMS). Any one can decrypt.
  • Adding a recipient re-wraps the data key. Values stay as they are.

SOPS v3.13.3, age v1.3.2, Flux v2.9.6. Flux basics : Flux CD v2. Short version : GitOps with Flux: Deep Dive.

plaintext value stringData.token data key random, 256-bit, per file AES256-GCM ENC[AES256_GCM,...] own IV, key path as AAD per recipient age PGP AWS KMS Vault same file in Git ENC values + sops: block (wrapped keys, MAC, lastmodified) any one master key unwraps the data key

2.What stays readable

  • Keys, apiVersion, kind and metadata stay plain, so a diff shows which field changed. Encrypt only data and stringData.
  • Comments : the default unencrypted_suffix encrypts all. With encrypted_regex, only those inside the selected keys.
  • The MAC covers every value, plain or not. A hand edit of namespace gives "MAC mismatch" (exit 51). mac_only_encrypted: true (3.9.0+) covers encrypted values only.
apiVersion: v1
kind: Secret
metadata: {name: webhook-token, namespace: default}
stringData:
    token: ENC[AES256_GCM,data:...,iv:...,tag:...,type:str]
sops:
    age:
        - recipient: age1...
          enc: # wrapped data key
    encrypted_regex: ^(data|stringData)$
    lastmodified: "..."
    mac: ENC[...]
    version: 3.13.3  # sorted keys, 4-space indent

3..sops.yaml creation rules

  • The first matching path_regex wins. It is read relative to the config file.
  • The config is found from the current directory upwards, not from the file's folder.
  • Overlay trap : a rule with no selector encrypts apiVersion, kind and metadata too. An overlay's own .sops.yaml counts only if you cd into it.
  • sops edit and sops set reuse the selector stored in the file.
creation_rules:
  # Helm values: whole file
  - path_regex: .*values\.yaml$
    age: age1...
  # Secret manifests: values only
  - path_regex: clusters/prod/.*\.ya?ml$
    encrypted_regex: ^(data|stringData)$
    age: age1...

Rule order matters : under the Secret rule a values file stays plain.

4.Everyday commands

TaskCommand
Make an age key (post-quantum : -pq)age-keygen -o age.agekey
Encrypt a Secret, values onlysops encrypt --age age1... --encrypted-regex '^(data|stringData)$' --in-place s.yaml
Encrypt using .sops.yamlsops encrypt --in-place s.yaml (run inside the repo)
Edit, keeping the MAC validsops edit s.yaml
Read one valuesops decrypt --extract '["data"]["password"]' s.yaml
Set one value (JSON)sops set s.yaml '["stringData"]["k"]' '"v"'
Is it encrypted ?sops filestatus s.yaml
MAC over encrypted values onlysops --mac-only-encrypted encrypt -i s.yaml (global flag, before the subcommand)
Readable git diff*.yaml diff=sopsdiffer in .gitattributes, then git config diff.sopsdiffer.textconv "sops decrypt"
In short : SOPS seals each value with one data key per file, wraps that key for every recipient, keeps the YAML shape and metadata readable, and picks its rule from the nearest .sops.yaml above where you run it.

1.Where Flux decrypts

  • Only kustomize-controller decrypts. helm-controller never does : it reads Secrets already in the cluster.
  • Runs only when spec.decryption.provider: sops is set on the Kustomization that applies the objects. On the root flux-system it does nothing for its children.
  • No MAC check in the controller : patches and generators would break it. The CLI does check.
source artifact 1. decrypt sources 2. kustomize build 3. decrypt objects 4. postBuild substitution 5. normalize 6. guard, apply generator files, envs, local patches, in place any kind with sops.mac, marked Secret data sees plaintext stringData into data, unknown fields dropped no MAC check anywhere here

2.The decryption block

Field or keyEffect
providerRequired. Only sops
secretRef.nameKey Secret, same namespace
serviceAccountNameSecret-less cloud or Vault auth
decrypt: DisabledSkips the post-build pass only
substitute: disabledKeeps ${...} in values
watch: EnabledKey Secret edit starts a run

Last three : annotations or labels under kustomize.toolkit.fluxcd.io/ and reconcile.fluxcd.io/.

kind: Kustomization  # kustomize.toolkit.fluxcd.io/v1
spec:
  path: ./apps/production
  sourceRef: {kind: GitRepository, name: flux-system}
  decryption:
    provider: sops
    secretRef:
      name: sops-age  # entry must end in .agekey

CLI : flux create kustomization ... --decryption-provider=sops --decryption-secret=sops-age.

3.Where the keys come from

Secret entryHolds
*.agekeyage identities (PQ from Flux 2.9)
*.ascarmored PGP key, no passphrase
sops.vault-tokenVault or OpenBao token
sops.aws-kmsstatic AWS keys (YAML)
sops.azure-kvservice principal or identity
sops.gcp-kmsservice-account JSON

Any other name, such as age.key, is ignored without a warning.

Three sources
  • secretRef : a Secret with the entries in the table.
  • Global age key : --sops-age-secret (2.7+). Still needs provider: sops per Kustomization. A secretRef replaces it, no merge.
  • Cloud identity : the controller's own, or serviceAccountName with the ObjectLevelWorkloadIdentity gate (2.6+, else Stalled). See RBAC.
  • Vault, in order : static token, --sops-vault-configmap (2.9+), VAULT_TOKEN. No fallback.

4.Generators and Helm values

  • Generator files are decrypted before the build. Format comes from the key name (values.yaml=).
  • Helm : generate a Secret from the encrypted values, point valuesFrom at it. The hash suffix upgrades the release on change.
  • Values also land in Helm release Secrets (helm get values).
secretGenerator:
  - name: podinfo-values
    files:
      - values.yaml=my-values.enc.yaml
configurations: [kustomizeconfig.yaml]
# kustomizeconfig.yaml: nameReference
# Secret -> spec/valuesFrom/name (HelmRelease)

5.No safety net

Decryption off, ciphertext under stringData : the Secret goes live with the raw ENC[...] text and the Kustomization says Ready=True. ConfigMaps and generated Secrets too. kubectl apply -k never decrypts.

Why, and the fix
  • Normalize drops the top-level sops field, so the guard ("is SOPS encrypted, configuring decryption is required") only fires for ciphertext under data:.
  • Land decryption in an earlier commit than the first encrypted file.
In short : kustomize-controller decrypts between build and apply, only where provider: sops is set, with keys found by name or cloud identity. Nothing refuses ciphertext.

1.Picking a master key

BackendIn .sops.yamlFlux gets it fromWatch out
ageage: age1...*.agekey entry or global keySimplest. PQ keys need Flux 2.9, sops 3.12.1+
PGPpgp: <fingerprint>*.asc entryNot deprecated, but docs prefer age
Cloud KMSkms, gcp_kms ...creds entry or workload identityNo key file. KMS must answer every run
Vault, OpenBaohc_vault_transit_uritoken, or Kubernetes authRole name fixed to {namespace}_{sa}

2.Keyless AWS KMS

  • One KMS key, no age, no static AWS keys. Kustomizations set only provider: sops, so the controller uses its own identity.
  • A sidecar, aws-signing-helper serve, plays the EC2 metadata service. AWS_EC2_METADATA_SERVICE_ENDPOINT points the SDK at it.
  • It signs with a cert-manager client cert and gets short-lived creds for a kms:Decrypt-only role.
  • Fresh cluster : SOPS Kustomizations retry until the CA exists.
cert-manager private ClusterIssuer client cert Secret client auth, 1 year kustomize-controller Pod mounted manager SOPS + AWS SDK signing helper 127.0.0.1:9911 IMDS IAM Roles Anywhere cert in, temp creds out kms:Decrypt AWS KMS unwraps the data key no static AWS keys in Git or the cluster

3.Rotation

  • updatekeys : recipients from .sops.yaml, same data key.
  • rotate : new data key, every value re-encrypted. Do it after removing a recipient.
  • Leaked key (docs order) : drop it, updatekeys, rotate, commit, then change the real passwords. History still opens with the old key.
$ sops updatekeys -y secrets/*.yaml
$ sops rotate -i secrets/db.yaml
# new key Secret in the cluster, then
$ flux reconcile ks apps

A key Secret edit starts no run unless labelled reconcile.fluxcd.io/watch: Enabled.

4.When it breaks

SymptomCauseFix
"failed to load age identities ... no master key was able to decrypt"No *.agekey entry loaded: misnamed, or a secretRef hides the global keyRename to identity.agekey
"MAC mismatch" (CLI, exit 51)Hand edit, or decrypting merged build outputsops edit, or decrypt the patch file
"illegal base64 data at input byte 3"Committed **SOPS** preview output, or plain text under data:Use stringData (Secrets). Never commit previews
"cipher: message authentication failed"Plain patch onto an encrypted objectEncrypt the patch too
Secret value changed after decrypt${...} was substitutedsubstitute: disabled or $${var}
Ready=True, ENC[...] liveNo decryption on that KustomizationAdd provider: sops

flux build and flux diff never decrypt : Secret values show as **SOPS**, ConfigMaps as ENC[...].

5.SOPS or something else ?

SOPS in FluxSealed SecretsExternal Secrets
In Gitciphertext in normal filesa SealedSecreta reference, no values
Decryptskustomize-controllerits own controllerpulls from a store
Rotationper file, by handby hand (key renewal is not rotation)in the store
Needs liveKMS or Vault at reconcileonly its controllerthe store, at sync

More on the third option in External Secrets. Argo CD needs a plugin such as ksops for SOPS, and its docs caution against secret-generating plugins.

In short : Pick age or a cloud KMS, avoid static credentials, rotate with updatekeys and rotate, and read the error first : most failures are a key name, a MAC or a missing switch.