KUBERNETESNOTES
A comprehensive collection of Kubernetes concepts, explained in simple words with clear diagrams and examples. Perfect for quick revision and interview preparation!
- Core Concepts
- Architecture
- Pods and Deployments
- Services and Ingress
- Config and Storage
- GitOps and Certificates
- Autoscaling, Gateway API
- Serverless, Identity
- Observability
- Controllers, Operators, RBAC
- Easy to understand
- Well-structured
- Interview focused
- Quick revision
- Handwritten style for better retention
Start here
Contents
Part 2.Beyond Core
- Flux CD v2GitOps sync from Git, plus a web UI
- Argo CDGitOps apps with a live web UI
- Argo CD Image Updaterbumps image tags for Argo CD apps
- Actions Runner ControllerGitHub runners, one Pod per job
- Gitea Actions Runnersa polling runner pool for Gitea
- Building Images in the Clusterdind, BuildKit, Buildah or kaniko
- cert-managerautomatic TLS certificates
- External Secrets Operatorsecrets from Vault and clouds
- CloudNativePG OperatorPostgres clusters as CRDs
- Crossplanecloud resources declared in YAML
- KEDAevent driven autoscaling
- Karpenteradds nodes for pending Pods
- metrics-serverCPU and memory for HPA
- kube-state-metricsobject state as metrics
- MetalLBLoadBalancer IPs on bare metal
- Gateway API CRDsthe standard routing types
- Gateway APIsuccessor to Ingress
- kgatewayEnvoy based gateway
- Knative Operatorinstalls and upgrades Knative
- Knativescale to zero services
Part 2.Beyond Core (continued)
- OpenTelemetry Operatorcollectors and auto-instrumentation
- Prometheus and ServiceMonitorsmetrics, PromQL, scrape by selector
- Alertmanager Routinggroup, route and silence alerts
- Loki and Log Pipelines with Alloylabel-indexed logs, Alloy pipelines
- Kubernetes Events as a Signalwhy it changed, kept past 1h
- Tracing with Tempospans, TraceQL, trace to logs
- Continuous Profiling with Pyroscopeflame graphs, eBPF, span profiles
Part 3.Advanced Concepts
Under the hood: the machinery that core objects and every add-on rely on, grouped by topic.
- Controllers and Reconciliationwatch, compare, fix, repeat
- Custom Resources (CRDs)your own kinds in the API
- Operatorsapp know-how in a controller
- API Machinery and Server-Side Applyversions, watches, field owners
- Owners, Finalizers and Garbage Collectionwho cleans up, and when
- Admission Controlmutate and check each request
- RBAC and Service Accountswho can do what, and as whom
- Pod Securityprivileged, baseline, restricted
Top to bottom is the path of one change. Blue names are the topics that explain each step.
Part 3.Advanced Concepts (continued)
Who you are and what you trust, trusted images, where Pods run and how they are sized and scaled, then the network and storage under every Pod.
- OIDC Authentication for the API Serverkubectl login with ID tokens
- Keycloak and Single Sign-Onone login for every app
- OpenBao: Secrets as a Serviceshort-lived secrets behind policies
- Private PKI and Trust Distributionyour own CA, trusted everywhere
- Pod Certificatesa cert per Pod, from the kubelet
- Images, Registries and Pullsdigests, pull policy, credentials
- Image Signing: Sigstore and Notationwho published this digest
- SBOMs, Provenance and Attestationssigned claims about a digest
- Enforcing Image Policyverify at admission and on the node
- Advanced Schedulingaffinity, taints, spread, priority
- Dynamic Resource AllocationGPUs and other devices, by claim
- Requests, Limits and QoSreserve, cap, and who is evicted first
- In-Place Pod Resizenew CPU and memory, same Pod
- Vertical Pod Autoscalerrequests sized from real usage
- Horizontal Pod Autoscaler in Depthreplica math, behavior, scale to zero
- Pod Networking and CNIhow a Pod gets an IP and a route
- The Service Dataplanefrom a Service IP to one Pod
- Cluster DNS in Depthfrom a name to a Service IP
- Network Policy in Depthwho may talk to whom, proven
- CSI Drivers and the Volume Lifecyclesidecars, attach, stage, publish
- Volume Topology and Schedulingzones, capacity, attach limits
- Volume Operations and Failure Modesresize, mount cost, stuck detach
- Snapshots, Clones and Backupsnapshots, restores, Velero backups
Top half: traffic on its way to the Pod. Bottom half: the life of its volume.
Part 3.Advanced Concepts (continued)
- Cluster APIwhole clusters as Kubernetes objects
- GitOps with Flux: Deep Diverunning Flux at scale, across clusters
- Secrets in Git with SOPSencrypted Secrets that Flux decrypts
Tip.Start from a question
Start with Part 1. Then read each topic's In short first.
| Situation | Read |
|---|---|
| A Pod cannot reach a Service by name | Cluster DNS, Service Dataplane |
| Only some Pods should reach a database | Network Policy |
| A Pod is Pending: its volume is in another zone | Volume Topology |
| A volume is stuck attaching or detaching | Volume Operations, CSI Drivers |
| You need yesterday's data back | Snapshots and Backup |
| Keep a password in Git safely | Secrets in Git with SOPS, External Secrets |
| A GitHub Actions job waits, no runner picks it up | ARC |
| A Gitea runner rollout cancels running jobs | Gitea Runners |
| CI must build images, but privileged Pods are banned | Image Builds, Pod Security |
| Only signed images may run | Image Signing, Image Policy |
| An object will not delete | Finalizers and GC |
| Two tools fight over the same field | API Machinery |
| Pods need GPUs or other devices | DRA, Scheduling |
| A ServiceMonitor is ignored, no targets appear | Prometheus |
| Too many alert messages, or none at all | Alertmanager |
| Logs are rejected or slow to query | Loki and Alloy |
| A Pod failed an hour ago and its Events are gone | Events |
| One request is slow, but which service and which line ? | Tempo, Pyroscope |
| kubectl should log you in with your company account | OIDC Auth, Keycloak |
| Every app asks for its own password | Keycloak |
| A Pod needs a database password that expires | OpenBao, External Secrets |
| Internal TLS fails with "unknown authority" | Private PKI |
| Each Pod needs its own short-lived certificate | Pod Certificates, SPIRE |
| A container is OOMKilled, or slow under a CPU limit | Requests and Limits |
| A running Pod needs more CPU, without a restart | In-Place Resize |
| Nobody knows what requests to set | VPA, Requests and Limits |
| The HPA scales too late, too often, or not at all | HPA in Depth |
Part 2 adds 29 topics, grouped by what they do.
Part 3 looks under the hood, with 34 topics on how it all fits together.
Core Kubernetes · 01
What is Kubernetes
1.What is Kubernetes ?
Kubernetes is an open-source platform that runs containers across a cluster of machines.
You do not give it a list of steps. You declare the state you want as API objects, and control loops keep pulling the cluster toward it.
- ManagedEKS, GKE, AKS run the control plane.
- Distributionsk3s, RKE2, OpenShift, Talos package it with defaults.
- Localkind, minikube, k3d on a laptop.
- Extendyour own object types and controllers.
- NameGreek for helmsman or pilot. K8s = K, 8 letters, s.
- RootsIdeas from Borg, Google's in-house cluster manager.
- 2014First commit June 6. Google open-sources it.
- 2015v1.0 on July 21. Google pledges it to the new CNCF.
- OSLinux control plane. Linux or Windows workers.
- HomeCNCF hosts it.
- Nowv1.37, patch v1.37.1.
2.Why does it exist ?
An image packages your app. Production needs more : keep it running on many machines, with no downtime. By hand, you are the scheduler, health checker, load balancer and upgrade script.
| Chore | One host, by hand | Kubernetes |
|---|---|---|
| Placement | You pick a host, then SSH in | Scheduler packs Pods by CPU / memory requests |
| Crashed container | Restart policy, same host | Restart, kill on failed checks, no traffic until ready |
| Dead machine | Down until you rebuild it | Deployment replaces the Pods on healthy nodes, after about 5 min |
| More copies | Start more, edit the proxy | kubectl scale, or an HPA fed by metrics-server |
| Upgrade | Stop old, start new, undo by hand | Rolling update, kubectl rollout undo to go back |
| Finding the app | Hard-coded host IP:port | DNS name or IP, load-balanced across copies |
| Config, secrets | Env files on the host, or in the image | ConfigMaps and Secrets, no image rebuild |
| Source of truth | Shell history and memory | API objects, plus manifests kept in Git |
3.Desired vs current state
Every object is a record of intent. You write its spec (what you want). Kubernetes writes its status (what is true now).
A controller is a never-ending loop, like a thermostat. It watches, compares spec with status, and asks the API server for changes until they match.
1.What you get, and what you do not
- Service discovery and load balancing (Services)
- Rollouts and rollbacks at a set pace
- Bin packing : Pods placed by CPU / memory requests
- Self-healing : restart, replace, kill on failed checks
- Storage, batch jobs, secrets and config
- Horizontal scaling, IPv4 / IPv6 dual-stack
- Not an all-in-one PaaS : no code builds, no CI/CD
- No built-in databases, caches or middleware
- No required logging, monitoring or config language
- Does not set up or manage your machines
- Upstream ships no metrics-server (an HPA needs one)
2.Everything is an API object
Pods, Namespaces, ConfigMaps, Events : each is an object behind one HTTP API on the API server. kubectl and controllers all go through it. You write YAML, kubectl sends JSON.
apiVersion: apps/v1 # core kinds use just v1 kind: Deployment metadata: name: web spec: # desired state, yours replicas: 3 # 1 if left out selector: # required, immutable matchLabels: {app: web} template: # the Pod template metadata: labels: {app: web} # = selector spec: containers: - name: web image: nginx:1.30 # no status here : Kubernetes writes it
3.Three ways to manage objects
| Technique | Example | Works on | Best for |
|---|---|---|---|
| Imperative commands | kubectl create deployment web --image=nginx:1. | Live objects | Dev, trying things |
| Imperative object config | kubectl create -f web.yaml, then replace -f | Single files | Production |
| Declarative object config | kubectl diff -f configs/, then kubectl apply -f configs/ | Folders of files | Production |
apply keeps changes made by other writers. Use one technique per object, mixing them is undefined. replace drops whatever the file leaves out, even an HPA's changes.
4.First commands
| Command | What it does |
|---|---|
kind create cluster | Local cluster, nodes are containers (or minikube start) |
kubectl version --client | Client version. Stay within 1 minor of the server |
kubectl config get-contexts | Contexts (cluster, user, namespace) in ~/.kube/config |
kubectl api-resources | Every type this cluster serves, with its group/version |
kubectl explain pods. | Field docs from the cluster's own schema |
kubectl create deployment web --image=nginx:1. | Starter manifest, nothing sent to the cluster |
kubectl diff -f web.yaml | Preview apply. Exit 1 = changes found, above 1 = error |
kubectl apply -f web.yaml | Create or update objects to match the file |
kubectl get deploy web -o yaml | Spec and status, side by side |
kubectl scale --replicas=5 deployment/web | Imperative. The next apply undoes it if the file sets replicas |
kubectl get pods -w | Watch live. Delete a Pod, see a new one appear |
kubectl diff then kubectl apply.Core Kubernetes · 02
Architecture
1.What is in a cluster ?
Two halves. The control plane stores the state you asked for and keeps steering toward it. The nodes run your Pods. You need at least one.
- Control plane : kube-apiserver, etcd, kube-scheduler, kube-controller-manager, plus cloud-controller-manager on a cloud.
- Node : kubelet, a container runtime, usually kube-proxy.
- Add-ons : a CNI network plugin (required) and cluster DNS.
- Cloud : in-tree cloud code is gone since v1.31, so clouds run cloud-controller-manager.
2.Old words, new words
- masterSay control plane. kubeadm switched its node label in v1.24.
- minionSay node, or worker node.
- client-serverNo. Hub-and-spoke around kube-apiserver.
- dockershimGone in v1.24. Nodes use a CRI runtime.
- kube-dnsCoreDNS since v1.13, behind a Service still named kube-dns.
- /healthzDeprecated (v1.16), like componentstatuses (v1.19). Use
/livez,/readyz.
3.Who talks to whom
kube-apiserver is the hub. The others watch it and write back to it. Only kube-apiserver touches etcd.
Arrows point from the caller. Dashed : kube-apiserver calls the kubelet for logs, exec and port-forward. A CNI plugin such as Cilium can replace kube-proxy.
4.The parts, one by one
| Part | Runs on | Job | Port |
|---|---|---|---|
| kube-apiserver | Control plane | HTTP API, authn, authz, admission. Only etcd client. | 6443 |
| etcd | Control plane | Holds every API object, the cluster's only source of truth. Back it up. | 2379-2380 |
| kube-scheduler | Control plane | Gives each unscheduled Pod a node | 10259 |
| kube-controller-manager | Control plane | Built-in loops (Deployment, ReplicaSet, Node ...) | 10257 |
| cloud-controller-manager | Cloud only | Node, route and load balancer loops, cloud API | 10258 |
| kubelet | Every node | Registers the node, runs its Pods via CRI | 10250 |
| container runtime | Every node | Pulls images, makes Pod sandboxes and containers for the kubelet (CRI v1) | socket |
| kube-proxy (optional) | Every node | Packet rules for Service virtual IPs | 10256 |
| CNI plugin (add-on) | Every node | Pod IPs and Pod-to-Pod traffic | - |
| CoreDNS (add-on) | kube-system Pods | Cluster DNS for Services and Pods | - |
Ports are TCP. NodePort range : see Services.
1.From apply to running
- 1-3, API :
kubectl applysends the Deployment over HTTPS. kube-apiserver checks it (authn, authz, mutating admission, validation, validating admission) and saves it in etcd. - 4-5, controllers : the Deployment controller makes a ReplicaSet, which makes Pods with no node yet (Pending).
- 6, scheduler : filters and scores the nodes, then posts a binding.
- 7-11, node : the kubelet sees the Pod and asks the runtime for a sandbox. The runtime calls CNI for the Pod IP, pulls the image, starts the containers. The kubelet reports Running.
No one calls the next step. Each part watches the API and reacts to a change.
2.Inside a node
- The kubelet registers its Node and renews a Lease in
kube-node-leaseas its heartbeat. - It drives the runtime over CRI gRPC on a local socket. CRI v1 only : containerd, CRI-O, or Docker via cri-dockerd.
- Static Pods : files in the kubelet's manifest folder run with no API server. kubeadm puts the control plane in
/.etc/ kubernetes/ manifests - Since v1.35 the kubelet refuses cgroup v1 unless overridden. v1.36+ needs containerd 2.0+.
3.kube-proxy is not the network
4.HA control plane
- 3+ control plane nodes, odd count for etcd quorum : 3 survive 1 loss, 5 survive 2.
- kube-apiserver : all copies active behind a load balancer (TCP check, 6443).
- Scheduler, controller-manager : one leader per Lease, the rest wait.
- Stacked etcd (kubeadm default) : fewer hosts, but a dead node takes an apiserver and an etcd member.
- External etcd : 3+ own hosts, failures stay apart.
Bottom row : scheduler and controller-manager, each with its own leader Lease.
5.Commands
| Command | Shows |
|---|---|
kubectl get nodes -o wide | Nodes, kubelet version, CONTAINER-RUNTIME column |
kubectl get pods -n kube-system -o wide | Static control plane Pods, CoreDNS, kube-proxy, CNI, and their nodes |
kubectl get --raw='/readyz?verbose' | API server checks one by one. /livez for liveness |
kubectl -n kube-system get lease -l apiserver.kubernetes.io/identity=kube-apiserver | One Lease per running kube-apiserver |
sudo crictl ps | On the node : containers via CRI, even with the API server down |
Core Kubernetes · 03
Pods
1.What is a Pod ?
The smallest thing Kubernetes deploys : one or more containers that are always placed and scheduled together on one node, like a small logical host for one app. API : apiVersion: v1, kind: Pod.
- One container per Pod is the usual case. A second one only for a tightly coupled helper, e.g. one that refreshes the web server's files.
- Never add containers to get copies. Copies are more Pods, made by a workload (Deployments and Scaling).
2.Network and storage
- One IP per address family (
status.podIPs: max one IPv4 + one IPv6). - Hostname = Pod name by default. Other Pods must use IP networking to reach it.
spec.volumesis declared once. Each container mounts what it needs with its ownvolumeMounts(Volumes and Storage).
| Shared in the Pod | Set per container |
|---|---|
| IP address and ports | Image and command |
| localhost and IPC | Env variables |
| Volumes it mounts | Probes, resources |
3.Four kinds of container
| Kind | Declared in | When it runs | Probes | Restarts |
|---|---|---|---|---|
| App | spec.containers | All together once init is done, for the whole Pod life | liveness, readiness, startup | Pod restartPolicy (per-container rules beta v1.35) |
| Init | spec.initContainers | Before the apps, one at a time, each must exit 0 | None, no lifecycle hooks either | Retried, unless restartPolicy is Never |
| Native sidecar | initContainers + restartPolicy: Always | Starts in init order, stays up the whole Pod life, stops last | Yes, its readiness counts for the Pod | Always, on any exit (GA v1.33) |
| Ephemeral | via kubectl debug | Added to a running Pod to troubleshoot | Not allowed, no ports either | Never (GA v1.25) |
4.Example : init, sidecar and app
apiVersion: v1
kind: Pod
metadata: {name: web} # DNS label, becomes the hostname
spec:
volumes: [{name: html, emptyDir: {}}]
initContainers:
- name: seed-content # init : must exit 0
image: busybox:1.28
command: [sh, -c, 'echo hello > /html/index.html']
volumeMounts: [{name: html, mountPath: /html}]
- name: refresher # native sidecar
image: busybox:1.28
restartPolicy: Always
command: [sh, -c, 'while sleep 10; do date > /html/time.txt; done']
volumeMounts: [{name: html, mountPath: /html}]
containers:
- name: nginx
image: nginx:1.14.2
ports: [{containerPort: 80}]
volumeMounts: [{name: html, mountPath: /usr/share/nginx/html}]
1.Pod phases (status.phase)
- A rough summary, not a state machine. An in-place Pod restart can go back to Pending. STATUS is not the phase.
| Phase | Means (STATUS you may see) |
|---|---|
| Pending | Accepted, but not every container is set up : no node yet, image pulls, init steps. Init:0/1, ContainerCreating, ImagePullBackOff |
| Running | On a node, all containers created, at least one running or restarting. Running, CrashLoopBackOff |
| Succeeded | Every container exited 0 and none will restart. Completed |
| Failed | All stopped, at least one failed, no restart coming. Pods of a lost node end here too. Error |
| Unknown | State cannot be read, usually because the node does not answer. |
2.Container states and restarts
- ImagePullBackOff : bad name or tag, or a private registry without an imagePullSecret.
| Policy | Exit 0 | Exit 1+ | Where |
|---|---|---|---|
| Always (default) | restart | restart | Only value in a Deployment |
| OnFailure | done | restart | Jobs |
| Never | done | done | Jobs, one-off Pods |
| Sidecar | restart | restart | Ignores the Pod policy |
- Restarts happen in the same Pod, on the same node. RESTARTS goes up, the phase stays Running. Backoff resets after 10 min of clean running.
3.Why not a bare Pod ?
- Scheduled once. Node failure, eviction or delete ends it for good.
- Deployment, StatefulSet, DaemonSet and Job build Pods from a template and replace lost ones (new UID).
- An HPA scales a workload, never a single Pod.
- Delete : 30s grace, TERM (after preStop), then KILL.
- Two containers in one Pod cannot both use :80.
- A helper in
spec.containersis not a sidecar : no start or stop order. - Pod fields are mostly immutable. Change the template instead.
--grace-period=0 --forcedoes not wait for the kubelet. The process may still run.
4.Commands
| Command | What it does |
|---|---|
| kubectl run web --image=nginx --dry-run=client -o yaml | Prints a Pod manifest skeleton, creates nothing |
| kubectl get pods --field-selector=status.phase=Running | Filters on the real phase, not on STATUS |
| kubectl describe pod web | State, Last State, exit code, Conditions, then Events. Conditions in kubelet order : PodReadyToStartContainers (sandbox ready, GA v1.37), Initialized, Ready, ContainersReady, PodScheduled |
| kubectl logs web -c nginx --previous | Logs of the crashed run of one container |
| kubectl debug -it web --image=busybox:1.28 --target=nginx | Adds an ephemeral debug container |
Core Kubernetes · 04
Deployments and Scaling
1.What is a Deployment ?
A Deployment keeps N identical Pods running for a stateless app. It never creates Pods itself: it owns ReplicaSets, and they own the Pods.
- One ReplicaSet per Pod template version, named
web-<pod-template-hash>. - Old ones stay at 0 replicas as revision history (see Rollouts and Probes).
- Links live in
metadata.. Edit the Deployment, not its ReplicaSets.ownerReferences
2.Labels and Selectors
Labels are key/value tags, a selector is a query over them. A ReplicaSet finds its Pods by selector, never by name.
- Key : optional prefix (DNS subdomain, 253 max),
/, name (63 max).kubernetes.io/andk8s.io/are reserved. - Value : 63 max, may be empty.
- Match :
=,==,!=, or set-basedin,notin, exists (key), not exists (!key). - YAML : matchLabels and matchExpressions, all ANDed. No OR.
A selector the template does not satisfy is rejected. In apps/v1 it is never defaulted and is immutable.
3.A Deployment YAML
apiVersion: apps/v1
kind: Deployment
metadata: {name: web}
spec:
replicas: 3
selector: {matchLabels: {app: web}}
template:
metadata: {labels: {app: web}}
spec:
containers:
- name: web
image: nginx:1.29
resources: {requests: {cpu: 100m}}- selectorMust match template labels (extras ok). Immutable.
- replicasDefault 1. Leave out under an HPA.
- templateA change makes (or reuses) a ReplicaSet.
| What broke | Who fixes it | You see |
|---|---|---|
| Container crashed | kubelet, in place, with backoff | RESTARTS climbs, CrashLoopBackOff |
| Pod deleted or evicted | ReplicaSet makes a new Pod | New name, new IP |
| Node failed | ReplicaSet, after node grace + 300s toleration | Nothing for about 6 min, then Pods on other nodes |
| Stray Pod matches | ReplicaSet adopts it, deletes extras | Back to replicas |
4.Which workload controller ?
| Controller | Use it for | Gives you | Watch out |
|---|---|---|---|
| Deployment apps/v1 | Stateless web apps, APIs, workers | Interchangeable Pods, declarative updates | Pod names are random. Use a StatefulSet for stable ones |
| StatefulSet apps/v1 | Databases, queues, quorum systems | db-0, db-1, ... : stable DNS, own PVC, in order | You make the headless Service. PVCs stay |
| DaemonSet apps/v1 | Log shippers, node exporters, CNI agents | One Pod per eligible node | No HPA. Control-plane nodes need tolerations |
| Job batch/v1 | Migrations, one-off batch runs | Runs to completion, backoffLimit 6 | restartPolicy Never or OnFailure |
| CronJob batch/v1 | Backups, nightly reports | A new Job per schedule tick | Runs may overlap (Allow). Use spec.timeZone |
1.Scaling by hand
Scaling only edits spec.replicas. No rollout, no new revision.
$ kubectl scale deployment/web --replicas=5
Undone by an apply that sets replicas, or an HPA.
- Not yet on a node
- Pending, Unknown
- Not ready
- Lower deletion cost
- Doubled up on a node
- Ready for less time
- More restarts
- Newer
The first rule that differs decides (v1.37). The cost is the controller. annotation: the one sure lever.
2.HorizontalPodAutoscaler
An HPA (autoscaling/v2) edits replicas for you. Every 15s its controller reads metrics and writes the target's /scale.
4 Pods at 90% CPU, target 70% : ceil(4 x 90 / 70) = ceil(5.14) = 6, kept within min..max.
- Within 10% of target : no change. Several metrics : the biggest answer wins.
- Utilization is a % of the requests. No request, no action.
metrics-server is an add-on that kubeadm does not install, see metrics-server. Custom or external metrics need an adapter, such as KEDA.
3.HPA manifest and defaults
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: {name: web}
spec:
scaleTargetRef: {apiVersion: apps/v1,
kind: Deployment, name: web}
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target: {type: Utilization,
averageUtilization: 70}| Field | Default | Note |
|---|---|---|
| minReplicas | 1 | 0 needs an Object or External metric (beta) |
| maxReplicas | required | At least minReplicas |
| metrics | 80% CPU | Highest result wins |
| scaleUp | no wait | +100% or +4 Pods per 15s |
| scaleDown | 300s | Highest pick in that window |
| tolerance | 10% | Per-HPA field GA in v1.37 |
| sync | 15s | kube-controller-manager |
replicas or each apply resets it (removing it later can drop to 1 once). Set requests.cpu : 100m, so 70% is 70m.4.Commands
| Command | What it does |
|---|---|
| kubectl create deployment web --image=nginx:1.29 --replicas=3 | Create one with 3 Pods |
| kubectl get deploy,rs,pods -l app=web --show-labels | The whole chain, with pod-template-hash |
| kubectl get pods -l 'env in (prod, qa)' | Set-based selector query |
| kubectl scale --current-replicas=2 --replicas=3 deployment/web | Scale only if it is still at 2 |
| kubectl autoscale deployment web --min=2 --max=10 --cpu=70% | Make an HPA (--cpu-percent is deprecated) |
| kubectl get hpa web --watch | Current vs target metric, replicas |
| kubectl describe hpa web | Conditions (ScalingActive, ...) and events |
| kubectl delete deployment web --cascade=orphan | Remove it, keep ReplicaSets and Pods |
Core Kubernetes · 05
Services
1.Why a Service ?
Pods come and go, and each new Pod gets a new IP. A Service puts one stable DNS name, virtual IP and port in front of the Pods its label selector matches.
- Clients use the name. Traffic goes to ready Pods (if none is ready, to Pods still serving while they stop).
- L4 only (TCP, UDP, SCTP). HTTP routing and TLS : Ingress or Gateway API.
2.How it works
Two watch loops. No traffic goes through either.
- The EndpointSlice controller (kube-controller-manager) watches the selector and the matching Pods.
- It writes Pod IPs, ports and ready state into EndpointSlices labelled
kubernetes.(100 per slice by default).io/ service-name=web - kube-proxy on every node watches both and programs kernel rules for
10.96.0.15:80.
One Pod per connection. iptables walks a chain of random-probability jumps, nftables does one verdict-map lookup, ipvs uses a kernel IPVS virtual server (round robin). The ClusterIP never goes on the wire.
3.The types nest
Types are layers, not rivals. Each keeps what the smaller one gets, so NodePort and LoadBalancer Services still answer inside on their ClusterIP.
A LoadBalancer drops its NodePorts only with allocateLoadBalancerNodePorts: false. ExternalName is the odd one : only a DNS CNAME, nothing is proxied.
| Type | What it gets | Reach it from | Good for |
|---|---|---|---|
| ClusterIP | A virtual IP from the service range | Inside the cluster only | Service to service calls |
| Headless | No IP. DNS returns the Pod IPs | Inside, through DNS | StatefulSets, peer discovery |
| NodePort | ClusterIP + a port on every node | + outside on NodeIP:nodePort | Your own external LB, labs |
| LoadBalancer | NodePorts + an external LB IP | + outside on the LB IP | Production L4 (cloud or MetalLB) |
| ExternalName | No IP. A CNAME to externalName | Inside, cluster DNS only | Alias for an outside host |
4.port, targetPort, nodePort
- portwhat clients dial on the ClusterIP
- targetPortthe Pod's port. Defaults to port, or names a containerPort
- nodePortthe same port on every node
- protocolTCP (default), UDP or SCTP
1.DNS, headless, ExternalName
Every Service gets <svc>.<ns>..
- Plain
webonly works in the same namespace. Elsewhere useweb.prod. - SRV :
_http._tcp.web...needs a named port. - Headless :
clusterIP: None(the word, not an empty field). No VIP, kube-proxy skips it. - No selector : no EndpointSlices. You write them (outside DB, migrations).
BusyBox 1.37 nslookup : a dotless name gets only the search list, a dotted one is sent as typed, exit 1 on any NXDOMAIN. An SRV query with no _http._tcp lists every port.
2.Traffic from outside
Kubernetes ships no load balancer. A cloud-controller-manager, or the controller named in loadBalancerClass, builds one and fills .. Until then EXTERNAL-IP says <pending>. Bare metal : MetalLB.
externalTrafficPolicy (NodePort and LB) :
- Clusterdefault. Any node, SNAT, the app sees a node IP.
- Localclient IP kept. Nodes with no local Pod fail the LB health check.
Local drops only outside traffic : a Pod, or the node itself, calling a node with no local Pod is still answered.
3.Traffic knobs
- sessionAffinity : None or ClientIP (timeout 3 h by default, 24 h max).
- internalTrafficPolicy : Local = same-node endpoints only, or a drop (a timeout).
- trafficDistribution : PreferSameZone or PreferSameNode. A hint, Local wins.
| kube-proxy | Status in v1.37 |
|---|---|
| iptables | Default on Linux |
| nftables | Stable since v1.33, a future default |
| ipvs | Deprecated in v1.35, removal planned v1.43 |
| kernelspace | The only Windows mode |
4.Commands
| Command | What it does |
|---|---|
| kubectl expose deployment web --port=80 --target-port=8080 | Service using the Deployment's selector (add --type) |
| kubectl create service clusterip web --tcp=80:8080 | Selector app=web. --clusterip="None" = headless |
| kubectl create service externalname db --external-name my.database.example.com | CNAME alias for an outside host |
| kubectl get svc web / kubectl describe svc web | TYPE, CLUSTER-IP, EXTERNAL-IP, PORT(S), selector |
| kubectl get endpointslices -l kubernetes.io/service-name=web | Every endpoint, ready or not. Empty = bad selector or no Pods |
| kubectl get endpoints web | Ready ones only, like describe svc, after Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice |
| kubectl run tmp -it --rm --restart=Never --image=busybox:1.28 | Throwaway Pod, then nslookup web.prod |
| curl http://localhost:10249/proxyMode | On a node : which mode kube-proxy runs |
5.Gotchas & Good to know
- One Pod per connection : gRPC and HTTP/2 stick.
- A bare
webfrom another namespace leaves the cluster : web is a public TLD. - ExternalName breaks Host and TLS names.
- Env vars skip Pods older than the Service.
- Two or more ports : each needs a name.
- nftables : NodePorts on the primary IP only.
- Skip
externalIPs(deprecated v1.36) andloadBalancerIP(deprecated v1.24). - v1.37 : names may start with a digit.
Core Kubernetes · 06
ConfigMaps and Secrets
1.What are they ?
Two core v1 objects that keep settings out of the container image. Same shape, different job.
- ConfigMapPlain config, no secrecy at all.
dataholds UTF-8 text,binaryDataholds bytes (base64 in YAML). - SecretA small confidential value (password, token, key), kept out of images and Pod specs.
- BothNamespaced, max 1 MiB. A Pod only references ones in its own namespace.
- Base64An encoding, not encryption.
Y2hhbmdlLW1lis just change-me, one decode away.
| ConfigMap | Secret | |
|---|---|---|
| Made for | Plain settings | Small confidential values |
| Value fields | data, binaryData | data (base64), stringData (plain text, write-only) |
| type | None | Opaque if not set |
| In etcd | Plaintext | Plaintext too, by default |
| At rest | Encryption is opt-in | Encryption is opt-in |
| On nodes | Nothing special | Only nodes whose Pods need it, in tmpfs, gone with the Pod |
| describe | Prints values | Byte counts only, but a token Secret's token in full |
| Volume field | configMap.name | secret.secretName |
2.How a Pod reads them
Four ways in : env vars, $(VAR) in command or args, files in a read-only volume, or app code calling the Kubernetes API. The kubelet watches each object and hands it to the Pod.
containers: - name: app image: nginx envFrom: # every key, once - configMapRef: name: app-config volumeMounts: - name: creds # files, refreshed mountPath: /etc/creds volumes: - name: creds secret: secretName: db-creds # not .name optional: true # start if missing
3.What sees an edit ?
| Read as | Pod spec field | Sees edits ? | To pick up a change |
|---|---|---|---|
| One key as env var | configMapKeyRef / secretKeyRef | No | Restart the Pods |
| All keys as env vars | envFrom : configMapRef / secretRef | No | Restart the Pods |
| Command / args | $(VAR_NAME) from an env var | No | Restart the Pods |
| Volume, whole mount | volumes[]. / .secret | Yes, within ~90 s | App re-reads the file |
| Volume with subPath | volumeMounts[]. | No | Restart the Pods |
| Kubernetes API | App code, get / watch | Yes, live | App handles the event |
| Immutable object | immutable: true | Locked | New name, then roll the Pods |
Restart = kubectl rollout restart deployment/NAME (see Deployments). A container restart also re-reads env, and subPath once its volume has updated. Volumes follow in 60 to 90 s, Pod by Pod; a Pod annotation refreshes at once. The kubelet skips watching immutable objects; reusing one's name can leave a node on the old copy.
1.Who can read a Secret ?
Default cluster, no hardening : every arrow is a way in. Close them one by one.
- etcdEncryption at rest (this path only).
- RBAC
listorwatchon secrets = read-all. - PodsPod-create = read. Split by namespace.
- NodesNo privileged containers. Mount Secrets only where needed.
- SourceExternal store, see External Secrets.
2.Encryption at rest
Off by default. Pass an EncryptionConfiguration with kube-apiserver --encryption-provider-config; edits need a restart unless its automatic reload flag is on. Use KMS v2 (stable v1.29), or secretbox (not aescbc) for a local key.
kind: EncryptionConfiguration apiVersion: apiserver.config.k8s.io/v1 resources: - resources: [secrets, configmaps] providers: - kms: # 1st encrypts writes apiVersion: v2 name: my-kms endpoint: unix:///run/kms.sock - identity: {} # no crypto, migration only
3.Secret types
All but Opaque and bootstrap start with kubernetes.io/.
tls.crt + tls.keyimagePullSecrets, as is dockercfgAlso built in : basic-auth, ssh-auth, dockercfg (legacy), bootstrap token. For SA tokens use kubectl create token.
4.Gotchas
immutable: true(GA v1.21) cannot be undone.- Encryption applies on write: rewrite old ones (
kubectl replace, StorageVersionMigration). binaryDatakeys never become env vars.- Static Pods can never use Secrets or ConfigMaps (v1.37).
- Many small Secrets eat memory. Cap the count (ResourceQuota).
5.Commands
| Command | What it does |
|---|---|
kubectl create configmap NAME --from-file=FILE | Also --from-literal, --from-env-file. Key = file name. |
kubectl create secret generic NAME --from-literal=k='v' | Opaque Secret. Single-quote special characters. |
kubectl get secret NAME -o jsonpath='{. | Read one value. Base64 hides nothing. |
kubectl create secret ... --dry-run=client -o yaml | Prints plain base64. Keep it out of Git, or encrypt it. |
kubectl auth can-i list secrets --as=USER -n NS | Can USER read every Secret in NS ? |
kubectl get secrets -A -o json | kubectl replace -f - | Rewrite all Secrets with the current first provider. |
list on secrets or Pod-create rights can read a Secret. Turn on encryption at rest (KMS v2), keep RBAC tight, and rewrite old objects after any key change.Core Kubernetes · 07
Volumes and Storage
1.Why volumes ?
A crashed or stopped container comes back with a clean filesystem, so the files it wrote are lost. Containers in one Pod also need a place to share files.
- Every volume survives a container restart.
- Ephemeral volumes die with the Pod. Persistent ones (a PV used through a PVC) outlive it.
- Declare in
spec.volumes, mount per container withvolumeMounts(mountPath, readOnly, subPath). Volumes do not nest.
2.Volume types by lifetime
| Type | Lives | Good for | Watch out |
|---|---|---|---|
| emptyDir | Pod | scratch, sharing files | medium: Memory = tmpfs (RAM). Set sizeLimit. |
| configMap, secret, downwardAPI | Pod | config, secrets, Pod info | Read-only files. subPath mounts update only on a restart. |
| projected | Pod | many sources, one folder | Sources must be in the Pod's namespace. |
| ephemeral (generic) | Pod | per-Pod scratch PVC | PVC <pod>-<volume> is deleted with the Pod. |
| csi (inline) | Pod | driver-specific volumes | The CSIDriver must list the Ephemeral mode. |
| hostPath | node | node agents, static Pods | Security risk. Contents differ per node. |
| persistentVolumeClaim | PV | data that has to last | PVC must be in the Pod's namespace. |
| local (PV only) | PV | fast disks on one node | Static, needs nodeAffinity. Node down = Pod stuck. |
3.PV, PVC and StorageClass
- PV : cluster-scoped, static (admin) or dynamic.
- PVC : namespaced request: size, modes, class.
- Binding : 1:1, PV claimRef + PVC volumeName.
- Pod : uses claimName, PVC in the same namespace.
- StorageClass : names a provisioner + parameters.
- Provisioner : makes the PVs, often a CSI driver you install (in-tree cloud types too, v1.25).
- Kubernetes ships no built-in NFS provisioner.
| Access mode | Means |
|---|---|
| ReadWriteOnce RWO | read-write, one node |
| ReadOnlyMany ROX | read-only, many nodes |
| ReadWriteMany RWX | read-write, many nodes (NFS-like) |
| ReadWriteOncePod RWOP | read-write, one Pod |
Modes match claims to PVs. RWO keeps an attached or local volume on one node; Pods there share it. RWOP admits one Pod.
4.Reclaim policy
- Delete : deleting the PVC removes the PV and its storage. Default for dynamic PVs (copied from the class).
- Retain : PV goes Released, data kept for manual cleanup. Default for hand-made PVs.
- Recycle : deprecated.
1.Dynamic provisioning
With volumeBindingMode: WaitForFirstConsumer :
- Admin creates a StorageClass.
- User creates a PVC. It is Pending.
- User creates a Pod with claimName.
- Scheduler picks a node for the Pod.
- Driver makes a volume for that node.
- Driver creates the PV.
- PV and PVC bind. PVC is Bound.
- kubelet mounts it into the Pod.
2.StatefulSet claims
- One PVC per Pod per template: <template>-<sts>-<ordinal>.
- A moved Pod gets its old PVC back (more).
- Scale down or delete keeps PVCs, unless persistentVolumeClaimRetentionPolicy says Delete (v1.32).
- Templates are immutable: resize each PVC.
3.Commands
| Command | What it is for |
|---|---|
kubectl get pv,pvc -A | Status, claim, class, reclaim policy of all of them |
kubectl get sc | Provisioner, binding mode, expansion, (default) |
kubectl get csidrivers | Installed CSI drivers (Persistent, Ephemeral) |
kubectl describe pvc <name> -n <ns> | Events say why it is Pending |
kubectl patch pv <pv> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' | Keep the PV and data before deleting the claim. Back to Delete, local-path cleans it while Released, not once Available |
kubectl patch pvc <pvc> -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}' | Grow the volume, then follow the PVC events |
kubectl patch pv <pv> --type=json -p '[{"op":"remove","path":"/ | Cleaned Released PV back to Available. Remove only /spec/claimRef/uid to keep it for the same claim name |
kubectl delete pvc <name> -n <ns> | Careful : under Delete the data goes too. In use, it stays Terminating and kubectl waits (--wait=false skips it) |
4.Gotchas & good to know
- WaitForFirstConsumer : Pending until a Pod is scheduled is normal. A nodeName pin keeps it stuck.
- No storageClassName = default class (even one added later).
""= class-less PVs only. - Two default classes: the newest wins. Keep one.
- Volumes only grow. Never edit a PV's capacity.
- Grow : allowVolumeExpansion: true and a driver that can grow, then raise the PVC request. Failed? Lower it to above the old size (v1.34). FS: XFS, ext3, ext4 (online OK).
- Snapshots : CSI only, CRDs from your distribution. Restore via PVC dataSource.
- Databases want an operator, see CloudNativePG.
Core Kubernetes · 08
Namespaces and Ingress
1.What is a Namespace ?
A scope for names. A name is unique inside its namespace, and the next namespace can reuse it.
- No nesting. Each namespaced object lives in exactly one.
- Name = RFC 1123 DNS label. The fixed label
kubernetes.is added for you.io/ metadata. name - Deleting one deletes all inside (phase
Terminating): Pods first, then the rest. - Production : make your own, skip
default.
| Built in | Used for |
|---|---|
| default | Your objects when you name no namespace |
| kube-system | Objects Kubernetes itself creates |
| kube-public | Readable by anyone, even anonymous. Mostly cluster use |
| kube-node-lease | One Lease per node, the kubelet heartbeat for spotting dead nodes |
The kube- prefix is reserved for system namespaces.
2.Names and policy, not traffic
Each of these is namespaced and acts only at home :
- Role + RoleBinding : RBAC, here only
- ResourceQuota : total cap, 403 if over
- LimitRange : per Pod / Container limits
- NetworkPolicy : traffic (section 4)
DNS crosses the line too. A Service is <svc>.<ns>.. A bare api only finds your own namespace's api (the search list starts there), so use api.team-b or the full name.
3.Namespaced or not ?
| Kind | In a ns ? | Short |
|---|---|---|
| Namespace | No | ns |
| ResourceQuota | Yes | quota |
| LimitRange | Yes | limits |
| Role / RoleBinding | Yes | none |
| ClusterRole | No | none |
| NetworkPolicy | Yes | netpol |
| Ingress | Yes | ing |
| IngressClass | No | none |
apiVersion : v1 for the first three, rbac. for RBAC, networking. for the last three.
4.NetworkPolicy : the real wall
- Namespaced L3/L4 allow-list for TCP, UDP, SCTP.
- Pods start open. Once a policy selects a Pod for Ingress or Egress, only allowed traffic passes. Policies add up, no deny rules.
- Both ends must agree : egress at the source and ingress at the target.
- No
policyTypes? Ingress is implied. Egress only if it has egress rules. - Needs an enforcing network plugin (Antrea, Calico, Cilium, Kube-router, kind's kindnet). Without one it is stored and ignored, with no warning.
- A policy cannot name a namespace. Match the label
kubernetes..io/ metadata. name=web - In one
fromentry, namespaceSelector + podSelector mean AND. As two list items (two dashes) they mean OR. - Default-deny egress also blocks DNS. Allow the CoreDNS Pods.
- Test a deny policy before you trust it.
1.Ingress : rules, not a proxy
An Ingress (networking., namespaced) is just HTTP routing rules. A controller carries the traffic.
- No controller ? Stored, does nothing.
- HTTP / HTTPS only (80, 443). Else use a NodePort or LoadBalancer Service.
- Backends : Services in its own namespace.
- v1.37 : GA since v1.19, frozen. It stays, but the project recommends Gateway API.
2.Inside an Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: {name: shop, namespace: shop}
spec:
ingressClassName: example-class
tls: # hosts = rule hosts
- {hosts: [shop.example.com], secretName: shop-tls}
rules:
- host: shop.example.com
http:
paths:
- path: /api
pathType: Prefix # required
backend:
service: {name: api, port: {number: 8080}}
- path: /
pathType: Prefix
backend: # port: number OR name
service: {name: web, port: {name: http}}
Host first (exact, or *.foo.com = one label), then the longest path, Exact winning ties. No match : spec., else the controller decides.
| Type | Path | Request | Match ? |
|---|---|---|---|
| Exact | /foo | /foo/ | no |
| Prefix | /foo/ | /foo/ | yes |
| Prefix | /foo/ | /foo/ | no, whole parts |
| Prefix | /foo/ | /foo/ | yes |
| ImplementationSpecific : up to the IngressClass | |||
- TLS ends at the LB or controller, plain text after. Port 443 only, hosts share it by SNI.
- Secret type
kubernetes.io/tls(tls.crt,tls.key). A rule with no host gets no TLS. Certs : cert-manager.
3.Picking a controller
- Cluster-scoped.
spec.controller(domain-prefixed, immutable) names the controller. - Default :
ingressclass.. Two defaults = class-less Ingresses rejected.kubernetes. io/ is-default-class: "true" kubernetes.annotation : deprecated.io/ ingress. class
- Last releases 2026-03-19, archived 2026-03-24. No more fixes, not even security. Installs keep running.
- Not F5's NGINX Ingress.
nginx.annotations don't carry over.ingress. kubernetes. io/* - Way out : Gateway API (ingress2gateway 1.0 converts) or another controller.
4.Commands
| Command | What it tells you |
|---|---|
kubectl get ingressclass | Classes and their controllers |
kubectl get ing -A | Empty ADDRESS = no controller took it (yet) |
kubectl describe ingress shop -n shop | Rules, backends with endpoints, events |
kubectl create ingress shop --rule="shop. | Draft YAML. Trailing * = Prefix, else Exact |
kubectl create secret tls shop-tls -n shop --cert=tls.crt --key=tls.key | The Secret for spec.tls |
kubectl get pods -A --selector app. | Still running the retired controller ? |
Core Kubernetes · 09
Rollouts and Probes
1.What starts a rollout ?
A rollout swaps Pods from an old ReplicaSet to a new one. Pods are never edited in place.
- Trigger : any change under
.spec.template(image, env, labels...). Each one makes a new ReplicaSet, a new revision. - Changing
replicasis scaling, not a rollout. .: RollingUpdate (default) or Recreate.spec. strategy. type - Recreate : all old Pods gone first, then the new ones. Downtime by design.
ReplicaSets and scaling : Deployments and Scaling.
2.Step size
- maxSurgePods allowed above replicas. 25%, rounds up.
- maxUnavailablePods allowed to be unavailable. 25%, rounds down.
- both 0Rejected : the rollout could never move.
A number or a percent. It is not one Pod at a time : 10 replicas start 3 extra Pods and drop 2 at once.
Terminating Pods do not count as available, so the real count can briefly pass max Pods until their grace period ends.
| replicas | surge | unavailable | min available | max Pods |
|---|---|---|---|---|
| 1 | 1 | 0 | 1 | 2 |
| 3 | 1 | 0 | 3 | 4 |
| 4 | 1 | 1 | 3 | 5 |
| 10 | 3 | 2 | 8 | 13 |
3.Readiness sets the pace
A new Pod is available once it is Ready and stays Ready for minReadySeconds (default 0). Beyond maxUnavailable, old Pods only go as new ones become available.
- No readinessProbe ? Ready the moment it starts, so traffic and the rollout run ahead of the app.
- A new Pod that never gets Ready stalls the rollout. Old Pods keep serving.
- Capacity can dip by default.
maxUnavailable: 0keeps full capacity.
| Progressing | reason | rollout status |
|---|---|---|
| True | NewReplicaSetAvailable | exits 0 |
| True | ReplicaSetUpdated | keeps waiting |
| False | ProgressDeadlineExceeded | exits 1 |
progressDeadlineSeconds (default 600, must be greater than minReadySeconds) gives the False row. The controller keeps retrying and never rolls back. That is your job.4.Revisions and rollback
- Each ReplicaSet is a revision.
revisionHistoryLimit(default 10) is how many old ones stay. 0 means no undo. rollout undoreverts the Pod template only.- The reused ReplicaSet gets a new number : rev 2 comes back as rev 4.
- Resume a paused Deployment before you undo.
1.Three probes, three jobs
The kubelet runs the probes. Liveness and readiness run side by side and never wait for each other.
| Probe | On failure | Use it for |
|---|---|---|
| startup | Kill, restartPolicy | Slow starters. The others wait. |
| liveness | Restart, same Pod, same node | Hangs. Never a dependency check. |
| readiness | No Service traffic, no restart | Warm-up, overload, deps down |
One check per probe. No probe counts as Success.
- execthe command exits 0
- httpGetstatus 200 to 399
- tcpSocketthe port accepts a connection
- grpchealth check says SERVING (GA v1.27)
2.Probes in a Pod template
containers:
- name: nginx
image: nginx:1.16.1
ports: [{name: http, containerPort: 80}]
startupProbe:
httpGet: {path: /, port: http}
periodSeconds: 5
failureThreshold: 12 # up to 60s to start
readinessProbe:
httpGet: {path: /, port: http}
periodSeconds: 5
livenessProbe:
tcpSocket: {port: http}
3.Timing
| Field | Default | Note |
|---|---|---|
| initialDelaySeconds | 0 | From container start. Liveness and readiness also wait for startupProbe |
| periodSeconds | 10 | How often |
| timeoutSeconds | 1 | One try fails after this |
| successThreshold | 1 | Only 1 for liveness, startup |
| failureThreshold | 3 | Fails in a row = failed |
4.Readiness gates
spec.adds a Pod condition that an outside controller sets.readinessGates[]. conditionType - Ready = containers ready and every gate True. Missing counts as False.
kubectl drain). Deleting Pods and rolling updates skip it, since maxUnavailable is the rollout's own budget. Healthy = Ready, and maxUnavailable: 0 means drain never ends.5.Commands
| Command | What it does |
|---|---|
| kubectl set image deployment/web nginx=nginx:1.16.1 | Changes the template image, so a rollout starts |
| kubectl rollout status deployment/web --timeout=5m | Waits. Exit 0 when done, 1 once past the deadline |
| kubectl annotate deployment/web kubernetes.io/change-cause="to 1.16.1" | Fills CHANGE-CAUSE in history (replaces --record) |
| kubectl rollout history deployment/web --revision=2 | Lists revisions, or one revision's template |
| kubectl rollout undo deployment/web --to-revision=2 | Back one revision, or to N. --dry-run=server previews |
| kubectl rollout pause / resume deployment/web | Batches edits into one revision (Deployments only) |
| kubectl rollout restart deployment/web | New Pods for all, through a normal rolling update |
| kubectl describe pod <pod> | Probe failures show as Warning Unhealthy events |
| kubectl create pdb web-pdb --selector=app=web --min-available=50% | A budget (or --max-unavailable, never both) |
Core Kubernetes · 10
StatefulSets and DaemonSets
1.Why a StatefulSet ?
Deployment Pods are interchangeable. A StatefulSet is a set of Pods with names. It promises four things. The first two survive a reschedule :
- a fixed name and DNS name per Pod
- its own storage per Pod
- ordered start and scaling
- ordered rolling updates
Need none of them ? Use a Deployment (Deployments and Scaling). Stable apps/v1 since v1.9.
| A Pod in a | Deployment | StatefulSet |
|---|---|---|
| Name | <rs>-<random> | <sts>-<ordinal> |
| Replaced by | a new Pod, new name | same name, same PVC |
| DNS name | none per Pod | one per Pod |
| Storage | shared PVC or none | own PVC per Pod |
| Order | any | 0 first, top down |
2.Identity : name, DNS name, PVC
| Piece | What web-1 gets |
|---|---|
| Name, hostname | web-1 |
| DNS name | web-1. |
| PVC | www-web-1 : template, set, ordinal |
| Labels | apps., statefulset. |
The pod-name label lets a Service pick one replica. ordinals.start (GA v1.31) shifts the range : start 1 gives db-1 to db-3.
3.The headless Service
- You create it and name it in
serviceName. Deleting the set leaves it behind. - Optional since v1.33, but no DNS names without it. Nobody checks that it exists.
- DNS lists Ready Pods only.
publishNotReadyAddresses: truelets peers meet while booting. - Use names, not Pod IPs : IPs change. See Services.
4.Own PVC, in short
volumeClaimTemplates: one PVC per template per ordinal. A replaced Pod mounts the same one.- PVCs stay on scale down and delete, unless the retention policy (GA v1.32) says Delete.
- Templates are immutable : grow each PVC by hand (Volumes and Storage).
- Zonal disks : use
WaitForFirstConsumer.
5.Order : OrderedReady vs Parallel
| OrderedReady (default) | Parallel | |
|---|---|---|
| Scale up | 0 up, each waits for lower Pods Running and Ready | at once |
| Scale down | from the top, each waits until the previous is gone | at once |
| Sick Pod | scale down waits, unless the sick one goes next | no wait |
| Change it | immutable : orphan delete, recreate | |
minReadySeconds makes each step wait for Available too. A Failed Pod (eviction, reboot) comes back with the same name.
6.At most one web-1
Never two Pods with one name : a Pod on a lost node stays Terminating, and no twin starts. Ways out, safest first :
- Machine really gone : delete the Node object.
- Node confirmed shut down : taint it
node.. Pods are force-deleted, volumes detach. Remove the taint after.kubernetes. io/ out-of-service - Last resort : force delete the Pod. Risks split brain.
- Never
terminationGracePeriodSeconds: 0. - Deleting the set is not ordered : scale to 0 first.
- With an HPA, leave
replicasout of the manifest.
1.Rolling update and partition
A template change makes a new ControllerRevision. RollingUpdate (default) replaces Pods from the highest ordinal down : same name, same PVC, new template.
- Each step waits for Running and Ready, plus
minReadySeconds(Rollouts and Probes). partition(default 0) : only Pods at or above it update. Lower ones stay old, even when deleted.- It counts from
ordinals.start: with start 5, partition 1 updates ord-6 and up.
2.Three update strategies
| RollingUpdate (default) | OnDelete | Recreate (alpha, v1.37) | |
|---|---|---|---|
| Who replaces | controller, top down | you, by deleting Pods | controller, all at once |
| Old and new | side by side | side by side, no limit | never : downtime |
| rollout status | works | refuses | refuses |
| Switch with | default | "rollingUpdate": null | same, plus a feature gate |
OnDelete : scaled-up Pods get the new template too. Recreate needs StatefulSetRecreateStrategy on kube-apiserver and kube-controller-manager; beta is planned for v1.38, not promised.
3.maxUnavailable (Beta)
- Several Pods per step. On by default in v1.37 (default 1). In v1.35.4+ and v1.36 the field is dropped silently.
- A percent rounds down, minimum 1. The docs say up.
- OrderedReady : several go, but they return one at a time. Parallel : together. Every unavailable Pod counts.
| maxUnavailable | Pods per step |
|---|---|
| 1 (default) | 1 |
| 30% | 1 |
| 60% | 2 |
| 70% | 2 |
4 replicas, seen on v1.37.1. 60% again on a kind cluster.
4.The stuck rollout
A template that never gets Ready stops an OrderedReady rollout. Old Pods keep serving.
rollout undoalone is not enough : the controller waits on the stuck Pod forever.- Delete it : it comes back on the reverted template. Parallel (v1.37) and Recreate do this for you.
- Upstream : working as intended, a Pod may hold data.
5.Revisions and commands
| Command | What it is for |
|---|---|
kubectl get pods -l app=nginx -L controller-revision-hash | Which Pod runs which revision |
kubectl patch sts web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}' | Canary : only web-2 takes the new template |
kubectl rollout status sts/web --timeout=10m | The only deadline you get. Checks Ready, not minReadySeconds |
kubectl rollout undo sts/web --to-revision=3 | Pod template only. The old revision returns with a new number |
kubectl annotate sts/web kubernetes.io/change-cause='bump nginx' | CHANGE-CAUSE in rollout history (--record is deprecated) |
10 ControllerRevisions kept. No progressDeadlineSeconds, rollout pause fails. PDBs limit drains, not updates. GitOps ? Revert in Git.
partition stages a canary, OnDelete waits for you, and a Pod stuck on a bad template must be deleted by hand after the revert.1.DaemonSet : one Pod per node
A DaemonSet has no replica count : the list of eligible nodes is the count. A node joins, it gets a Pod. It leaves, its Pod is garbage-collected.
- For node work : CNI and CSI node plugins, log and metrics agents, kube-proxy.
- No
/scale: no HPA,kubectl scale dsfails. --cascade=orphankeeps its Pods for the next DaemonSet with that selector.
2.Which nodes get a Pod ?
nodeNamein the template, if set, must match.nodeSelectorand required node affinity must match.- Every NoSchedule and NoExecute taint must be tolerated.
CPU and memory are not checked : a full node leaves the Pod Pending. Free room, or use priorityClassName: system-node-critical. Each Pod is pinned with matchFields metadata.name and the scheduler binds it (Advanced Scheduling).
| Added toleration | Effect | So the Pod |
|---|---|---|
| not-ready, unreachable | NoExecute | never evicted |
| disk, memory, pid pressure | NoSchedule | on busy nodes |
| unschedulable | NoSchedule | on cordoned nodes |
| network-unavailable | NoSchedule | hostNetwork only |
not-ready:NoSchedule: a NotReady node gets no new Pod. CNI and kube-proxy tolerate it themselves.- The control-plane taint : add it yourself.
3.Updates : maxUnavailable or maxSurge
- Default : maxUnavailable 1, maxSurge 0. Percents round up.
maxSurgeneedsmaxUnavailable: 0. Old goes once new is Available.- OnDelete : only deleted Pods and new nodes update.
- Bad template ? Push a fixed one : broken Pods go first, and use up the budget.
4.Deployment vs StatefulSet vs DaemonSet
| Deployment | StatefulSet | DaemonSet | |
|---|---|---|---|
| How many | replicas, HPA | replicas, HPA | one per eligible node |
| Storage | shared PVC or none | own PVC per ordinal | hostPath, CSI node mounts |
| Order | none | OrderedReady or Parallel | none |
| Strategies | RollingUpdate, Recreate | RollingUpdate, OnDelete, Recreate (alpha) | RollingUpdate, OnDelete |
| Deadline, pause | yes | no | no |
| Good for | stateless web, APIs | databases, quorum, brokers | CNI, CSI, log and metrics agents |
5.Troubleshooting
| Symptom | Check | Likely cause, fix |
|---|---|---|
| Stuck at N-1/N | kubectl get pods -L controller-revision-hash | bad template : revert, delete the stuck Pod |
| No StatefulSet Pods | kubectl describe sts web | name over 52 characters, Pod Security, quota |
| Pending after a move | kubectl describe pod web-2 | its volume is pinned to another zone |
| DESIRED too low | node taints and labels | untolerated taint, or nodeSelector |
| DaemonSet CURRENT 0 | kubectl describe ds agent | Pod Security : use a privileged namespace (Pod Security) |
| Drain refuses | kubectl drain output | add --ignore-daemonsets : they tolerate cordon |
| DaemonSet rollout hangs | events, dead nodes | hostPort with maxSurge, or a Pod stuck Terminating |
Core Kubernetes · 11
Jobs and CronJobs
1.What is a Job ?
A Job runs Pods until enough of them succeed, then stops. A CronJob makes a new Job on a schedule. Both are stable batch/v1 workloads (see Deployments and Scaling).
- job-controller in kube-controller-manager makes the Pods, counts the finished ones and writes the result.
restartPolicymust beNeverorOnFailure. Unset means Always : rejected.
| Object | Ends | Lost Pod back | Trigger |
|---|---|---|---|
| Job | yes | yes | create |
| CronJob | yes | via its Jobs | schedule |
| Bare Pod | yes | no | create |
| Deployment | no | yes | - |
| ScaledJob | yes | via its Jobs | event |
Use a Job even for one task : a bare Pod lost with its node is not replaced.
2.The retry loop
A Job is a retry loop with a failure budget. A failed Pod is replaced after a delay, until the failures pass backoffLimit.
- backoffLimit counts failures you tolerate, not tries : the Job fails when
failed > backoffLimit. - Default 6 : the 7th failed Pod ends the Job.
0fails on the first. - Delay : 10s, doubling, capped at 10 min (the docs still say six). A success resets it.
- activeDeadlineSeconds beats the budget. It counts from
startTime, Pending and image pulls included.
| Failure | 1 | 2 | 3 | 4 | 5 | 6 | 7+ |
|---|---|---|---|---|---|---|---|
| wait | 10s | 20s | 40s | 80s | 160s | 320s | 600s |
3.Never vs OnFailure
| Never | OnFailure | |
|---|---|---|
| Retried by | controller, new Pod | kubelet, same Pod |
| Delay cap | 10 min | 5 min (CrashLoopBackOff) |
| Budget | failed Pods | failed Pods or restarts |
| At limit | failed Pods stay | Pod killed, logs may go |
| Default of | create job | create cronjob |
4.Completions and parallelism
| Kind | completions / parallelism | Done when |
|---|---|---|
| Single | unset / unset | one Pod succeeds |
| Fixed | N / P | N Pods succeeded |
| Indexed | N / P | each index 0..N-1 once |
| Queue | unset / P | a success, none active |
parallelismcan change any time,0pauses.completionsis frozen (Indexed : only with parallelism, same value).
completions stays null (the docs say it copies parallelism; it does not).JOB_COMPLETION_INDEX and hostname <job>-<i>. A headless Service as subdomain lets peers find each other.1.How a Job ends
Always in two steps : an interim condition first (the Pods still running are deleted), then the final one, with the same reason, once no Pod is left terminating.
Pods deleted at the end count as failed : one exit 42 in a parallelism 3 Job showed failed: 3.
| What fired | Reason on both |
|---|---|
backoffLimit | BackoffLimitExceeded |
activeDeadlineSeconds | DeadlineExceeded |
| podFailurePolicy FailJob | PodFailurePolicy |
maxFailedIndexes | MaxFailedIndexesExceeded |
| a failed index | FailedIndexes |
| successPolicy | SuccessPolicy |
| completions reached | CompletionsReached |
kubectl describe job shows no conditions : use get job -o yaml.
2.Pod failure policy
spec:
podFailurePolicy:
rules:
- action: FailJob # known bug
onExitCodes:
operator: In
values: [42]
- action: Ignore # not our fault
onPodConditions:
- type: DisruptionTarget
- FailJobFail the whole Job now.
- FailIndexFail this index only (needs backoffLimitPerIndex).
- IgnoreNot counted, but still delays the retry.
- CountNormal. Also the result when no rule matches.
Rules run in order, first match wins (max 20). Exit 0 never matches. It forces restartPolicy: Never and podReplacementPolicy: Failed. GA v1.31.
DisruptionTarget : preemption, API eviction, taint manager, Pod GC or kubelet termination. Not your bug.
3.Per-index budget, success policy
- backoffLimitPerIndex : each index gets its own retries. Indexed only, like the rest here (GA v1.33).
- maxFailedIndexes : past it, the Job fails at once. Unset : every index runs, then any failed one fails the Job (not Complete, as the API comment says).
- successPolicy : the Job succeeds early once listed indexes (or a count of them) succeed.
- Failures still come first : the docs' success example can end BackoffLimitExceeded. Raise backoffLimit.
4.Replace, hand off, suspend
- podReplacementPolicy :
TerminatingOrFailed(default) counts a deleting Pod as failed and replaces it after the retry delay (about 10 s).Failedwaits until it has failed : one Pod per index at a time. GA v1.34. - managedBy : hands the Job to another controller (MultiKueue). If nothing implements it : no Pods, STATUS Running forever. GA v1.35.
- suspend : running Pods get SIGTERM, counted as neither. Resume resets startTime, so the deadline restarts. Kueue admits Jobs this way.
- Yes : parallelism, backoffLimit, activeDeadlineSeconds, ttlSecondsAfterFinished, podReplacementPolicy, maxFailedIndexes, suspend.
- No : completionMode, NonIndexed completions, backoffLimitPerIndex, podFailurePolicy, successPolicy, managedBy.
- Template : scheduling and resources, only while suspended. Never the image : recreate, or use Flux's force annotation.
1.A CronJob stamps out Jobs
- At each tick the cronjob-controller creates
<cronjob>-<minutes since epoch>. The fixed name blocks a double run. - The Job copies
jobTemplate, is owned by the CronJob and notes its slot in an annotation. - Edits and
suspend: trueonly touch future runs (open issue 128155 : ascheduleedit can run a passed minute). Names stop at 52 characters.
2.Schedule syntax
| minute | hour | day | month | weekday |
|---|---|---|---|---|
| 0-59 | 0-23 | 1-31 | 1-12, jan-dec | 0-6, sun-sat |
| 30 | 2 | * | * | 1-5 |
- Above : 02:30, Monday to Friday. Also
*/15,1,5,@hourly,@daily,@weekly,@monthly,@yearly. - Rejected :
7as Sunday, a seconds field,@reboot,TZ=. - Day and weekday both set means either :
0 0 13 * 5runs on the 13th and every Friday.
- No
timeZone: kube-controller-manager's own zone. On kubeadm that is UTC. - Set an IANA name :
timeZone: Etc/UTC.Localis rejected. - The scheduled-timestamp annotation is in that zone, UTC when unset.
- DST (library test, New York) :
30 2 * * *skipped on 2026-03-08,30 1 * * *twice on 2026-11-01. Etc/UTC dodges it.
3.Overlap and missed runs
| Policy | Last run still active |
|---|---|
| Allow (default) | a new Job starts anyway |
| Forbid | skipped (JobAlreadyActive), may start late when the blocker ends |
| Replace | old Jobs deleted, then the new one |
A hung run blocks Forbid forever : set activeDeadlineSeconds in the jobTemplate. Manual runs are invisible to Forbid.
- Catch-up is one Job, for the latest missed slot (downtime, suspend, Forbid).
startingDeadlineSeconds: older slots are dropped with no Job and no event (MissSchedule never fires).- Over 100 missed : Warning TooManyMissedTimes, and the latest run still starts.
- Watch
lastScheduleTimeor kube-state-metrics, not events : they thin out.
4.Cleanup
| Mechanism | Covers | Deletes with |
|---|---|---|
ttlSecondsAfterFinished | any finished Job, clock starts at Complete or Failed, 0 = at once | Foreground |
| history limits (3 and 1) | finished Jobs the CronJob owns, 0 keeps none | Background |
kubectl delete job | the Job and its Pods (--cascade=orphan keeps Pods) | Background |
| raw API DELETE | the Job only, its Pods are left behind | Orphan |
A standalone Job without TTL stays forever. Owners and propagation : Garbage Collection.
5.Debugging, in order
| Command | Look for |
|---|---|
kubectl describe job pi | Events : the reason, FailedCreate (quota, Pod Security, webhook) |
kubectl get pods -l batch.kubernetes.io/job-name=pi --field-selector=status.phase=Failed | Only the failed Pods. Then describe pod : exit code, OOMKilled |
kubectl logs job/pi --all-pods=true | Logs of every Pod, prefixed |
kubectl create job hello-retry --from=cronjob/hello | A Failed Job never restarts. Make a new one |
kubectl events --for cronjob/hello | JobAlreadyActive, UnexpectedJob, TooManyMissedTimes |
Core Kubernetes · 12
Nodes
1.What is a Node ?
A Node is a cluster-scoped v1 object that stands for one machine. You rarely write it : the kubelet creates its own Node when it starts, then keeps its status fresh.
- On the machine : the kubelet, a container runtime and kube-proxy (see Architecture).
- The name is unique in the cluster. Delete a rebuilt machine's old Node before it joins again.
- The object is a record, not the machine : deleting it stops nothing.
- The kubelet signs in as
system:node:<name>, groupsystem:nodes. Ready stays False until the runtime answers and the network plugin is up.
Cloud : the cloud-controller-manager adds providerID, zone, region and instance type, and deletes the Node once the instance is gone. Karpenter and Cluster API make nodes on demand.
2.What the object holds
| Field | Holds |
|---|---|
spec.podCIDR | this node's Pod address range, if the cluster gives one per node |
spec.taints, unschedulable | who may land here |
status.addresses | Hostname, InternalIP, ExternalIP, InternalDNS, ExternalDNS |
status.conditions | Ready and the pressure flags |
status.nodeInfo | kernel, OS image, runtime and kubelet versions |
status.images | up to 50 images on the node |
3.Capacity vs allocatable
- Stock kubelet : allocatable memory is capacity minus 100Mi, CPU is all of it. Managed services reserve more.
- The scheduler adds up requests against allocatable, never usage.
podsis 110 by default (maxPods). Cloud CNIs often change it.
4.Labels and roles
| Label | Set by |
|---|---|
kubernetes.io/hostname, /os, /arch | kubelet |
topology.kubernetes.io/zone, /region | cloud, or you |
node.kubernetes.io/instance-type | cloud |
node-role.kubernetes.io/<role> | kubeadm, or you |
Old guides say node-role.kubernetes.io/master : it is control-plane now.
- The column lists the
<role>part of eachnode-rolelabel, or<none>. The value is ignored. - kubeadm sets
control-plane. Any other is yours :kubectl label node n3 node-role.kubernetes.io/worker= - A kubelet cannot give itself a role : NodeRestriction forbids it.
5.Reading kubectl describe node
| Part, in print order | What to read there |
|---|---|
| Roles, Labels, Taints, Unschedulable | which Pods may land here, and whether it is cordoned |
| Lease, Conditions | is it alive : each condition has a reason and a message. A healthy Ready : kubelet is posting ready status |
| Addresses, Capacity, Allocatable, System Info | where it is, how big, which kernel, runtime and kubelet |
| Non-terminated Pods, Allocated resources | every Pod here, then the sum of their requests and limits. Not usage : a node can be 95% allocated and idle (for usage, kubectl top node needs metrics-server) |
| Events | recent history, such as RegisteredNode and NodeReady |
1.Two heartbeats
| Heartbeat | How often |
|---|---|
Lease renewTime | every 10s |
Node status | on a change, else every 5 min |
- One Lease per node in
kube-node-lease, so a healthy node need not rewrite a large status every 10s. - No change for 50s (40s before v1.32) : the node lifecycle controller calls the node late.
2.NotReady means two things
| Ready False | Ready Unknown | |
|---|---|---|
| Written by | the kubelet | the controller |
| Means | here, but cannot run Pods (runtime down, no CNI) | nobody has heard from the kubelet |
| Taint | not-ready | unreachable |
| get nodes | NotReady | NotReady |
Other conditions : MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable. Reason and message : describe node.
3.A node dies : the timeline
- No controller setting moves the Pods. The 300s is
tolerationSeconds: 300, added at admission to each Pod without its own. It starts when the taint lands. - The containers may still run on the lost machine. Kubelet back : Ready turns True, both taints go by themselves, deleted Pods' containers are killed.
- Faster failover : give the Pods their own
unreachableandnot-readytolerations with a smallertolerationSeconds. Lab, with 20s : deleted 73s after the last heartbeat.
| At 350s | Deployment | StatefulSet | DaemonSet |
|---|---|---|---|
| its Pod is | replaced at once : a Pod marked for deletion no longer counts | stuck Terminating, not replaced : its name is still taken | never evicted : it tolerates both taints with no time limit |
4.The node protects itself
- Node-pressure eviction : past a threshold the kubelet sets a condition, the controller adds a NoSchedule taint, and the kubelet evicts Pods. At once, no PodDisruptionBudget asked.
- Memory : Pods using more than they requested go first, then by priority. No QoS class is safe, only critical Pods are spared.
- The evicted Pod stays listed : phase
Failed, reasonEvicted, STATUS oftenError.
| Signal | Evict under | Condition |
|---|---|---|
memory.available | 100Mi | MemoryPressure |
nodefs.available | 10% | DiskPressure |
nodefs.inodesFree | 5% | |
imagefs.available | 15% | |
imagefs.inodesFree | 5% | |
pid.available | no default | PIDPressure |
Hard defaults on Linux. Setting one key in evictionHard drops the others. The condition stays True 5 min after the pressure ends.
| API eviction (drain) | Taint eviction | Node-pressure | OOM kill | |
|---|---|---|---|---|
| Who | you, through the API server | taint-eviction controller | kubelet | Linux kernel |
| Budgets (PDB) | honoured | ignored | ignored | not involved |
| Pod afterwards | deleted | dead node : Terminating till it returns | Failed, Evicted, stays listed | same Pod, container restarted |
5.Shutdown
- Graceful node shutdown is on and still does nothing :
shutdownGracePeriodandshutdownGracePeriodCriticalPodsare both 0. Set them (say 30s and 10s) and the kubelet holds the systemd shutdown to stop ordinary Pods, then critical ones.
kubectl taint nodes n3 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute gets its Pods deleted, stuck and DaemonSet ones too, and volumes detached at once. Only when the node is really off : if not, old containers run beside the new. Remove the taint yourself.
1.Cordon, drain, uncordon
- cordon sets
spec.unschedulable. Running Pods stay, DaemonSet Pods still land. - drain cordons, then evicts each Pod through the Eviction API, which asks its PodDisruptionBudget. A refusal is retried every 5s, for ever unless
--timeoutis set. - uncordon clears the flag, no Pod moves back. Drain never stops the kubelet or deletes the Node.
2.What drain refuses
| Pod on the node | drain says | Flag, and what it costs |
|---|---|---|
| DaemonSet Pod | cannot delete DaemonSet-managed Pods | --ignore-daemonsets : leaves them running |
| uses an emptyDir | cannot delete Pods with local storage | --delete-emptydir-data : that data is lost |
| no controller | cannot delete Pods that declare no controller | --force : the Pod is gone for good |
| budget would break | Cannot evict pod as it would violate the pod's disruption budget. | fix the budget or add replicas (--disable-eviction ignores budgets) |
The first three stop the drain before anything is evicted. One replica with minAvailable: 1 blocks it for ever.
3.Labels, nodeSelector, a taint
$ kubectl label node n3 pool=gpu $ kubectl taint nodes n3 dedicated=gpu:NoSchedule nodeSelector: {pool: gpu} # Pod template tolerations: - {key: dedicated, value: gpu, effect: NoSchedule}
Label : Pods in. Taint : others out (Advanced Scheduling).
4.Removing a node
- Drain,
kubectl delete node, thenkubeadm reseton the machine. delete noderemoves only the object. Its Pods go about a minute later, no budget asked.- A running kubelet does not register again. After a restart it is back in seconds, without hand-added labels, taints or the cordon.
5.Version skew and upgrades
- A kubelet may be up to three minors older than the API server, never newer. Control plane first.
- Then node by node : new kubeadm,
kubeadm upgrade node, drain, new kubelet and restart, uncordon.
get on nodes/proxy reaches the kubelet API, exec included : treat it as admin (RBAC).
6.Why is it NotReady ?
| describe node shows | Cause | Look at, then fix |
|---|---|---|
| Unknown, "Kubelet stopped posting node status." | kubelet, machine or network down | journalctl -u kubelet : start it |
| False, "cni plugin not initialized" | no CNI plugin yet | ls /etc/cni/net.d : install the CNI |
| False, "container runtime is down" | containerd stopped | crictl info : restart the runtime |
| False, "PLEG is not healthy" | runtime too slow to list containers | node load, disk I/O : restart the runtime |
7.Commands
| Command | What it shows or does |
|---|---|
kubectl get lease -n kube-node-lease | One Lease per node : the heartbeat |
kubectl get pods -A -o wide --field-selector spec.nodeName=n3 | Every Pod on one node |
kubectl drain n3 --ignore-daemonsets --delete-emptydir-data | Cordon and evict. Then kubectl uncordon n3 |
kubectl debug node/n3 -it --image=busybox | A Pod with the host root at /host. Delete it after |
Core Kubernetes · 13
Quick Revision
1.Object map
apiVersion : v1 for most. apps/v1, discovery.k8s.io/v1, storage.k8s.io/v1, networking.k8s.io/v1. Ingress is GA but frozen, Gateway API is the way forward. Checked for Kubernetes v1.37 (supported : 1.37, 1.36, 1.35).
| Ch. | Object | Does what |
|---|---|---|
| 02 | Node | Machine that runs Pods |
| 03 | Pod | Containers, one IP, volumes |
| 04 | Deployment | Rolling updates over ReplicaSets |
| 04 | ReplicaSet | Keeps N Pods running |
| 05 | Service | Stable name + IP for its Pods |
| 05 | EndpointSlice | The live Pod IPs |
| 06 | ConfigMap | Plain settings, 1 MiB max |
| 06 | Secret | Sensitive values, 1 MiB max |
| 07 | PVC / PV | The claim / the disk |
| 07 | StorageClass | How new disks get made |
| 08 | Namespace | Name scope, quotas, defaults |
| 08 | Ingress | HTTP routes in (frozen) |
| add-on | Gateway API | Newer routing, as CRDs |
2.Triage : STATUS, then where to look
| STATUS | What it means | Run | Usual fix | Ch. |
|---|---|---|---|---|
| Pending | FailedScheduling : requests fit no node, an untolerated taint, an unbound PVC, a hostPort clash | kubectl describe pod <pod>kubectl get events --field-selector reason=FailedScheduling | Lower requests, add nodes, tolerate, fix the PVC | 03, 02 |
| ImagePullBackOff | Bad name or tag, image not pushed, or no imagePullSecret | kubectl describe pod <pod> | Fix the ref, push it, add a pull Secret | 03 |
| CrashLoopBackOff | Keeps exiting. Waits 10 s, 20 s, 40 s ... up to 300 s between restarts | kubectl logs <pod> --previous | Fix the app, config or probe timing | 03 |
| OOMKilled | Over the memory limit, Exit Code 137. RESTARTS climbs | kubectl describe pod <pod>kubectl top pod <pod> --containers | Raise the limit, or fix the leak | 03 |
- 0Exited cleanly
- 1The app reported an error
- 137128 + 9, SIGKILL : OOM or killed
- 143128 + 15, SIGTERM : asked to stop
- STATUS shows a container reason when there is one, else
status.phase. - ImagePullBackOff : phase Pending. CrashLoopBackOff : usually Running.
3.Commands
| Look | What it is for |
|---|---|
| kubectl get pods -o wide | STATUS, node, Pod IP. 03 |
| kubectl describe pod <pod> | Events, Last State. 03 |
| kubectl logs <pod> --previous | The run that crashed. 03 |
| kubectl events --for pod/<pod> --types=Warning | Its warnings only, the same ones describe lists. 09 |
| kubectl top pod <pod> --containers | Live CPU and memory. Needs Metrics Server |
| kubectl debug -it <pod> --image=busybox:1.28 --target=<c> | No shell in the image ? Add an ephemeral one. 03 |
| Change | What it is for |
|---|---|
| kubectl apply -f <file> | Create or update. 01 |
| kubectl diff -f <file> | Preview an apply. 01 |
| kubectl exec -it <pod> -- sh | Shell in a container, via the kubelet. 02 |
| kubectl port-forward svc/<svc> 8080:80 | localhost:8080 to one Pod behind the Service. 02 |
| kubectl rollout status / undo deploy/<d> | Wait for a rollout, or go back one revision. 09 |
| kubectl scale deploy/<d> --replicas=3 | Set replicas by hand. An HPA overrides it. 04 |
1.Requests and limits
Set per container, for CPU and for memory.
- requestWhat kube-scheduler reserves. The Pod lands only where the sum of requests fits, even if real use is low.
- limitA ceiling the kernel enforces with cgroups.
- Going above the request is fine while the node has room.
- Limit only ? The API server copies it into the request, so a Pod with only CPU and memory limits is still Guaranteed.
- LimitRange : namespace defaults, min and max, applied at admission. Break them, get 403 Forbidden.
See : scheduler 02, HPA % of requests 04, LimitRange and ResourceQuota 08.
resources:
requests: { cpu: "250m", memory: "128Mi" }
limits: { cpu: "500m", memory: "256Mi" } # 0.5 CPU
2.QoS classes
Kubernetes derives status.qosClass from the resources. You never set it, and it is fixed once the Pod is created.
- Pod-level
spec.resources(Beta since v1.34, on by default) count too.
| Class | Rule | Under node pressure |
|---|---|---|
| Guaranteed | Every container sets CPU and memory, request = limit | Evicted last, never over its requests |
| Burstable | Some CPU or memory request or limit, not Guaranteed | Early if over its requests, late if under |
| BestEffort | No CPU or memory request or limit anywhere | Early, no request to stay under |
- Using more than its requests ? Goes first.
- Then lower Priority first.
- Then the furthest over its requests.
- An in-place resize (stable since v1.35) that would change the class is rejected.
3.Labels vs annotations
| Labels | Annotations | Ch. | |
|---|---|---|---|
| Purpose | Identify and group objects | Describe objects for tools and people | 04, 09 |
| Selected by | Services, workloads, kubectl -l | Never | 05, 04 |
| Selectors | A Service matches on equality only. Deployment, ReplicaSet, Job and DaemonSet also take In, NotIn, Exists, DoesNotExist. A Deployment's selector is immutable and must match its Pod template labels. | 05, 04 | |
| Key | [prefix/]name, name 63 chars max. kubernetes.io/ and k8s.io/ are reserved. Same rules for both. | 04 | |
| Value | 63 chars max, may be empty | Any string, 256 KiB in total per object | 04 |
| Example | app: web | kubernetes.io/ | 04, 09 |
| Recommended | Prefix app. plus nameinstanceversioncomponentpart-ofmanaged-by | none | |
Part 2Beyond Core
Core Kubernetes gives you Pods, Services and a control loop. These 29 topics cover the add-ons that plug into that loop through CRDs and controllers, the CI that builds images, and the signals that show what it is doing.
- Why it exists
- Components
- Diagram
- Commands
- In short
- Each add-on solves one gap in core
- Most are CRDs plus a controller
- Install order matters (CRDs first)
- CI and image builds run in Pods too
- Flux or Argo CD can deliver every add-on
- Signals: metrics, logs, traces, profiles
GitOps and delivery
Flux CD v2
1.What is Flux ?
Flux watches a Git repo (or OCI image or bucket) and keeps the cluster matching it, forever, by itself.
You push to Git, Flux applies it. No kubectl apply from laptops.
- CNCF Graduated (Nov 2022)
- Latest : Flux v2.9.6
- Helm, Kustomize and plain YAML all work
- Runs in the
flux-systemNamespace
Sources say where (Git, OCI, Helm). Appliers say what (Kustomization, HelmRelease).
2.Why GitOps ?
GitOps = Git is the single source of truth, declarative, pulled automatically, continuously reconciled.
- Pull model : CI needs no cluster credentials
- Drift : live edits get reverted by Flux
- Rollback : just
git revert - Many clusters : one repo, different paths
- Reconcile runs every
spec.interval - Also runs on a webhook (Receiver)
prune: true: delete in Git, deleted in cluster- Per-Kustomization
serviceAccountNamelimits a team kubectl get fluxcd -Ashows all- Flagger, a sister project, adds canary and blue/green
3.The Controllers (Toolkit)
Small controllers, each owns its CRDs. Composable like Lego.
Dashed ones are optional.
1.The Reconcile Loop
Drift gets undone at the next loop. A failure keeps the last good state and retries.
- Fetch source
- Build (kustomize / helm template)
- Diff vs live (dry-run SSA)
- Apply, only if drift or new revision
- Health check, set status Ready
- Wait interval, loop
2.Getting Started
flux bootstrapinstalls controllers in flux-system- Define a source (GitRepository, OCIRepository, HelmRepository)
- Define what to apply (Kustomization or HelmRelease)
- Push to Git
- Flux fetches, applies, runs health checks
- Drift and deletions fixed at next interval
- notification-controller tells Slack or Git
- Use
flux create source git ... --exportto make YAML you commit. - After bootstrap,
flux checkshould show every controller Ready.flux get all -Alists the rest.
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata: {name: app, namespace: flux-system}
spec:
interval: 1m
url: https://github.com/example/app
ref: {branch: main}
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: {name: app, namespace: flux-system}
spec:
interval: 10m
path: ./deploy
prune: true
wait: true
timeout: 3m
sourceRef: {kind: GitRepository, name: app}3.CRD Cheat Sheet
| Kind | apiVersion | Controller | What it does |
|---|---|---|---|
| GitRepository | source.toolkit.fluxcd.io/v1 | source | Git branch, tag, semver or commit |
| OCIRepository | source.toolkit.fluxcd.io/v1 | source | OCI artifact as source |
| HelmRepository | source.toolkit.fluxcd.io/v1 | source | Helm chart index (http or oci) |
| Bucket | source.toolkit.fluxcd.io/v1 | source | S3-compatible bucket |
| Kustomization | kustomize.toolkit.fluxcd.io/v1 | kustomize | Apply a path from a source |
| HelmRelease | helm.toolkit.fluxcd.io/v2 | helm | Declares a Helm release |
| Provider / Alert | notification.toolkit.fluxcd.io/v1 | notification | Where and which events to send |
| ImagePolicy | image.toolkit.fluxcd.io/v1 | image-reflector | Choose latest tag |
Check versions with kubectl api-resources | grep toolkit
1.Image Automation
Flux can write new image tags back to Git. Then the normal pull flow deploys them.
flux bootstrap ... --components-extra2.flux Commands
| Command | Description | Example |
|---|---|---|
flux check --pre | Check the cluster meets requirements (K8s v1.34.1+) | flux check |
flux bootstrap github | Install controllers, commit to Git, self-manage | flux bootstrap github --owner=loren --repo=fleet --path=clusters/prod |
flux get all -A | Status of every Flux object | flux get kustomizations |
flux reconcile kustomization <name> | Sync now | flux reconcile kustomization flux-system --with-source |
flux logs --level=error -A | Controller logs | flux logs -f --kind=Kustomization --name=apps |
flux trace deployment/<name> | Who owns this object | flux trace deployment/podinfo -n default |
flux suspend / resume kustomization | Pause for a live hotfix | flux suspend kustomization apps |
flux diff kustomization <name> | Preview changes vs live | flux diff kustomization apps --path=./apps |
3.Flux vs Argo CD vs CI Push
- Two things named Kustomization (Flux vs
kustomize.)config. k8s. io - v1beta2 / v2beta2 removed in v2.8, run
flux migrate - Hotfix ?
flux suspendfirst, or the next loop reverts it prune: truedeletes what you drop from Git- Encrypt secrets (SOPS or External Secrets)
dependsOnneedswaitto really wait- Interval minimum is 60 seconds
- Stuck Ready=False ? Read the message
- Repo layout, health, drift, multi-tenancy : Flux deep dive
1.Flux Operator in brief
- An operator from ControlPlane that installs and upgrades Flux from one
FluxInstancenamedflux, instead offlux bootstrap. distribution.version: "2.9.x"floats patches only. Pin the minor on purpose.- A
FluxReportholds the observed state. AGPL-3.0, pre-GA (0.x). More : Flux deep dive.
2.The Flux Web UI
In the operator since v0.37.0, on by default, plain HTTP on port 9080 (Service flux-operator, leader pod only).
- ShowsFlux objects, workloads, logs, events, graph, history
- DoesReconcile, suspend, resume, download, restart, run job, delete pod
- NeverEdits manifests : Git is the way in
3.Login modes and roles
| Mode | Runs as | Actions |
|---|---|---|
| None | Operator SA (cluster-admin) | Off |
| Anonymous | One shared identity | By RBAC |
| OAuth2 (OIDC) | Each user, via CEL | By RBAC |
- flux-web-user
get, list, watchon*/*: Secrets too - flux-web-adminSame, plus
reconcile, suspend, resume, download, restart
valuesFrom Secrets in clear. RBAC has no "except" : list core minus secrets plus every other group, and keep get on ResourceSets.4.How a request is authorized
- No session : 401, then
/oauth2/authorizeanswers 303,Locationon the IdP (code + PKCE). - Code to
/oauth2/callback, ID token verified. - CEL
validationspass; claims become user and groups (both empty : refused). - Calls go as the operator SA, with impersonation.
- RBAC answers for the user. Buttons first run a SelfSubjectAccessReview.
Impersonatedalso needsget+patch,FineGrained(v0.51.0) only the verb.
A removed group keeps access until the ID token expires. Keycloak : Full group path off, so groups read team, not /team, as in RBAC. See OIDC auth; Argo CD has its own policy.
5.Exposing it safely
- No auth :
port-forwardonly. Else its own host : HTTPRoute toflux-operator:9080, TLS at the Gateway. - FluxInstance
cluster.networkPolicy: true(default) admits onlyflux-systempods.install.yamlhas no web policy, so admit the proxy pods to 9080 (NetworkPolicy). The chart's policy admits all namespaces. - Config holds
clientSecret: a SOPS Secret, hot-reloaded.
6.Commands
| Command | Does |
|---|---|
kubectl -n flux-system port-forward svc/flux-operator 9080:9080 | Open the UI on localhost:9080, nothing exposed |
flux-operator create secret web-config ui --base-url=... --issuer-url=... --client-id=... --client-secret-stdin --export | Config Secret as YAML (OIDC by default). Encrypt it |
kubectl auth can-i list secrets --as=jane --as-group=viewers | What a UI user may do (should be no) |
GitOps and delivery
Argo CD
1.What is Argo CD ?
A GitOps continuous delivery tool for Kubernetes. The desired state lives in Git (or a Helm or OCI repo). A controller keeps comparing it with the live cluster, and can sync the cluster back to it.
- Pull model : Argo CD pulls from Git, so CI needs no cluster credentials.
- Renders Kustomize, Helm, Jsonnet, plain YAML, or a plugin.
- Extras : web UI, CLI, SSO and its own RBAC on top of Kubernetes.
- Lives in
argocd, drives one or many clusters.
2.Words to know
| Word | Meaning |
|---|---|
| Target state | What Git says should run |
| Live state | What really runs in the cluster |
| Sync status | Synced, OutOfSync or Unknown |
| Sync | Make live match target (apply) |
| Refresh | Re-read Git and diff, no apply. Hard refresh also drops the cache |
| Health | Is it running and serving ? A separate answer : Synced can still be Degraded |
Argo project, CNCF graduated 2022-12-06. Latest v3.5.4 (2026-10-06), tested on Kubernetes v1.33 to v1.36. The three newest minors get patches. 3.6 is a release candidate.
3.The parts
A few workloads; Argo's own share one image (quay.). Real state lives in CRs, ConfigMaps and Secrets.
- repo-server clones and renders. No Kubernetes rights, no stored credentials.
- application-controller is a StatefulSet and a classic controller. Scale it to shard clusters.
- notifications-controller sends alerts. dex is optional with direct OIDC. redis is safe to lose.
4.Detection vs correction
Argo CD always notices drift. Fixing it is opt-in, one switch at a time.
- Every 120s (plus up to 60s jitter), or at once on a Git webhook, the controller refreshes each app.
- repo-server renders the manifests, the controller diffs them with the live objects, and marks the app Synced or OutOfSync. That is detection.
- Correction is a sync, and it is manual by default : press Sync, or run
argocd app sync.
| syncPolicy | New commit in Git | File removed from Git | Someone runs kubectl edit |
|---|---|---|---|
| none (default) | OutOfSync, waits for you | OutOfSync, waits for you | OutOfSync, waits for you |
| automated | Synced by itself | Keeps running, stays OutOfSync | Stays OutOfSync, not reverted |
| + prune: true | Synced by itself | Deleted from the cluster | Stays OutOfSync, not reverted |
| + selfHeal: true | Synced by itself | Deleted only if prune is on too | Reverted to Git |
OpenGitOps asks for "continuously reconciled". Out of the box Argo CD only observes continuously. Flux fixes drift on every interval.
5.Turning it on
spec:
syncPolicy:
automated:
enabled: true # 3.1+, unset = on
prune: true # default false
selfHeal: true # default false
CLI : argocd app set APP --sync-policy automated --auto-prune --self-heal. enabled: false pauses auto-sync but keeps the flags.
- Auto-sync tries once per commit SHA (plus parameters). A SHA that failed is not tried again.
- Self-heal waits a backoff : 2s, times 3, max 300s. Docs saying "5 seconds" are stale.
- No
retryblock : auto-sync retries 5 times (5s doubling, max 3m). Aretryblock withoutlimit: 0 retries. - No rollback while auto-sync is on.
1.The three CRDs
All are argoproj. only (no beta, no v1), namespaced, and live in argocd by default. See CRDs.
| Kind (short) | Job | Key fields |
|---|---|---|
| Application (app) | Source(s) to one destination | project, source, destination, syncPolicy |
| ApplicationSet (appset) | Template that stamps out many Applications | generators, template, strategy |
| AppProject (appproj) | Guard rails : what may go where | sourceRepos, destinations, roles, syncWindows |
2.An Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: guestbook
namespace: argocd
finalizers: [resources-finalizer.argocd.argoproj.io]
spec:
project: default
source:
repoURL: https://github.com/my-org/apps.git
path: guestbook # targetRevision unset = HEAD
destination:
server: https://kubernetes.default.svc
namespace: guestbook
- targetRevision : a branch, a tag, a semver range (matches tags only) or a commit SHA.
- sources : a list instead of
source, e.g. a Helm chart plus values from your repo. - destination :
serverURL or clustername, never both. - finalizer : without it, deleting the Application leaves every resource running. See finalizers.
- project : required. The
defaultproject allows anything. A project that may deploy into argocd is as good as admin.
3.Sync phases and waves
- Order : phase, wave (default 0, negatives allowed), kind (Namespaces first, custom resources last), name.
- Apply the first wave that is not yet synced and healthy.
- Wait for Healthy, pause 2s, next wave. Prunes go in reverse wave order.
4.Hooks
kind: Job # batch/v1 metadata: name: db-migrate annotations: argocd.argoproj.io/hook: PreSync argocd.argoproj.io/hook-delete-policy: HookSucceeded argocd.argoproj.io/sync-wave: "-1" spec: ... # a normal Job
- Kinds : PreSync, Sync, PostSync, SyncFail, Skip. On app deletion : PreDelete (3.3+), PostDelete.
- No delete policy means
BeforeHookCreation. Hooks skip a selective sync. - Helm : Argo CD runs
helm template, sohelm lsshows nothing. pre-install / pre-upgrade become PreSync, post-install / post-upgrade become PostSync.
5.Health
| Kind | Healthy when |
|---|---|
| Workloads | updated replicas = desired (rollouts) |
| LB Service, Ingress | LB hostname or IP in status |
| PVC | phase is Bound |
| Job, CronJob | Complete. A failed Job, or a CronJob whose last run failed (3.2+), is Degraded |
| Custom resources | 238 bundled Lua checks, or yours |
| Application | no check (since 1.8) |
- App health = worst direct child. Best to worst : Healthy, Suspended, Progressing, Missing, Degraded, Unknown.
- Own check : Lua in
argocd-cm, keyresource..customizations. health.<group>_<kind> - App of apps : add one for
argoproj., or waves never wait for a child app.io_ Application
1.ApplicationSets
Generators make parameter sets. The template is rendered once per set into an Application.
| Generator | Params from | Re-reads |
|---|---|---|
| list | fixed values | never |
| clusters | clusters, by label | Secret events |
| git | dirs or files in a repo | every 3 min |
| matrix, merge | child generators combined | fastest child |
| pullRequest | open PRs | every 30 min |
Also scmProvider, clusterDecisionResource, plugin. 3.6 (RC) adds oci. RollingSync (Beta) syncs the apps step by step and turns their auto-sync off.
- Bootstrap : since 3.4 the docs recommend it, with the cluster generator, over app of apps (Cluster API).
- Admin only : it can create Applications in any project. Put a template variable in
projectonly if admins own every generator source.
2.Install : server-side since 3.3
$ kubectl create namespace argocd $ kubectl apply -n argocd --server-side --force-conflicts \ -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.4/manifests/install.yaml
kubectl apply fails. See SSA.| Manifest | For |
|---|---|
| install.yaml | trying it, not production |
| ha/install.yaml | production, 3+ nodes |
| namespace-install | one namespace, no CRDs |
| core-install.yaml | no API server, UI or SSO |
--force-conflictsoverwrites live edits. Customize with Kustomize patches.- Self-managed Argo CD needs
ServerSideApply=true.
3.Commands
| Command | Does |
|---|---|
argocd admin initial-password -n argocd | First admin password. Change it, then delete its Secret |
kubectl port-forward svc/argocd-server -n argocd 8080:443 | Reach the API unexposed, then argocd login localhost:8080 |
argocd app get my-app --hard-refresh | Status. Hard refresh also drops the manifest cache |
argocd app diff my-app | Git vs live. Exit 0 same, 1 different, 2 error |
argocd app sync my-app --prune --dry-run | Preview a sync that also deletes extras. Drop --dry-run to run it |
argocd app history my-app, argocd app rollback my-app ID | Roll back to a past sync. Not while auto-sync is on |
argocd appset generate appset.yaml | Show the Applications an ApplicationSet would make |
argocd cluster add CONTEXT | Register a cluster (adds an admin-level argocd-manager SA) |
4.Argo CD vs Flux
| Topic | Argo CD | Flux |
|---|---|---|
| UI | Built in | In Flux Operator |
| Users | Own SSO + RBAC | Kubernetes RBAC |
| Apply | Client-side, SSA opt-in | Server-side apply |
| Drift | Seen always, fixed if selfHeal | Fixed every interval |
| Helm | helm template, no release | Real Helm releases |
Both CNCF graduated in 2022. New image tags into Git : Argo CD Image Updater, a separate argoproj-labs controller.
3.0 tightened RBAC defaults. 3.5 renders with Helm 4 only. 3.6 (RC) diffs ServerSideApply apps server-side.
5.Gotchas
ignoreDifferencesonly hides the diff. Sync still writes those fields unlessRespectIgnoreDifferences=true.- An HPA or KEDA owns replicas ? Leave
replicasout of Git. - Secrets injected while rendering are cached in plain text in Redis. Keep them in the cluster with External Secrets.
- Since 3.0 Argo CD tracks resources with an annotation, not a label. Sync every app once after upgrading.
GitOps and delivery
Argo CD Image Updater
1.What is Image Updater ?
A controller from argoproj-labs. It watches container registries and bumps the image tag (or digest) of Argo CD Applications. Argo CD then syncs it. Flux users have image automation instead.
- Apps built with Helm, Kustomize or a Plugin (v1.3.0+). Plain YAML and Jsonnet apps are skipped.
- It never touches the Deployment, only the Application source. Argo CD renders the rest.
- Runs in the cluster that holds the Applications (the hub), not in workload clusters.
| v0.x (legacy) | v1.x (now) | |
|---|---|---|
| Config | Annotations on each app | An ImageUpdater CR |
| API | K8s, or Argo CD API + token | Kubernetes API only |
| Last release | v0.18.0 (2025-12-25) | v1.3.0 (2026-08-13) |
Bridge : useAnnotations: true on a ref (v1.1.0+) reads the old annotations, minus PR mode, plugin targets and cosign checks.
v1alpha1, and the README says not yet for critical production. One replica, polls every 2m.2.How one check works
- Trigger : the 2m timer, a CR spec change, or a registry webhook. Editing an Application does not.
- Match : Applications in the CR's namespace that fit
namePatternandlabelSelectors. - Live ? : the image, registry included, must be in the app's
status., else skipped (unlesssummary. images forceUpdate). - Pick : list tags, filter, the strategy chooses.
- Write : to the Application spec or to Git, plus an
ImagesUpdatedEvent. - Sync : auto-sync rolls it out. Without it the app just shows OutOfSync.
3.The ImageUpdater CR
apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata: {name: nginx-patches, namespace: argocd}
spec:
applicationRefs:
- namePattern: "web-*" # glob
labelSelectors: {matchLabels: {team: web}}
images:
- alias: nginx
imageName: nginx:1.26.x # a constraint
commonUpdateSettings: {updateStrategy: semver}
manifestTargets:
helm: {name: image.repository, tag: image.tag}
# no writeBackConfig = method argocd
4.Key fields
- applicationRefs : which apps.
namePattern(glob, required), optionallabelSelectors. - images : an
alias(no "/") and animageName, the repo plus a version constraint. - manifestTargets : one of helm, kustomize, plugin. Helm defaults to
image.name/image.tag, most charts wantimage.repository. - commonUpdateSettings : strategy, tag filters, forceUpdate, pullSecret, platforms. At spec, ref or image level, the most specific field wins.
- writeBackConfig : argocd or git, at spec or ref level, never per image.
- imagesVerification : cosign public key check (v1.3.0+).
5.Matching rules
- A CR only sees Applications in its own namespace.
spec.namespacewas removed in v1.2.0, using it fails. - Inside one CR the most specific ref wins : an exact name beats a glob, labels add weight.
- One CR per Application. Two CRs on one app overwrite each other every cycle.
1.The tag is a constraint
For semver, the tag in imageName is a version range (Masterminds semver), not the version you run today.
| Tag | Means | Picks |
|---|---|---|
| nginx:1.26.3 | Exact pin | Never moves |
| nginx:1.26 | ~1.26, so >=1.26.0 <1.27.0 | Newest 1.26.z |
| nginx:1.26.x | Any 1.26 patch | 1.26.3 (5 of 1209 fit) |
| nginx:1.x | Any 1.y.z | Newest 1.y.z |
| nginx | * : no pre-releases (v1.3.0+) | Newest, even a new major |
| app:2.x-0 | 2.x with pre-releases | rc and beta too |
2.The pin trap
- Writing today's version pins it forever, with no error.
- Two doc examples do it :
nginx:1.17.10(PR mode) andbackend:1.0.0(cosign sample). 1.27.5-alpinecounts as a pre-release, so a plain range skips it. Tags that are not semver are skipped too.- Before v1.3.0 an empty tag could jump to
-rctags.
3.Filters first, then the strategy
4.Tag filters
- allowTagsOne string,
regexp:<expr>orany. Empty allows all, a bare regex matches nothing. Unanchored : add^and$. - ignoreTagsAn array of globs, a plain string is rejected.
["*"]pauses the image. - platformsAn array. Default is the controller's own OS / arch, so an amd64 controller with arm64 nodes needs it (newest-build, digest).
images:
- alias: myimage
imageName: some/image # no range here
commonUpdateSettings:
updateStrategy: newest-build
allowTags: "regexp:^[0-9a-f]{7}$"
ignoreTags: ["latest", "master"]
platforms: ["linux/amd64", "linux/arm64"]
5.Four update strategies
| Strategy | Old name | Picks | Tag means | Registry work |
|---|---|---|---|---|
| semver (default) | - | Highest version in the range | The range | Tag list, cached |
| newest-build | latest | Newest image creation date | Nothing, use allowTags | A manifest per tag |
| alphabetical | name | Last tag in byte order | Nothing, use allowTags | Tag list, cached |
| digest | - | Digest of one mutable tag, writes tag@sha256: | The tag to watch | A manifest every cycle |
- Old names still work, with a warning. A typo like
boguspasses the API server and runs semver. calverexists on master only.
- newest-build uses the build date, not the push date. On a tie lexical order decides, so
latestcan win. - alphabetical :
2026-9-01sorts above2026-10-03.
6.Write-back : argocd vs git
| argocd (default) | git | git + PR (v1.2.0+) | |
|---|---|---|---|
| Writes | Application spec, via the K8s API | A commit on the branch | A new branch plus a GitHub PR or GitLab MR |
| Changes | Helm parameters, Kustomize images, plugin env | ., kustomization or values | Same files, after review |
| Creds | None, RBAC on Applications | Argo CD repo creds or git:secret:<name> | HTTPS token or GitHub App, no SSH |
| Lasts ? | No : a re-created app, app-of-apps or ApplicationSet can revert it | Yes, it is in Git | Yes, once merged |
1.Git and pull request write-back
spec:
writeBackConfig:
method: git:secret:git-creds # Secret here
gitConfig:
branch: main # base only
writeBackTarget: helmvalues:/helm/values.yaml
pullRequest: {github: {}} # or gitlab
- (no target)
<path>/., merged by Argo CD at render time.argocd-source-<app>. yaml - kustomizationEdits the images in
kustomization.. Takes a directory, not a file.yaml - helmvaluesEdits a values file. Bare means
<path>/.values. yaml
Plain method: git reuses Argo CD's repo creds. A git:secret: Secret must sit in the CR's namespace.
- Argo CD deploys only the branch the app tracks. Tracking
HEAD, a tag or a SHA ? SetgitConfig.branch. Amain:<tpl>write branch is force-pushed, PR mode refuses it.
2.Alpha : sharp edges in v1.3.0
- Registry auth (issue 1797) : if
/v2/answers without auth but repos need it (zot), creds are never sent. Every cycle gets 401. The fix (PR 1801) is on master only. - Stored defaults : a ref or image
commonUpdateSettingsblock is saved withupdateStrategy: semver. Repeat your strategy in every block. - READY lies :
Ready=Truecan sit next toError=True. git:secret:SSH or password creds skip host-key and TLS checks.
3.What changed when
| Version | What changed |
|---|---|
| v1.0.0 | CR replaces annotations |
| v1.2.0 | spec.namespace gone, PR mode, status |
| v1.3.0 | Plugin apps, cosign, Secrets in the CR's namespace, PR branch renamed |
| master | Auth fix, no stored defaults, calver |
No Argo CD or Kubernetes version matrix is published. CHANGELOG.md is stale, read the GitHub release notes.
4.Image Updater vs Flux vs Renovate
| Image Updater | Flux image automation | Renovate | |
|---|---|---|---|
| Runs | In the cluster, beside Argo CD | In the cluster, two controllers | In CI, outside the cluster |
| Config | One CR, v1alpha1 | Three CRs, v1 since Flux v2.7.0 | Config file, argocd manager opt-in |
| Finds images | App params, kustomization | Setter comments in any YAML | Manifests in Git |
| Writes | App spec (no Git), commit or PR | Always a Git commit | Pull requests |
| Rule of thumb | Argo CD shop | Flux shop, plain YAML | Review every bump |
5.Commands
| Command | What it does |
|---|---|
kubectl apply -n argocd -f https://raw.githubusercontent.com/ | Install CRD, RBAC and controller next to Argo CD (or the argo/ chart) |
kubectl apply --dry-run=server -f iu.yaml | Check a CR against the CRD rules, catches a string ignoreTags |
kubectl get imageupdater [<name> -o yaml] -n argocd | APPS, IMAGES, LAST CHECKED, READY. The YAML shows the stored updateStrategy and the Error condition |
kubectl -n argocd logs deploy/ | Logs. Set log.level: debug in the ConfigMap, then restart |
argocd-image-updater test nginx --semver-constraint 1.27 | Try a range, writes nothing : nginx:1.27.5. Needs a kubeconfig |
kubectl patch imageupdater <name> -n argocd --type merge -p '{"metadata":{"finalizers":null}}' | Drop the finalizer of a CR stuck Terminating after the controller went first |
CI and builds
Actions Runner Controller
1.What is ARC ?
- GitHub's Kubernetes operator for self-hosted Actions runners, and the reference implementation of its scale set APIs. This sheet is ARC 0.15.0 (
gha-runner-scale-setcharts), pre-1.0, CRDsv1alpha1. - Two Helm charts published only as OCI packages, plus one controller image with no
latesttag : always pass--version. Runner Pods are ephemeral : one job, then deleted. - Scale set mode is the only one GitHub supports. Legacy ARC (
actions.summerwind.dev) is community-only, with no controller release since v0.27.6. Moving over is a rebuild, not an upgrade.
2.The pieces
| Kind | One per | Does |
|---|---|---|
AutoscalingRunnerSet | scale set (Helm release) | Wanted state : URL, Secret, min and max, Pod template, mode |
AutoscalingListener | scale set, controller namespace | The listener Pod, its ServiceAccount and config Secret. Role and RoleBinding sit in the runner namespace |
EphemeralRunnerSet | scale set, same name | The replica count the listener patches |
EphemeralRunner | runner | One bare Pod, its JIT config, its phase |
All four are namespaced, actions.. Runner Pods are not Kubernetes Jobs (Jobs).
3.From queued job to runner Pod
- The listener long-polls over HTTPS : back at once if a message waits, else after about 50s.
- The message carries the assigned jobs count.
- The listener patches replicas with its own ServiceAccount.
- The controller asks for a JIT config per runner. The Pod gets only
ACTIONS_, ARC never injects the PAT or App key.RUNNER_ INPUT_ JITCONFIG - The runner registers with it, runs one job, then the Pod is deleted at once (0.15.0). Runners are created up to 8 at a time.
A job no runner takes is requeued up to 3 times, and GitHub gives up after 24 hours. No webhook, no REST polling budget.
4.The scaling rule
desired = min(minRunners + jobs, maxRunners)
Example, min 2 and max 10 : 5 assigned jobs give 7 runners, 12 give 10. No schedule is built in : patch minRunners from a CronJob (Jobs).
minRunners is idle runners kept warm, not a floor on total Pods. Unset maxRunners means no cap. Both at 0 is a drain : no runners at all.5.Targeting a scale set
runs-on: <scale set name>: the Helm release name orrunnerScaleSetName, at most 45 characters.- Extra labels (0.14.0+) use the chart key
scaleSetLabels, not the docs'runnerScaleSetLabels, and reach GitHub only at creation. self-hostedis never added for you.
jobs:
build:
runs-on: example-set
1.Install
| Command | Does |
|---|---|
helm install arc -n arc-systems --create-namespace --version 0.15.0 oci:// | Controller and CRDs, own namespace |
helm install arc-runner-set -n arc-runners --create-namespace --version 0.15.0 -f values.yaml oci:// | One scale set, another namespace. Name = runs-on target |
kubectl create secret generic gh-app -n arc-runners --from-literal=github_app_id=... --from-literal=github_app_installation_id=... --from-file=github_app_private_key=key.pem | App Secret for githubConfigSecret |
2.Auth to GitHub
| Credential | Level | Needs |
|---|---|---|
| GitHub App (recommended) | repo or org | repo : Administration read and write, Metadata read. org : Self-hosted runners read and write |
| Classic PAT | repo, org, enterprise | repo, admin:org, manage_. The only option for enterprise sets |
| Fine-grained PAT | repo or org | repo : Administration read and write. org : Administration read, Self-hosted runners read and write |
- Create the Secret yourself in the scale set namespace and reference it by name. IDs are quoted strings, never
--set. A copy also lands in the controller namespace (listener config). A node clock ahead of GitHub gives 401.
3.Container modes
| containerMode.type | Pod shape | Privileged | Runner's Kubernetes rights |
|---|---|---|---|
| unset | runner container only | no | none |
dind | runner + dind sidecar | yes | none |
kubernetes | runner + -workflow Pod, PVC | no | Pods, exec, logs, Jobs, Secrets |
kubernetes-novolume (0.13.0+) | as above, no PVC | no | same, no batch/jobs |
- dind is privileged : it needs the privileged level and is node root (Image Builds). A native sidecar.
kubernetesneedskubernetesMode, or the render hasWorkVolumeClaim spec: nulland the API server rejects it.- Every job needs a job container (
ACTIONS_). InRUNNER_ REQUIRE_ JOB_ CONTAINER kubernetesmode the workflow Pod is pinned to the runner's node.
4.The runner image
ghcr.io/: Ubuntu 24.04, UID 1001,actions/ actions-runner dockergroup GID 123, passwordless sudo, Docker CLI and Buildx. The daemon binaries are there, but nothing starts them.- ARC turns runner self-update off, so the image tag is the runner version. GitHub stops sending jobs to a runner more than 30 days behind : pin and bump, or float
latestwithimagePullPolicy: Always(pulls).
5.Rendering for GitOps
- controllerServiceAccount : set it, or
helm templatefails ("No gha-rs-controller deployment found"). - CRDs (0.6 to 1.3 MB each) : client-side apply fails,
may not be more than 262144 bytes. Use server-side apply (API machinery). - Argo CD :
flags., or it fights ARC's children.excludeLabelPropagationPrefixes: ["argocd.argoproj.io/instance"] - Delete order : scale sets first, controller last, or finalizers strand objects (finalizers). In Flux,
dependsOnthe controller.
1.Security model
- Never on public repositories : anyone who can open a pull request runs code. Ephemeral runners stop persistence, not a job reading its secrets.
- dind is node root (Image Builds). In the kubernetes modes the runner's Role can read Secrets in its namespace, the App key included : keep
ACTIONS_, one scale set per namespace.RUNNER_ REQUIRE_ JOB_ CONTAINER=true - Workflow Pods run as the namespace's
defaultServiceAccount : setserviceAccountNamein a hook template and bind it narrowly (RBAC). - Write an egress policy : DNS, 443 out, block
169.254.0.0/16. Workflow Pods carry another label, so they need their own (NetworkPolicy).
2.Upgrades and the version guard
A minor upgrade is a reinstall :
- Uninstall every scale set, wait for cleanup.
- Uninstall the controller.
- Delete the four
actions.CRDs.github. com - Reinstall both charts at the same version.
Applying both new charts at once lets the old controller delete the new sets.
- Patch upgrades update in place since 0.15.0. Every minor since 0.11.0 changed all four CRDs, and Helm never upgrades CRDs. Flux does only with
upgrade.crds: CreateReplace(CRDs, Flux deep dive). - An outdated runner exits with code 7. 0.15.0 parks the whole set
Outdateduntil the runner template changes (a new image does,maxRunnersdoes not).
--set flags instead of the full values. The chart has no values schema, so a missing work volume claim renders null and helm template raises no error. Do better : keep every value in one file, render both charts from it, diff the full set spec, grep for null.3.Watching it
- Since 0.15.0
kubectl getshows only min, max and phase : counts live in metrics, which the chart turns off (no Service either). Set themetrics:block and scrape listeners by Pod label (Prometheus). - Gauges :
gha_,assigned_ jobs gha_,desired_ runners gha_,busy_ runners gha_, and newcontroller_ failed_ ephemeral_ runners gha_. Opt-incontroller_ outdated_ ephemeral_ runners job_workflow_reflabels add a series per pull request.
4.ARC, KEDA or a fixed pool
| ARC scale sets | KEDA github-runner | Gitea Runner | |
|---|---|---|---|
| Learns about jobs | listener long poll, told the count | REST API every pollingInterval | runner polls Gitea |
| Scales | 0 to N, one Pod per job | ScaledJob or Deployment | fixed pool you size |
| Registration | JIT per runner, by the controller | your runner image registers | token, then .runner |
| Rate limits | none on the scaling path | 5,000/h PAT, 15,000/h App | n/a |
KEDA adds self-hosted, linux, x64 labels by default, ARC never adds self-hosted. Drains and OOM kills end jobs : karpenter.sh/do-not-disrupt helps with autoscalers, not kubectl drain (Karpenter, limits).
5.Commands
| Command | Does |
|---|---|
kubectl get autoscalingrunnersets -n arc-runners | Min, max and phase per set |
kubectl get pods -n arc-systems -l app.kubernetes.io/ | One listener per set |
kubectl logs -n arc-systems -l app.kubernetes.io/ | Controller logs (default names) |
kubectl patch autoscalingrunnerset <name> -n arc-runners --type=merge -p '{"spec":{"minRunners":2}}' | Warm runners, from a CronJob |
CI and builds
Gitea Actions Runners
1.Gitea Actions in brief
- Built into Gitea, with GitHub Actions workflow syntax. Files live in
.gitea/(elseworkflows .github/). On since Gitea 1.21. These notes use Gitea 28.1.0 and Gitea Runner 5.0.0.workflows - The server queues jobs, runners execute them. No CRDs or controller (contrast ARC).
act_runnerbecame Gitea Runner at v1.0.0 (2026-05-05) : imagegitea/(the old one stopped at 0.6.1).runner
2.The pull model
- The runner connects out to Gitea and polls every 2s, backing off to 5s when idle. Gitea never dials it : no Service, no inbound port. Offline after a minute.
- Plan four connections : runner and job containers to Gitea (never register with
localhost), runner and job containers to the internet (actions come from github.com unlessDEFAULT_). See NetworkPolicy.ACTIONS_ URL = self
3.Registering a runner
| Level | Takes jobs from | Token in the UI |
|---|---|---|
| Instance | every repository | /-/ |
| Organization | that org's repositories | /org/ |
| User | that user's repositories | /user/ |
| Repository | that repository | /<owner>/ |
- The docs print the org path without
/org/: a 404. A token is reusable until reset. registertrades it for the runner's own credential, written to .runner. Persist it, never copy it : lose it and the runner registers anew, leaving a stale offline entry.- Ephemeral runners (Gitea 1.24+) take one job, credentials revoked on assignment. You start them.
4.Labels pick where a job runs
| Label | Job runs | Needs |
|---|---|---|
ubuntu-latest: | in a container from that image | a Docker daemon |
ubuntu-latest: | as its own Pod (v4.1.0+) | Kubernetes 1.30+, a Role, an image with sh, tar, tail, env |
linux-amd64: | on the runner's machine, no isolation | tools installed there; no service containers unless the job sets container: |
- Grammar
<name>[:<schema>[:<args>]]. A runner gets a job only if it has everyruns-onlabel. Since 5.0.0 a job withoutruns-onfails. No match : it waits, then is cancelled after 24h. - Labels come only from the runner (config,
GITEA_orRUNNER_ LABELS --labels), re-declared on every start. Gitea cannot edit them. Never call a host labelubuntu-latest.
5.Commands
| Command | Does |
|---|---|
gitea --config /etc/gitea/app.ini actions generate-runner-token --scope myorg | Print a token on the Gitea host (no scope = instance) |
kubectl -n ci create secret generic runner-secret --from-literal=token="$TOKEN" | Store it with no trailing newline, unlike echo | base64 |
gitea-runner register --no-interactive --instance URL --token-file /run/token --labels ubuntu-latest:docker://IMAGE | Register. --token-file keeps the token out of ps |
kubectl -n ci logs runner-0 -c runner | grep -i declare | Confirm it registered and declared its labels |
runs-on label, and runs each one in a container, a Pod or on the host. Register it once and keep its .runner file.1.Three shapes
The trade : full features and privileged, or unprivileged and fewer features. Upstream recommends the dind image for container jobs : jobs never get the host's socket. Privileged dind is node root (Image Builds, Pod Security).
| DinD StatefulSet | Rootless dind | kubernetes:// job Pods | |
|---|---|---|---|
| Privileged container | yes, dockerd | yes, still privileged | no |
| Docker actions | yes | yes | no, the job fails |
| Services | containers on the job network | same | extra containers, on localhost |
| Job limits | the runner Pod's, shared | same | each job Pod's own, from pod_ |
| Extra RBAC | none | none | Role : pods, pods/, pods/, secrets |
2.The DinD StatefulSet
- A StatefulSet : each Pod keeps its own
.runnerviavolume.Claim Templates - dind is a native sidecar : up before the runner, down after it (1.29+, GA 1.33, Pods).
require_makes the runner wait.docker: true - Give dind's data root a volume, or every new Pod re-pulls all job images.
terminationaboveGrace PeriodSeconds shutdown_(upstream : 3h). Pin the tags in upstream'stimeout statefulset-.dind.yaml
3.kubernetes:// job Pods
- ShapeNew in v4.1.0. One bare Pod per job, not a Job : a
jobcontainer plus one per service. - NeedsKubernetes 1.30+, a namespaced Role (RBAC). Job Pods get no SA token by default.
- Template
pod_merges into every job Pod (template name: job,$services). Set resources there.pod_add per-label overrides, such as atemplates.<label> nodeSelector. - FailsDocker actions,
docker://steps,options,volumes,credentials. Pull secrets go in the template.
4.config.yaml keys that bite
| Key | Default | Why it matters |
|---|---|---|
runner. | 1 | jobs at once per process; each concurrent docker job takes a subnet from the daemon's pool |
runner. | 0s | running jobs are cancelled at once on SIGTERM, and Gitea does not re-queue them |
container. | false | true makes every job container privileged and turns off option stripping |
container. | "" | jobs get the daemon's socket; "-" keeps it out |
metrics. | false | /metrics, /healthz, /readyz on 127.0.0.1:9101, no auth (Prometheus) |
config init writes the file, config set edits a key. config generate is a commented reference, not a file to copy.
5.Sizing, and no autoscaler
- Concurrency = replicas x
runner.capacity. Maintainers advise more runners withcapacity: 1over one big one. - No upstream controller, CRD or scaler, and no KEDA scaler (contrast ARC). The chart
gitea/v0.1.2 defaults to runner 2.0.1 : pin your own tag.helm- actions - Job containers on Docker's 1500 MTU hang on big transfers over a smaller pod network : lower dockerd's
--mtu(job networks inherit it since 5.0.0).
volumeClaimTemplates are immutable, so the new one starts on fresh volumes, every Pod registers anew and the old entries stay offline. Do better : pick a neutral name up front. If you must rename, delete the stale runners.kubernetes:// Pods trade features for no privileges.1.Trust model
- Privileged dind is node root for every workflow author, and rootless dind is still privileged (Image Builds). Pod Security Baseline rejects it, a native sidecar included : the runner namespace needs the
privilegedlevel. - Mitigate with a tainted node pool (scheduling), trusted repositories, egress limits, or
kubernetes://Pods. - An instance runner takes every repository's jobs : with stranger PRs, use repository or organization runners.
2.What the runner strips
- Since 3.0.0 workflow
container.optionslose host-escape flags and--privilegedis ignored (more in 5.0.0).container.privileged: trueturns that off. - The runner's own options are trusted (unfiltered since 3.3.1). A workflow's
--env-fileand--label-filealways fail.
container.docker_host: "-".container.privileged: true so one repository's daemonless builds work (Buildah needs user namespaces the default seccomp profile blocks). Every job container on that runner turns privileged. Do better : run a separate privileged pool with its own label, since the runner's container.options reach every job on it too.3.Fork PRs, secrets and the job token
- Gitea decrypts owner and repo secrets and sends them to the runner inside the task. Fork pull requests, except
pull_request_target, get only the job token (GITEA_TOKEN, aliasGITHUB_TOKEN), clamped read-only. - Several approval-gate bypasses for fork PR runs were fixed in 2026, some only in 28.0 : keep Gitea current.
- The job token cannot publish packages : use a token with
write:package(Image Builds). Noid-tokenexists, so no keyless signing from jobs (Image Signing).
4.Differences from GitHub Actions
| Area | Gitea 28.1, runner 5.0 |
|---|---|
| Contexts | gitea.*, with github.* as an alias. No GraphQL. |
| Actions without a host | cloned from github.com unless DEFAULT_ |
| Token scopes | no statuses, checks, deployments, id-token; adds code, releases, wiki, projects |
runs-on | labels only, no runner groups; required since runner 5.0.0 |
5.Upgrading across majors
| To | Check |
|---|---|
| 1.0 | renamed; since 1.0.5 env overrides like GITEA_ are ignored : use config.yaml |
| 2.0 | DOCKER_ and DOCKER_ no longer give pull credentials |
| 3.0 | one process per .runner, option stripping, cache v2 on |
| 4.0 | with cache. jobs reach the cache through their runner; valid_ globs |
| 5.0 | git 2.34.1+ on the runner, ${NAME} expands in config, runs-on required |
- An upgrade is an image swap : check the env and config schema at the target tag. Registrations carry over.
- Gitea 28 deletes completed runs after 400 days (
RUN_) : setRETENTION_ DAYS 0first to keep history.
CI and builds
Building Images in the Cluster
1.A RUN step is a container
- A builder unpacks the base image. Each
RUNruns in a new container over the layer stack and is saved as a layer. Then the image is pushed. - For that it calls
unshareorclonewithCLONE_,NEWUSER NEWNSandNEWPID, mounts overlay layers and a fresh/proc, and writes files as many UIDs. - No Docker socket on containerd nodes (Images and Pulls).
- No seccomp profile means
Unconfined, unless the kubelet setsseccompDefault: true(upstream defaultfalse). A builder can break the day a node flips it. - In dind, containers get Docker's default profile, so Buildah fails with
unshare(CLONE_: not permitted, unless the job container is privileged or unconfined.NEWUSER)
2.The four fences
3.The options
| Option | Pod needs | Lowest level |
|---|---|---|
| Docker-in-Docker | privileged: true | Privileged |
| Rootless dind | still privileged: true | Privileged |
| BuildKit rootless | Unconfined profiles, UID 1000 | Privileged (explicit Unconfined) |
| BuildKit userns | hostUsers: false plus privileged: true | Privileged |
| Buildah, userns Pod | hostUsers: false, UID 1000, no seccomp field | Baseline, while seccomp is effectively Unconfined |
| kaniko forks | root, no extra capabilities | Baseline |
Nothing that runs RUN steps with real namespaces is shown to fit Restricted. Levels : Pod Security.
4.Docker-in-Docker
- The official image says
--privilegedis required, and the rootless variant needs it too : "rootless" is the daemon's UID, not the Pod's power. - A privileged Pod is node root. Any workflow author on that runner can own the node, and the namespace must allow
privileged. - ARC's
dindmode and Gitea's dind sidecar are this : ARC, Gitea runners. Best as a native sidecar. - TLS off (
DOCKER_) and arguments empty or starting with a flag : the stock entrypoint listens onTLS_ CERTDIR="" tcp://0.0.0.0:2375, no auth. Keep TLS, or start the arguments with the worddockerd.
5.kaniko, archived
- Google archived it in June 2025 (last release v1.24.0). Forks : osscontainertools publishes images, Chainguard source tags only.
- It is the exception : commands run in its own root and are snapshotted in userspace, so it needs root but no new namespaces and fits Baseline. Its README says it does not make untrusted builds safe, and it must run in its own image.
- No multi-arch index of its own : use
manifest-tool.
docker call fails and every workflow grows its own until docker info loop. Do better : run dind as a native sidecar with a docker info startup probe, as ARC's dind mode does, so the job starts only once the daemon answers.1.Three shapes, and who gives you which
| Runner | Daemon in the job ? Build with |
|---|---|
ARC dind | yes, privileged sidecar : docker build, Buildx |
ARC kubernetes | no : daemonless, remote BuildKit, or a hook template that adds dind |
| Gitea with dind | yes, job containers privileged only with container. : docker build, Buildah in a privileged job |
Gitea kubernetes:// | no : daemonless or remote BuildKit |
2.BuildKit on Kubernetes
- ExamplesUpstream
examples/: Pod, Deployment plus Service with mTLS, StatefulSet (warm cache), Job withkubernetes buildctl-. Each privileged, rootless (recommended) or userns. All usedaemonless.sh mastertags : pin one. - RootlessStill seccomp and AppArmor
Unconfined, UID 1000, and--oci-: leftover step processes are not killed and steps can signal the daemon.worker- no- process- sandbox - Buildx driver
docker buildx create --driver kubernetesmakes the Pods, privileged by default.rootless=trueswaps in Unconfined profiles. Apersistent-option gives a StatefulSet with a PVC (v0.34.0+).volume- claim - Trap
requests.sets a limit, not a request : scratch use above it evicts the builder.ephemeral- storage - Client
buildctl --addr kube-pod://<pod>needspods/exec(RBAC).
3.Buildah in a user-namespace Pod
spec: # abridged
hostUsers: false
containers:
- name: buildah
image: quay.io/buildah/stable:v1.43.4-immutable
securityContext: {runAsUser: 1000} # else root
volumes: # storage.conf points overlay here
- {name: storage, emptyDir: {sizeLimit: 10Gi}}
- No
privileged, capabilities,/dev/fuseor seccomp field : Baseline on the field rules, but it works only while the node runs PodsUnconfined. - Rootless BuildKit cannot do this (rootlesskit needs its own mappings) : BuildKit in a user namespace runs the rootful image with
privileged: true.
4.User namespaces and Pod Security
| Rule | With hostUsers: false |
|---|---|
Restricted runAsNonRoot, runAsUser; Baseline procMount | skipped; Unmasked allowed only with it |
| Policy version | only v1.35 and later : a namespace pinned to enforce- gets none |
| Still checked | privileged, capabilities, seccomp, AppArmor, host namespaces |
privileged: true | allowed, still the Privileged level |
The field, node needs and UID mapping : Pod Security.
The Buildah Pod is one published setup (CERN, 2025), not upstream advice.
5.Storage and scratch
vfsworks anywhere but copies every layer : slow and disk-hungry. Overlay on an overlay root fails (a mount_) : use an emptyDir or PVC, or fuse-overlayfs (needsprogram is required /dev/fuse).- Image-declared
VOLUMEs (dind's/var/lib/docker, Buildah's/var/lib/containers) with no mount become a fresh directory in containerd's state on the node for every container start : the data is lost on each restart. - Mount an emptyDir with a
sizeLimitand setlimits., or a big build fills the node or is evicted (Requests and Limits).ephemeral- storage
RUN steps in real namespaces without privileged, and only while seccomp stays unconfined.1.Multi-arch
- dockerd's own builder (
docker:dindhas no emulators) and Buildah need binfmt handlers : a registration from a privileged container changes every Pod on that node until reboot. Pin the binfmt image : Buildxqemu.imageandsetup-qemu-actionboth default to a floatinglatest. - Emulated steps can crash (
qemu: uncaught target signal 11) : keepRUNout of the final stage. - Native builders : pick nodes by
kubernetes.io/arch(Scheduling).
| Strategy | Needs | Catch |
|---|---|---|
| QEMU | bundled in BuildKit images, else binfmt handlers | much slower, node-wide |
| Native nodes | one builder per arch (nodeselector=) | needs those nodes |
| Cross-compile | $BUILDPLATFORM stage, ARG TARGETOS TARGETARCH | toolchain must cross-compile |
| Per-arch, then merge | buildx imagetools create, buildah manifest, manifest-tool (kaniko) | extra merge step |
2.Reproducible output
SOURCE_. BuildKit stamps metadata from it; layer file times needDATE_ EPOCH=$(git log -1 --pretty=%ct) rewrite-, and a failed rewrite only warns : compare two digests.timestamp=true - Buildah 1.41+ :
--source-,date- epoch --rewrite-. kaniko :timestamp --reproducible. - Pin bases by digest and
compatibility-(30). Provenance records the build.version
3.Caching
- A per-job builder starts cold. A registry cache survives :
--cache-to type=registry,(ref=..., mode=max min, the default, skips earlier stages). RUN --mount=type=cacheis not exported : a warm builder needs a PVC-backed StatefulSet. Buildah needs--layersfor--cache-to, kaniko--cache-repo.- Never share a BuildKit cache between trusted and untrusted builds (a poisoning bug, fixed in v0.33.1).
4.Pushing
- A
dockerconfigjsonSecret asconfig.json(BuildKit$DOCKER_, kanikoCONFIG /kaniko/.docker/, Buildah--authfile). Record the digest. - Pull request builds never push. Only protected branches do, with a narrow write-only token (Gitea's job token cannot publish packages).
- Anonymous Docker Hub pulls : 100 per 6 hours per IPv4 address or IPv6 /64, shared by one NAT. Log in or mirror.
- Then sign and gate it, and let GitOps pick up the tag (Flux, Image Updater).
$BUILDPLATFORM stage where the toolchain allows it, move the heavy compile into a rarely rebuilt base image, and keep a mode=max registry cache.5.Commands
| Command | Does |
|---|---|
docker buildx create --bootstrap --name kube --driver kubernetes --driver-opt namespace=buildkit,rootless=true | BuildKit Pods as a Buildx builder |
docker buildx build --platform linux/amd64,linux/arm64 --push --cache-to type=registry,ref=<repo>:buildcache,mode=max --cache-from type=registry,ref=<repo>:buildcache -t <repo>:<tag> . | Multi-arch with a registry cache |
buildah build --platform linux/amd64,linux/arm64 --manifest <list> --source-date-epoch "$(git log -1 --pretty=%ct)" --rewrite-timestamp . | Multi-arch, reproducible |
buildah manifest push --all --digestfile digest.txt <list> docker://<repo>:<tag> | Push all, keep the digest |
kubectl label ns <builds> pod-security.kubernetes.io/enforce=privileged | Allow privileged here |
SOURCE_DATE_EPOCH and the timestamp rewrite, push a mode=max registry cache, and give push credentials only to protected-branch builds.Certificates and secrets
cert-manager
1.Why cert-manager ?
A Kubernetes add-on that automatically requests, stores and renews TLS certificates. Latest release : v1.21.3.
- Certs expire (Let's Encrypt certs last 90 days or less)
- Manual way : make a CSR, get it signed, paste into a Secret, remember to repeat. Nobody remembers.
- Kubernetes has no "issue and renew" object. Secrets just hold bytes.
- Adds
CertificateandIssueras CRDs, so a cert is declarative like a Deployment
2.Components
Runs as Deployments in the cert-manager Namespace.
- You declare WHAT (Certificate) and WHO signs (Issuer)
- Output : Secret of type
kubernetes.io/tlswithtls.crt,tls.key(andca.crtif the issuer hands over its CA. ACME does not.) - Pods mount it, Ingress and Gateway point at it
-> Wait, or run
cmctl check apicmctl check api- controllerThe brain. Reconcile loops for Certificate, CertificateRequest, Order, Challenge and Issuers.
- webhookValidates cert-manager resources. Stamps who asked on each CertificateRequest.
- cainjectorInjects CA bundles into webhook configs, CRDs and APIServices.
- acmesolverSmall pod that answers HTTP-01 challenges.
- startupapicheckInstall Job that checks webhook and API are reachable.
- cmctlCLI : status, renew, approve, check api.
3.Issuers and CRDs
Issuer is namespaced, ClusterIssuer is cluster-wide. Same spec, different scope.
- Built in : SelfSigned, CA, Vault, ACME, Venafi (now CyberArk's)
- External issuers plug in via
issuerRefgroup and kind - ACME issuer needs an account key Secret,
privateKeySecretRef
| Kind | apiVersion | Scope | What it is |
|---|---|---|---|
| Issuer | cert-manager.io/v1 | Namespaced | Signer, one namespace |
| ClusterIssuer | cert-manager.io/v1 | Cluster | Signer, any namespace |
| Certificate | cert-manager.io/v1 | Namespaced | Desired cert, goes to spec.secretName |
| CertificateRequest | cert-manager.io/v1 | Namespaced | One CSR + result, auto-created |
| Order | acme.cert-manager.io/v1 | Namespaced | ACME order (ACME only) |
| Challenge | acme.cert-manager.io/v1 | Namespaced | One domain proof (ACME only) |
1.ACME Issuance Flow
Certificate to Secret, step by step. The error always lives at the bottom of the chain.
CA, Vault and SelfSigned skip Order and Challenge. The Request goes straight to signing.
2.Solvers, Renewal, Ingress
- Renews at about 2/3 of lifetime. Default duration is 90 days.
- Controller watches Certificate + Secret. Expired, missing or mismatched triggers a new Request.
- Annotate an Ingress with
cert-manager.and it makes the Certificate for you (ingress-shim)io/ cluster-issuer - Gateway API : install the Gateway CRDs first (or restart cert-manager after), then set
config.gatewayAPI. enabled=true
| HTTP-01 | DNS-01 | |
|---|---|---|
| Proof | Token at /.well-known/acme-challenge | TXT record in DNS |
| Wildcards | No | Yes |
| Public HTTP route | Needed | Not needed |
| DNS API creds | No | Yes (Route53, Cloudflare) |
| Good for | Simple public sites | Wildcards, internal clusters |
3.Example, Commands and Gotchas
kind: ClusterIssuer
metadata: {name: letsencrypt-staging}
spec:
acme:
email: user@example.com
server: <staging URL>
privateKeySecretRef: {name: le-staging-key}
solvers:
- http01: {ingress: {ingressClassName: nginx}}
kind: Certificate
metadata: {name: example-com, namespace: sandbox}
spec:
secretName: example-com-tls
dnsNames: [example.com, www.example.com]
issuerRef:
{name: letsencrypt-staging, kind: ClusterIssuer}
- Test on the STAGING URL (
acme-staging-v02.). Production has rate limits.api. letsencrypt. org - Stuck cert ? Walk down : Certificate, Request, Order, Challenge
- "failed calling webhook" right after install ? Wait for the webhook
- Renewed Secret does not restart Pods. Reload in the app or use a reloader.
- Gateway listeners need protocol HTTPS or TLS, a hostname,
tls.mode Terminate, certificateRefs in the Gateway's namespace
| Command | Description |
|---|---|
helm install cert-manager oci:// | Install with Helm. crds.enabled=true installs the CRDs. |
kubectl get clusterissuer,issuer,certificate -A | List issuers and certs. READY=True is the goal. |
kubectl describe certificate <name> -n <ns> | Conditions and events of a stuck cert |
cmctl status certificate <name> -n <ns> | Detailed status with related resources |
cmctl renew <name> -n <ns> | Force a renewal now (also --all) |
Certificates and secrets
External Secrets Operator
1.What is ESO ?
ESO reads secrets from a vault or cloud secret manager (AWS, GCP, Azure, Vault...) and keeps a normal Kubernetes Secret in sync with them.
Git holds only pointers to secrets, never the values. Pods stay unchanged, they still mount a normal Secret.
Latest : v2.12.0 (CRDs use external-secrets.). CNCF Sandbox project.
Vault -> ESO -> Secret -> Pod
2.Why it exists ?
- Secrets in K8s are only base64. Committing them to Git leaks credentials.
- The real source of truth lives in AWS, GCP, Azure or Vault. Copying values by hand goes stale.
- Rotating a secret means someone must update every cluster.
Fix : declare WHERE the secret lives in Git, and a controller fetches and refreshes it.
- Declarative, no values in Git
- Re-fetches every refreshInterval
- Many backends, one API
- Output is a plain Secret
- Templating reshapes values
- PushSecret goes the other way
- Generators create values
- Works with Flux/Argo (just YAML)
3.Parts of ESO
- ControllerMain operator. Reconciles ExternalSecret and PushSecret, calls providers, writes Secrets.
- WebhookValidates the ESO CRDs (own deployment).
- Cert-controllerManages TLS certs for the webhook (own deployment).
- GeneratorsMake values: Password, UUID, SSHKey, ECR/GCR/ACR tokens and more.
- Install ESO with Helm
- Create a store with auth
- Create an ExternalSecret
- Controller writes the Secret
- Pod mounts the Secret
- Refresh loop keeps it current
4.How the Sync Works
- Watch ExternalSecret
- Resolve store
- Fetch remote data
- Apply template
- Write Secret
- Wait refreshInterval, repeat from 3
Ready=True on success, SecretSyncedError on failure.
1.ESO Kinds + Store Scoping
A SecretStore only works in its own namespace. A ClusterSecretStore is shared, so restrict it with conditions. Tip : one SecretStore per namespace, or one shared ClusterSecretStore.
| Kind | Scope | Purpose |
|---|---|---|
| SecretStore | Namespaced | Provider config and auth |
| Cluster | Cluster | Shared provider config |
| ExternalSecret | Namespaced | Fetch remote data into a Secret |
| Cluster | Cluster | ExternalSecrets in many namespaces |
| PushSecret | Namespaced | Push Secret to provider |
| Cluster | Cluster | PushSecret across namespaces |
2.Example ExternalSecret + Gotchas
kind: ExternalSecret
metadata: {name: db-creds}
spec:
refreshInterval: 1h
secretStoreRef: {name: vault, kind: SecretStore}
target: {name: db-creds, creationPolicy: Owner}
data:
- secretKey: password
remoteRef: {key: prod/db, property: password}
- Store must be Ready first
- Env vars need a reloader, files refresh
- Prefer workload identity (IRSA)
- Low refreshInterval hits rate limits
- Owner policy : delete ES, Secret goes too
- Migrate v1beta1 / v1alpha1 to v1
3.ESO vs Plain Secret vs CSI Driver
| Source of truth | Creates K8s Secret | Rotation | Pod changes | How it works | |
|---|---|---|---|---|---|
| Plain Secret | In cluster | Manual | Manual | None | Static |
| ESO | External provider | Yes, automatic | Every refresh | None | Controller + CRDs |
| Secrets Store CSI | External provider | Optional (sync) | Mount rotation | CSI volume in Pod | Node driver |
4.ESO Commands
| Command | Description | Example |
|---|---|---|
helm install external-secrets external-secrets/ | Install operator + CRDs--set installCRDs=false | --set installCRDs=false |
kubectl get externalsecret -A | List, with sync statuskubectl get es -n app | kubectl get es -n app |
kubectl describe externalsecret <name> | Conditions + eventskubectl describe es db-creds | kubectl describe es db-creds |
kubectl get secretstore, | Check stores are Readykubectl get css | kubectl get css |
kubectl -n external-secrets logs deploy/ | Controller logsadd -f to follow | add -f to follow |
kubectl annotate es <name> force-sync=$(date +%s) --overwrite | Force immediate refreshes = externalsecret | es = externalsecret |
Data
CloudNativePG Operator
1.What is CNPG ?
An operator that runs highly available PostgreSQL on Kubernetes. You write one Cluster YAML and it builds the Pods, volumes, services, failover and backups.
- A plain Postgres Deployment has no replication, no failover, no backups.
- StatefulSets give stable names and disks, but no Postgres knowledge.
- Operator = the DBA runbook as a reconcile loop.
- Talks to the Kubernetes API directly. No Patroni, no StatefulSet.
postgresql.cnpg.io/v1 kind Cluster.2.Cluster Layout
One primary plus N hot standbys. Each Pod has its own PVC, no shared disk. Replication is native Postgres WAL streaming.
| Service | Points to |
|---|---|
| app-db-rw | Primary only |
| app-db-ro | Replicas only |
| app-db-r | Any instance |
3.Parts and CRDs
- OperatorDeployment in cnpg-system. Helm names it cnpg-cloudnative-pg, the plain manifest cnpg-controller-manager. Watches the CRDs.
- Instance managerPID 1 in every Postgres Pod. Probes, WAL archiving, promotion.
- PoolerManaged PgBouncer (rw or ro).
- Barman Cloud pluginShips WAL and backups to object storage. Separate install : cert-manager first, then into the operator's namespace.
- kubectl cnpgCLI plugin for status, promote, psql.
| Kind | What it does |
|---|---|
| Cluster | The Postgres cluster |
| Backup | One backup request |
| ScheduledBackup | Cron style backups |
| Pooler | PgBouncer pooling |
| Database | Declarative DB |
| DatabaseRole | Declarative role |
| Publication / Subscription | Logical replication |
| ImageCatalog / ClusterImageCatalog | Image per Postgres major (namespace, cluster-wide) |
| FailoverQuorum | Sync failover quorum (with failoverQuorum) |
| ObjectStore | The plugin's bucket config (barmancloud.cnpg.io) |
Cluster YAML into a primary plus streaming replicas, each with its own PVC, behind -rw / -ro / -r Services.1.Failover
Automatic inside one Kubernetes cluster. Across clusters it is manual : promote the replica cluster yourself.
failoverDelay/switchoverDelaytune the timing.
spec.primaryLease (15/10/2/1 s) : a Lease named after the Cluster, held by the primary's Pod. A promotion gate, not a fence.2.CNPG vs DIY StatefulSet
| Topic | DIY STS | CNPG |
|---|---|---|
| Failover | Manual or Patroni | Built in |
| Replicas | You script them | instances: N |
| Backups / PITR | Your cron jobs | Backup CRDs |
| Pods | Ordinal order | Lowest free serial (1.30) : a recreated instance keeps its name |
| Config | Edit, restart | Rolling update |
| Passwords / TLS | Hand made | Generated |
- A graceful delete of the primary waits up to 180 s for open sessions (the smart shutdown) before the failover starts.
- After a node loss the phase stays
Failing overuntil the old Pod is evicted, even though a replica is already promoted.
3.Try It
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: app-db
namespace: apps
spec:
instances: 3
storage:
size: 10Gi
postgresql:
parameters:
max_connections: "200"
- Install operator into cnpg-system.
- Apply a Cluster with instances: 3.
- Operator makes primary Pod and PVC, then clones replicas.
- It adds -rw, -ro, -r Services and the app-db-app Secret.
- App connects to app-db-rw.apps.svc:5432.
- Failover is automatic. WAL ships only once a WAL archiver plugin is set up (next sheet).
1.Backups Need a Plugin
The in-tree barmanObjectStore is deprecated since v1.26 and goes in v1.32.0 (#11671). Backups and WAL archiving now come from a plugin, most often the Barman Cloud plugin.
- Install cert-manager first. The plugin runs in the operator's namespace.
- An
ObjectStore(barmancloud.cnpg.io) names the bucket and its keys. - The Cluster lists it under
plugins:. Nothing is archived until then. - Recovery always makes a new Cluster. If it archives too, give it its own
serverName: the source's fails withExpected empty archive. - Or back up with volume snapshots.
# in the Cluster's spec
plugins:
- name: barman-cloud.cloudnative-pg.io
isWALArchiver: true
parameters:
barmanObjectName: my-store
Without this entry, applying a Cluster archives nothing, and the Cluster still reads healthy.
- Delete a Cluster and its PVCs go, data included. The safety net is the bucket and the snapshots : Backup objects hold no data, and Poolers keep running, inactive.
- instances: 1 means no HA, and its primary PDB makes its node undrainable. Use 3 across nodes : the default anti-affinity is only preferred.
- The default image is the deprecated system flavour. Pin a dated minimal tag plus its digest : a bare digest is refused.
- Never write via a Pod IP or -r.
kubectl get stsshows nothing : look at pods, pvc, clusters.
2.Useful Commands
| Command | Description |
|---|---|
| helm repo add cnpg https://cloudnative-pg.io/charts | Add the chart repo (the old github.io address redirects here) |
| helm upgrade --install cnpg -n cnpg-system --create-namespace cnpg/cloudnative-pg | Install the operator |
| kubectl get cluster -n apps | List CNPG clusters |
| kubectl cnpg status app-db | Health, primary, lag |
| kubectl cnpg promote app-db app-db-2 | Manual switchover |
| kubectl cnpg psql app-db | psql on the primary |
| kubectl cnpg backup app-db -m plugin --plugin-name barman-cloud.cloudnative-pg.io | On-demand backup through the plugin. Without -m plugin it asks for barmanObjectStore and fails : the cluster has no backup section |
Cloud infrastructure
Crossplane
1.What is Crossplane ?
A control plane framework for platform engineering. You define your own Kubernetes APIs and Crossplane runs the controllers behind them : custom resources without writing an operator.
| Part | What it gives you |
|---|---|
| Managed resources | One object per cloud thing |
| Composition | Your own API from many parts |
| Operations | One-off, cron or watch runs |
| Package manager | Providers, functions, configs |
Use all four or only some. CNCF graduated 2025-10-28. Latest core v2.4.3 (2026-10-22), a new minor every quarter, the three newest supported.
2.How the parts fit
Helm installs two Deployments in crossplane-system. Packages add more pods.
- crossplane : core. Installs core CRDs, starts an XR controller per XRD, calls functions.
- rbac-manager : grants core access to XRs and MRs. Others may need a ClusterRole.
- Provider pod : one per provider, syncs MRs with a cloud API.
- Function pod : Deployment + Service, gRPC on 9443.
3.Providers and managed resources
A provider package adds MR CRDs and runs a controller. One managed resource (MR) = one external thing, like one S3 bucket.
- forProvider : desired settings, enforced. initProvider : used only at create. atProvider : what the cloud reports.
- managementPolicies : Observe, Create, Update, Delete, LateInitialize. Default
["*"]. - Keep the cloud resource on delete : leave
Deleteout. Namespaced MRs have no deletionPolicy. - Import : set
crossplane.and start withio/ external-name ["Observe"].
apiVersion: s3.aws.m.upbound.io/v1beta1 kind: Bucket # namespaced MR metadata: namespace: default generateName: crossplane-bucket- spec: forProvider: region: us-east-2 # providerConfigRef defaults to # {kind: ClusterProviderConfig, name: default}
4.What v2 changed
| Area | In v2 (since Aug 2025) |
|---|---|
| XRs | Namespaced by default. Cluster scope still allowed |
| Claims | Legacy. Only with a v1 XRD, scope LegacyCluster |
| MRs | Namespaced, in .m. groups. Cluster MRs are legacy |
| Compositions | Pipeline mode only. May include any Kubernetes kind |
| Operations | New, alpha : run a pipeline once, on cron or on watch |
| Provider CRDs | Become MRDs. With safe-start, an MRAP picks the CRDs |
| Removed | Native P&T, ControllerConfig, secret stores (use ESO) |
- One minor at a time, latest patch, up to v1.20.
- Run
crossplane beta upgrade check(v1.20.9+ CLI only). - Convert
mode: Resources, drop--registry, fully qualify packages. - Old claims and legacy MRs keep working.
1.XRD and Composition
XRD = schema of a new API, turned into a CRD plus an XR controller. Composition = how to build each XR.
apiVersion: apiextensions.crossplane.io/v2
kind: CompositeResourceDefinition
metadata: { name: apps.example.crossplane.io }
spec:
scope: Namespaced # default; or Cluster. Immutable
group: example.crossplane.io
names: { kind: App, plural: apps }
versions: # openAPIV3Schema trimmed
- { name: v1, served: true, referenceable: true }
---
apiVersion: example.crossplane.io/v1
kind: App # the XR
metadata: { namespace: default, name: my-app }
spec: { image: nginx } # + spec.crossplane fields
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata: { name: app-yaml }
spec:
compositeTypeRef: # immutable
apiVersion: example.crossplane.io/v1
kind: App
mode: Pipeline # the only mode in v2
pipeline: # 1 to 99 steps
- step: deployment
functionRef: { name: function-patch-and-transform }
input:
apiVersion: pt.fn.crossplane.io/v1beta1
kind: Resources # base Deployment + patches
- step: mark-ready # last, bar a sequencer
functionRef: { name: function-auto-ready }
2.How the pipeline runs
- Each step gets the snapshot, the desired state so far and a context.
- Pass on all desired state. Anything dropped is deleted.
- A step may set XR status, never the XR's own spec.
3.Composition functions
| function- | Latest | Input kind | Good for |
|---|---|---|---|
| patch-and-transform | v0.11.2 | Resources | v1-style patches, transforms, readiness checks. No loops or ifs |
| go-templating | v0.13.2 | GoTemplate | Go templates with Sprig, inline or from files |
| kcl | v0.12.3 | KCLInput | KCL programs, inline or oci:// |
| python | v0.6.0 | Script | compose() or operate(). Also runs in Operations |
| environment-configs | v0.9.2 | Input | Loads EnvironmentConfigs into the pipeline context |
| auto-ready | v0.7.3 | Input | Marks healthy composed resources ready. Put it last |
Install as a pkg. Function. SDKs : Go, Python. function-sequencer (v0.6.1) orders creation, after auto-ready.
4.Readiness and status
- A composed resource is ready only when a function returns
READY_TRUE. Unset = not ready. Crossplane never guesses. - auto-ready has checks for core kinds (Deployment, Service, Job ...). Others, MRs too, need
Ready=True. - No Ready condition at all (a composed Provider, a Cluster API v1beta2 Cluster) = XR stuck. Add an alpha CEL check.
- XR status : a function returns a partial XR with only
status.
1.CRD count and MRD activation
About 3 MiB of API server memory per CRD (docs estimate). provider-upjet-aws v2.9.0 ships 1063 namespaced + 1063 legacy service CRDs.
- Families : one provider per cloud service.
- MRDs (alpha API) : each MR CRD becomes an MRD. Only an active one makes a real CRD. One-way.
- Helm activates all by default. Install with
--set provider., then write MRAPs.defaultActivations={} - Copy names from
kubectl get mrds: the docs'*.examples match nothing.aws. m. crossplane. io
2.Operations (alpha)
A pipeline that runs to the end, like a Job for API objects. Off until --enable-operations.
- Operation : once. CronOperation : schedule. WatchOperation : on change.
- Functions need the
operationcapability (function-python). - Force-applies, no owner refs, can't delete. retryLimit 5 if unset.
3.Packages
Provider, Function, Configuration : pkg., fully qualified OCI images.
- Configuration : XRDs, Compositions,
dependsOn. - Each change = a new revision, one active.
- DeploymentRuntimeConfig shapes the pod, ImageConfig rewrites registries.
- Pin by
@sha256:. With signature checks on, tags could be swapped before v2.2.3 / v2.3.3.
4.Commands
| Command | Does |
|---|---|
helm install crossplane --namespace crossplane-system --create-namespace crossplane-stable/crossplane | Install core. Repo charts.crossplane.io/stable |
curl -sfL "https:// | CLI crossplane (was crank), pinned. Own release train |
kubectl get pkg ; kubectl get xrd | Packages INSTALLED / HEALTHY, XRDs ESTABLISHED |
kubectl get composite -A ; kubectl get managed -A | Every XR, every MR (slow with many CRDs) |
kubectl get mrds ; kubectl get mrap -o yaml | MRD state, and what each MRAP activated |
crossplane resource trace app my-app -n default -o wide | XR, children and MRs as a tree with status |
crossplane composition render xr.yaml composition.yaml functions.yaml | Run the pipeline locally (Docker) |
crossplane cluster top | CPU and memory of Crossplane pods. Needs metrics-server |
crossplane xpkg build --package-root=package/ then xpkg push | Build and push a package |
5.Crossplane vs Terraform vs Cluster API
| Aspect | Crossplane | Terraform / OpenTofu |
|---|---|---|
| Shape | Always-on control plane | A CLI you run |
| State | An object per resource | One locked state file |
| Drift | Reverted by the loop | Only when someone runs it |
| Abstraction | XRD = a real API + RBAC | Modules |
| Immutable field | Never applied | Plans a replacement |
- No
READY_TRUE, no Ready XR. Read events too. - P&T paths are Optional : typos are silent.
- Modern XRs publish no connection Secret.
- A Provider delete is never refused. MRs first.
- Uninstall : XRDs, MRs, Providers, core, CRDs.
paused: "true"also blocks deletes.- Webhooks off = Usages silently off.
- Many CRDs slow Argo CD : raise QPS.
Scaling and metrics
KEDA
1.What is KEDA ?
KEDA scales your workloads based on external events (queue length, Kafka lag, cron, Prometheus query) and can shrink them to zero pods when idle.
- Built-in HPA only scales on CPU / memory.
- A backlog of 10k messages can sit at low CPU.
- An HPA reaches zero only with an Object or External metric (beta in v1.37). Idle workers burn money.
- KEDA ships many ready-made scalers.
- Batch work wants one Job per event.
- Pay for workers only while there is work to do.
| Feature | HPA alone | KEDA |
|---|---|---|
| Signals | CPU, memory, custom metrics you provide | Dozens of event sources |
| To zero | Beta in v1.37, Object or External metric only | Yes (min 0) |
| Jobs | No | ScaledJob |
| Type | Built in | Add-on that feeds an HPA |
2.How KEDA scales
KEDA does not replace HPA, it drives it. Two tracks :
- 0 1 is done by keda-operator.
- 1 N is done by a normal HPA fed by KEDA metrics.
3.Components
- keda-operatorWatches ScaledObject / ScaledJob, creates the HPA, does 0 1.
- metrics-apiserverServes external.metrics.k8s.io so the HPA can read event metrics.
- admission-webhooksValidate KEDA resources on apply (not in the -core manifest).
- ScalersBuilt-in plugins that query an event source.
- HPA (generated)Created per ScaledObject, does 1 N.
- HTTP Add-onOptional, separate, beta. Scales on HTTP traffic.
1.ScaledObject
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: worker-scaler
spec:
scaleTargetRef:
name: worker
minReplicaCount: 0
maxReplicaCount: 20
pollingInterval: 30
cooldownPeriod: 300
triggers:
- type: rabbitmq
metadata:
queueName: tasks
mode: QueueLength
value: "10" # ~10 msgs per pod
authenticationRef:
name: rabbit-auth
2.Authentication
Credentials live in TriggerAuthentication (namespaced) or ClusterTriggerAuthentication (shared), referencing Secrets or pod identity.
3.ScaledObject vs ScaledJob
Use ScaledJob for long-running or one-shot tasks.
| Kind | Scope | Purpose |
|---|---|---|
| ScaledObject | Namespaced | Scale a workload on triggers |
| ScaledJob | Namespaced | Spawn Jobs per event |
| TriggerAuthentication | Namespaced | Credentials for a scaler |
| ClusterTriggerAuthentication | Cluster | Shared credentials |
1.Commands
| Command | Description | Example |
|---|---|---|
| helm repo add kedacore https://kedacore.github.io/charts | Add the official chart repo | helm repo update |
| helm install keda kedacore/keda -n keda --create-namespace | Install KEDA via Helm | kubectl apply --server-side -f keda-2.21.0.yaml |
| kubectl get pods -n keda | Verify operator, metrics server, webhooks | kubectl get pods -n keda -w |
| kubectl get scaledobject -A | List ScaledObjects, READY / ACTIVE | kubectl get so -n prod |
| kubectl describe scaledobject NAME | Conditions and events when nothing scales | kubectl describe scaledobject worker-scaler |
| kubectl get hpa | KEDA-made HPA, named keda-hpa-NAME | kubectl get hpa keda-hpa-worker-scaler |
| kubectl annotate scaledobject NAME autoscaling.keda.sh/paused=true | Pause at current replicas. KEDA deletes its HPA until the annotation goes | kubectl annotate scaledobject worker-scaler autoscaling.keda.sh/paused-replicas=0 |
| kubectl logs -n keda deploy/keda-operator | Scaler errors (auth, connection) | kubectl logs -n keda deploy/keda-operator --tail=50 |
| kubectl get apiservice v1beta1.external.metrics.k8s.io | Is the external metrics API Available ? | kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 |
| helm uninstall keda -n keda | Remove KEDA. Delete its objects first, while it runs, or they hang on finalizer.keda.sh | kubectl delete scaledobjects,scaledjobs --all -A |
2.When nothing scales
3.Gotchas & Good to know
- No second HPA on a scaled Deployment. The webhook refuses a ScaledObject over an existing HPA; a plain HPA made after it is admitted, then both stall with AmbiguousSelector.
- Activation is strictly greater : threshold 10, activation 50, metric 40 stays at zero.
- Scale to zero waits cooldownPeriod (300s) after the last active poll, then goes straight from any count to 0, past the HPA's scale-down window.
- Scale-down can kill pods mid-task. Handle SIGTERM or use ScaledJob.
- Several triggers can be combined with scaling modifiers.
- HTTP scaling is the separate HTTP Add-on (beta), or use Prometheus request rate.
- v2.21.0 breaks three things : a boundServiceAccountToken needs its audience mapped in the operator's settings, not the TriggerAuthentication (CVE-2026-77524); azure-pipelines scaleOnInFlight is now true; Temporal loses buildId, selectAllActive and selectUnversioned.
- A bad TriggerAuthentication shows only as not READY with the source's own error (Redis : NOAUTH). A missing one is only logged; a missing Secret key is not logged at all.
Scaling and metrics
Karpenter
1.What is Karpenter ?
A node autoscaler. It launches nodes for Pods that cannot be scheduled, then removes or replaces nodes that are empty, underused, outdated or too old.
- v1.14.1 core and AWS provider (2026-08-21), LTS on AWS until Jul 2027. Azure provider v1.14.3.
karpenter.sh/v1since v1.0. Tutorials with Provisioner, AWSNodeTemplate or v1beta1 are out of date.- 1.37 is not declared yet : v1.14 covers 1.26 to 1.36, these notes target 1.37. On 1.37 the AWS provider logs "not compatible" once and runs on, unsupported. Expect support in v1.15.
2.From Pending Pod to node
- kube-scheduler gives up :
PodScheduled=False, Unschedulable. - Karpenter skips Pods with a nodeName, a nomination, or a DaemonSet owner.
- Batch : 1s with no new Pod, 10s max.
- Biggest Pods first : existing nodes, planned nodes, then NodePools by weight.
- Size = requests + DaemonSet requests. Preferences relax one by one.
- Node Ready, kube-scheduler binds.
3.NodePool, NodeClass, NodeClaim
NodePool = what nodes may look like. NodeClass = how to build them on this cloud. NodeClaim = one machine.
| Kind | Holds |
|---|---|
| NodePool karpenter.sh/v1 | Requirements, taints, expiry, disruption, limits, weight |
| EC2NodeClass karpenter.k8s.aws/v1 | AMI, subnets, security groups, IAM role, kubelet |
| AKSNodeClass karpenter.azure.com | v1beta1. Image family, OS disk, maxPods |
| NodeClaim karpenter.sh/v1 | One machine. Spec is immutable |
Miss a timeout and the claim and instance are deleted. Consolidation waits for Initialized.
4.How Karpenter reads a Pod
| Pod field | What Karpenter does |
|---|---|
| requests | Sizes the node (containers, init, sidecars, overhead). Limits are ignored |
| nodeSelector, required affinity | A NodePool must allow it, custom labels included, or the Pod stays Pending |
| preferred affinity, ScheduleAnyway | Treated as required, then relaxed one at a time |
| tolerations | Must tolerate the NodePool taints. startupTaints need no toleration |
| PVC, WaitForFirstConsumer | Launches in the zone of the StorageClass or bound PV |
| DaemonSet Pods | Never trigger a node on their own. Counted as overhead |
Same rules as in Advanced Scheduling : Karpenter simulates them before it buys a node.
1.Six ways a node goes away
Replacing nodes is the job. Graceful methods can be paused or slowed, forceful ones only get a drain deadline.
| Method | Fires when | Budget reason | do-not-disrupt |
|---|---|---|---|
| Consolidation graceful | Pods fit elsewhere (delete), or elsewhere plus one cheaper node (replace) | Empty, Underutilized | Blocks it |
| Drift graceful | NodePool template or NodeClass changed, or a new AMI resolved | Drifted | Blocks it, if no TGP |
| Expiration forceful | Older than expireAfter (default 720h). No wait for a replacement | None, ignored | Does not stop it |
| Interruption forceful | AWS Spot warning, health event, instance stopped or terminated | None | Does not stop it |
| Node repair forceful | Alpha. Ready False or Unknown for 30 min. Skipped if over 20% unhealthy | None | Does not stop it |
| Manual delete | kubectl delete nodeclaim or node, drained through the finalizer | None | Does not stop it |
2.The graceful loop
- Order : Emptiness, static drift, Drift, multi-node, single-node consolidation. One method per pass.
- Draining uses the Eviction API, so PDBs hold it back.
- WhenEmpty : only nodes whose Pods cost nothing to disrupt (DaemonSet Pods).
- WhenEmptyOrUnderutilized (default) : any move that lowers cost.
- Balanced (v1.14+) : scores savings against disruption cost, moves only at score >= 0.5. Stops near-zero-savings churn.
- consolidateAfter : quiet time after the last Pod change.
Never= off.
3.What keeps it safe
apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: default }
spec:
template:
spec:
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: default
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand"]
expireAfter: 720h # forceful
terminationGracePeriod: 48h # deadline
disruption:
consolidateAfter: 1m # required in here
budgets:
- nodes: "20%" # any reason
- nodes: "0" # pause
schedule: "@daily" # UTC
duration: 10m
reasons: ["Underutilized"]
- Budgets : max nodes disrupted at once per pool (default 10%). By reason, with a UTC cron window.
- PDBs : every drain evicts, so a PDB can hold it.
- do-not-disrupt : on a Pod,
"true"or a duration like"30m". On a Node, only"true". Blocks graceful methods. Forceful drains skip those Pods until the TGP deadline. - terminationGracePeriod : near its end, leftover Pods are deleted, past PDBs and do-not-disrupt. Each still gets its own grace seconds.
1.Spot and interruptions
- Several capacity types allowed ? It prefers reserved, then spot, then on-demand.
- AWS Spot : price-capacity-optimized, up to 60 instance types per CreateFleet call.
- No capacity ? That type and zone is skipped for 3 min.
- 2-minute notice : the replacement starts and the drain runs in parallel.
- Spot nodes are consolidated by deletion only, unless the alpha SpotToSpotConsolidation gate is on.
- Do not also run AWS Node Termination Handler.
2.Karpenter vs Cluster Autoscaler
| Aspect | Cluster Autoscaler | Karpenter |
|---|---|---|
| Scales | Pre-built node groups, an expander picks one | Single instances, types picked per launch |
| Scale up | Unschedulable Pods, scan every 10s | Unschedulable Pods, batch 1s idle, 10s max |
| Scale down | Unneeded for 10 min, under 0.5 utilization | Consolidation policy and consolidateAfter (default 0s) |
| Lifecycle | Autoscaling only | Also expiry, drift (image upgrades), interruption, repair |
| Timeouts | Provision 15 min | Launch 5 min, registration 15 min |
| Pod opt-out | safe-to-evict: "false" | karpenter. |
| Versions | Use the release that matches the control-plane minor | One release spans a range of minors |
Migrating ? The guide scales Cluster Autoscaler to 0 replicas. The FAQ says the two can run side by side.
3.Providers
| Provider | Notes |
|---|---|
| AWS v1.14.1 | Reference, LTS. Helm, or managed as EKS Auto Mode (336h expiry, 21-day cap) |
| Azure v1.14.3 | Helm, or managed as AKS NAP (GA, last announced on v1.14.0). No Windows |
| Cluster API v0.2.0 | Experimental proof of concept, create and delete only. See Cluster API |
| IBM Cloud v1.0.5 | SIG owned |
| GCP v0.7.0 | CloudPilot AI, pre-1.0 |
Oracle ships an official OCI provider. The core README lists about a dozen more.
4.Gotchas
- A
disruption:block without consolidateAfter is rejected in v1.14. Many doc snippets fail on it. - A new DaemonSet never triggers a node. Give it a high, preempting PriorityClass.
- Agent taints (Cilium's) belong in startupTaints, or Karpenter keeps launching nodes.
- Editing expireAfter or TGP drifts the whole pool. So does an
@latestAMI alias, on every release. - Limits count node capacity, not requests, and can overrun in fast scale-outs.
5.Commands
| Command | Does |
|---|---|
helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version 1.14.1 -n kube-system | CRDs as their own release. Helm never upgrades a chart's crds/ folder |
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version 1.14.1 -n kube-system --set settings.clusterName=$CLUSTER_NAME --set settings.interruptionQueue=$CLUSTER_NAME | Controller : 2 replicas in kube-system, a PDB, metrics on :8080. Needs two non-Karpenter nodes in different zones |
kubectl get nodepools -o wide | NODECLASS, NODES, READY. wide adds WEIGHT, CPU, MEMORY |
kubectl get nodeclaims -o wide | TYPE, CAPACITY, ZONE, NODE, READY. wide adds IMAGEID, ID, NODEPOOL, NODECLASS, DRIFTED |
kubectl describe nodeclaim NAME | Which condition is False, DisruptionBlocked events |
kubectl logs -f -n kube-system -l app. | Follow the controller's decisions |
kubectl annotate node NODE karpenter. | Shield one node from graceful methods |
Scaling and metrics
metrics-server
1.What is Metrics Server ?
Metrics Server asks every kubelet for current CPU and memory use and keeps it in memory. It serves the numbers through the Metrics API. That is what makes kubectl top and the HPA work.
- Scrapes kubelets about every 15s
- Keeps only the latest sample (no history)
- Only CPU and memory, for Nodes and Pods
2.Why it exists ?
Kubernetes core does not ship a source of live Pod and Node usage.
Scraping every kubelet yourself means each consumer redoes auth, discovery and aggregation. Metrics Server is the reference implementation of the Metrics API.
kubectl topfails : "Metrics API not available"- HPA shows <unknown> targets
- No CPU / memory for autoscaling
- VPA recommender has no live data
3.Resource Metrics Pipeline
Metrics Server pulls from the kubelet. Consumers only talk to the API server, never to kubelets.
- cAdvisorReads cgroup stats per container (inside kubelet).
- kubeletExposes /metrics/resource.
- APIServicev1beta1.metrics.k8s.io points the aggregator at the Service.
- Servicemetrics-server in kube-system, port 443.
4.kubectl top pods : Request Flow
If the APIService is Available=False, kubectl top fails even when the Pod is Running.
5.Metrics Server vs Others
| Tool | Answers | Data kept |
|---|---|---|
| metrics-server | CPU / memory now ? | Latest, in memory |
| Prometheus | Behaviour over time ? | Time-series on disk |
| kube-state-metrics | State of objects ? | Object state, no usage |
- top memory = working set, not
freeordocker stats --metric-resolution= scrape interval (15s)- Not a monitoring source : never scrape it with Prometheus
kubectl top and the HPA. No history, so use Prometheus for that.1.HPA Uses It
The HPA asks the Metrics API for Pod usage, compares it with the target, then scales the Deployment.
ratio = currentUtil / targetUtil
desired = ceil(replicas x ratio)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: web
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: web
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
2.Sizing and Versions
| Metrics Server | 0.9.x | 0.8.x | 0.7.x | 0.6.x |
|---|---|---|---|---|
| Kubernetes | 1.34+ | 1.31+ | 1.27+ | 1.25+ |
| Default | 100m CPU, 200MiB (~100 nodes) |
| Per node over 100 | +1m CPU, +2MiB |
| HA | 2 replicas (high-availability.yaml) |
3.Commands
| Command | What it tells you |
|---|---|
| kubectl apply -f .../components.yaml | Installs Deployment, Service, APIService |
| helm install metrics-server metrics-server/metrics-server -n kube-system | Same install, with Helm |
| kubectl top nodes / kubectl top pods -A --sort-by=memory | Live CPU and memory per Node or Pod |
| kubectl get apiservice v1beta1.metrics.k8s.io | Available=True ? If not, describe it for the reason |
| kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes | Raw Metrics API, bypasses kubectl top |
| kubectl -n kube-system logs deploy/metrics-server | Scrape errors : TLS, no route to node |
4.Gotchas
- x509 errors on kind or minikube :
--kubelet-insecure-tlsis for testing only. Real fix is kubelet certs signed by the cluster CA. - "No route to host" : set
--kubelet-preferred-address-types(e.g. InternalIP). - Aggregation layer must be on. Kubelet needs webhook authn / authz, else 401 / 403.
kubectl get hpashows <unknown>/50% : Pods have no requests, or Metrics Server is down.- New Pods have no metrics for a scrape or two. Memory is working set, so it differs from
free. - Metrics API is metrics.k8s.io/v1beta1 : confirm with
kubectl api-versions | grep metrics.
kubectl top fails, and never keep --kubelet-insecure-tls outside a lab.Scaling and metrics
kube-state-metrics
1.What is kube-state-metrics (KSM) ?
A small service that watches the Kubernetes API and turns the state of objects (Pods, Deployments, Nodes...) into Prometheus metrics on /metrics.
- Metrics come from object state, not usage
- Read only : needs just list + watch RBAC
- Stateless Go binary, keeps objects in memory
- Metrics on
:8080, self metrics +/readyzon:8081
2.Why do we need it ?
- metrics-server gives only live CPU / memory
kubectlshows state only right now, no history- We want alerts like CrashLooping Pod or PVC Pending
Fix : KSM exposes object state as time series that Prometheus can scrape, graph and alert on. It shows raw API data, so values match the objects.
- Sits in kube-system or monitoring
- Runs as Deployment + Service + RBAC
- Pods, Deployments, Nodes, Jobs, PVCs, HPAs...
- Secrets : metadata only, never the data
- Ships no CRDs of its own
3.KSM vs metrics-server
Usage vs state. They complement each other, not replace.
4.Sharding and Scale
One instance holds all watched objects in memory, so memory grows with the cluster.
--shard=N --total-shards=M, total 1 means off- Auto sharding : a StatefulSet, ordinal = shard index
- Scrape filter :
/metrics?resources=pods - 2 replicas without sharding = duplicate metrics
5.Handy Metrics Cheat Sheet
| Metric | Meaning |
|---|---|
| kube_ | Pod phase (Pending / Running / Failed...) |
| kube_ | Restart counter, CrashLoop detection |
| kube_ | Why waiting (e.g. CrashLoopBackOff) |
| kube_ | Desired replicas |
| kube_ | Available replicas (lower than desired = stuck rollout) |
| kube_ | Node Ready, MemoryPressure... |
| kube_ | PVC Bound / Pending |
| kube_ | Info metric with K8s labels (allowlisted ones only) |
6.Gotchas
- Labels missing ? Set
--metric-labels-allowlist(e.g. pods=[app,team]) - No CPU / memory here, use metrics-server or cAdvisor
- Custom Resource State is feature-frozen, needs RBAC on the CRD and the custom resource
- Big cluster ? Use shards,
--resources, namespace filters - Helm chart lives in prometheus-community, versioned separately
kind: CustomResourceStateMetrics
spec:
resources:
- groupVersionKind:
group: myteam.io
version: v1
kind: Foo
metrics:
- name: active_count
each:
type: Gauge
gauge:
path: [status, uptime]
Passed with --custom-resource-state-config-file.
7.KSM Commands
| Command | Description | Example |
|---|---|---|
| helm install kube-state-metrics prometheus-community/kube-state-metrics -n monitoring --create-namespace | Install with the prometheus-community chart | helm repo add prometheus-community https://prometheus-community.github.io/helm-charts |
| kubectl apply -f examples/standard | Install from the repo manifests (uses kube-system) | git clone https://github.com/kubernetes/kube-state-metrics |
| kubectl -n monitoring port-forward svc/kube-state-metrics 8080:8080 | Reach the metrics endpoint locally | curl -s localhost:8080/metrics | grep kube_deployment_status_replicas |
| curl 'localhost:8080/metrics?resources=pods' | Scrape only some resource types | curl -s 'localhost:8080/ |
| kubectl -n monitoring logs deploy/kube-state-metrics | Debug RBAC forbidden errors on list / watch | kubectl auth can-i list pods --as=system:serviceaccount:monitoring:kube-state-metrics |
| kubectl -n monitoring get pod -l app.kubernetes.io/name=kube-state-metrics | Check the pod, /readyz on :8081 is the probe | curl -s localhost:8081/readyz |
Workload identity
SPIFFE
1.Why SPIFFE ?
SPIFFE is an open standard, not software. Every workload gets a name (SPIFFE ID) and a short-lived ID card (SVID). Services prove who they are without static secrets or IP addresses.
- Problem : how does service A know the caller is really service B ?
- IPs change and Pods are short-lived, so network identity breaks
- Static secrets (API keys, long-lived certs) leak and are rarely rotated
- Fix : one name format, one document format, one local API
- Zero trust : authenticate by cryptographic identity, not by network position
2.The Pieces
SPIFFE is the spec. SPIRE is the main implementation of it. Do not mix them up. Both are CNCF graduated.
- CSI driver : spiffe-csi v0.3.0 (the spire 0.30.4 chart still installs 0.2.7)
- Libraries : go-spiffe, java-spiffe, spiffe-helper
- SPIFFE IDThe name.
spiffe://<domain>/<path>, path meaning is up to the admin. - Trust domainIdentity boundary and root of trust. Lowercase, max 255 bytes, self-registered.
- SVIDThe document carrying the ID : X.509, JWT or WIT.
- Trust bundleJWKS of a domain's public keys.
- Workload APILocal gRPC API, Unix socket, no auth handshake.
3.Anatomy of an Identity
Scheme + trust domain + path = SPIFFE ID. No query, no fragment. Path / is a Kubernetes convention, not required.
| X.509-SVID | JWT-SVID | WIT-SVID | |
|---|---|---|---|
| Format | Certificate | Signed token | JWS compact, WIMSE |
| ID in | URI SAN | sub | sub |
| Replay | Low, proves key | Bearer, until exp | Low, key proof |
| Best for | mTLS | L7 hops, proxies | WIMSE use cases |
| Stable ? | Yes | Yes | Incubating |
| Check | Chain to bundle | Audience + exp | Possession proof |
1.Getting an Identity
The Pod never holds a secret to ask for its identity. The platform vouches for it.
Agent streams SVID + bundle back to the Pod and pushes again on rotation.
2.mTLS and Federation
Two workloads present X.509-SVIDs and check each other with the right trust bundle. Different trust domains swap bundles at a bundle endpoint.
https_web: Web PKIhttps_spiffe: authenticated with an SVID- Refresh on
spiffe_, the producer's interval in seconds (5 min if it is absent, and SPIRE's default)refresh_ hint - Short TTL means few revocation lists
3.In a Pod
The spiffe-csi driver (csi.spiffe.io) mounts the socket read-only, so the Pod needs no hostPath. Libraries find the socket through SPIFFE_.
volumeMounts:
- name: spiffe-workload-api
mountPath: /spiffe-workload-api
readOnly: true
env:
- name: SPIFFE_ENDPOINT_SOCKET
value: unix:///spiffe-workload-api/spire-agent.sock
volumes:
- name: spiffe-workload-api
csi:
driver: "csi.spiffe.io"
readOnly: true
1.Who is Who
SPIFFE is the spec. Other tools implement it, reuse its IDs, or get replaced by it.
| Thing | Issues SVIDs ? | Attests ? |
|---|---|---|
| SPIRE | Yes | Yes, node + workload |
| Istio / Consul | Yes, own CA | Partial (SA / ACL token) |
| cert-manager | SPIFFE-style certs | No, policy only |
| Pod Certificates | If its signer puts the ID in | API server checks node, Pod, SA |
| SA token | No, K8s API JWT | No |
| Static secret | No | No, holder wins |
2.Commands
| Command | Description | Example |
|---|---|---|
kubectl apply -f csidriver.yaml | Register the spiffe-csi driver | kubectl get csidriver csi.spiffe.io |
kubectl exec <pod> -- ls / | Check the socket is mounted | shows spire-agent.sock |
kubectl exec <pod> -- env | grep SPIFFE | Libraries find the socket here | SPIFFE_ |
spire-agent api fetch x509 -socketPath <sock> | Fetch an X.509-SVID like a workload (SPIRE CLI) | socket path from above |
openssl x509 -in svid.pem -noout -text | Inspect the URI SAN | URI:spiffe:// |
kubectl logs -n <ns> ds/ | Debug mount failures (name depends on install) | add -c spiffe-csi-driver |
first ? that fails
3.Gotchas
- SPIFFE is a spec, you install an implementation (SPIRE)
- SPIFFE has no CRDs. ClusterSPIFFEID comes from spire-controller-manager (check apiVersion first)
- SPIFFE ID is identity, not authorization. You still need policy
- Trust domains are lowercase and self-registered, so use one you own
- Write the trust domain in lowercase (the spec says MUST). Scheme and trust domain compare case-insensitively, the path case-sensitively
- Share the socket dir by hostPath between agent and CSI driver, or a driver restart breaks mounts
- The CSI driver itself needs a privileged namespace (hostPath). Pods that mount its volume still pass
restricted - Broken CSI driver means Pods fail to terminate fully
- JWT-SVIDs are bearer tokens : check audience, keep TTL short
- Clock skew or failed rotation shows up as sudden TLS errors
Workload identity
SPIRE
1.Why SPIRE ?
Pods talk to each other, but how does Pod A prove it is Pod A ? IPs change, so an IP is a bad identity.
- Static secrets (API keys, long-lived certs) get leaked, copied and never rotated
- Service Account tokens prove identity only to the k8s API, not across clusters or clouds
- SPIRE checks facts (node, namespace, SA, image) then issues a short-lived SVID
- Result : zero-trust mTLS with no secret to store
SPIFFE = the standard. SPIRE = the software. CNCF Graduated (TOC vote 2022-08-23, announced 2022-09-20).
2.Components
The hardened chart's recommended layout puts the server in spire-server and the agents in spire-system. One server, one agent per node.
- Node attestor proves the node (options in 3.)
- Workload attestor inspects the caller (k8s via kubelet, unix via uid/gid)
- SVID = X.509 cert (mTLS) or JWT (bearer calls)
- Trust domain = the security boundary, its trust bundle holds the CA certs
- SPIRE ServerSigning authority. Holds entries, issues SVIDs. Usually a StatefulSet.
- SPIRE AgentDaemonSet. Attests to the server, then attests local workloads.
- Workload APIUnix socket where a workload asks "who am I ?".
- controller-managerTurns CRDs into registration entries.
- SPIFFE CSI driverMounts the agent socket into Pods.
3.Attestation and SVID Flow
Two steps : the agent proves the node, then identifies the process.
| Node attestor | Proves with | Catch |
|---|---|---|
| k8s_psat | Projected SA token | Default on k8s |
| aws_iid | EC2 identity doc | AWS only |
| gcp_iit | GCE identity token | GCP only |
| azure_msi | Managed identity | Azure only |
| join_token | One-time token | Manual, tests |
| x509pop | Existing node cert | Needs your PKI |
one per agent, the server must run the same plugin
1.Registration Entries
Entry = SPIFFE ID + parent ID + selectors. The parent ID points at the agent allowed to hand it out.
ID looks like spiffe://
| Kind (cluster scope) | Use |
|---|---|
| ClusterSPIFFEID | Which pods get which ID |
| ClusterFederatedTrustDomain | Federate with another trust domain |
| ClusterStaticEntry | Hand-written entry for non-pod things |
all spire.
2.k8s Automation
spire-controller-manager is a reconcile loop, like a Deployment controller, but the output is SPIRE entries.
kind: ClusterSPIFFEID
metadata: {name: web}
spec:
className: spire-mgmt-spire
spiffeIDTemplate: "spiffe://{{ .TrustDomain }}\
/ns/{{ .PodMeta.Namespace }}\
/sa/{{ .PodSpec.ServiceAccountName }}"
podSelector:
matchLabels: {app: web}
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: defaultapiVersion: spire.spiffe.io/v1alpha1 . Pods labelled app: web in default get an ID built from their Namespace and ServiceAccount. className names the chart's controller manager (release namespace, then release): without it, the CR is ignored. A workload fetches it over the Workload API socket, no credentials needed.
web in default getsspiffe://<trust-domain>/ns/default/sa/web 1.Install and Inspect
Two Helm charts from the hardened repo, CRDs first, then server + agent + controller manager, both released into spire-mgmt.
| Command | Description |
|---|---|
helm upgrade --install --create-namespace -n spire-mgmt spire-crds spire-crds --repo https:// | Install the CRDs first (release namespace: spire-mgmt) |
helm upgrade --install -n spire-mgmt spire spire --repo https:// | Server, agent, controller manager. The values set the trust domain, the cluster name and global. |
kubectl get pods -n spire-server | Server and controller manager (agents and CSI driver: -n spire-system) |
kubectl exec -n spire-server spire-server-0 -c spire-server -- spire-server entry show | List registration entries |
spire-server entry create -spiffeID <id> -parentID <id> -selector <sel> | Create an entry by hand, e.g. -selector k8s:ns:default |
spire-server entry delete -entryID <id> | Delete an entry |
kubectl get clusterspiffeid | List ClusterSPIFFEID objects |
kubectl logs -n spire-system -l | Agent logs when attestation fails |
2.No SVID ? Debug Order
= the bug
3.Gotchas
- Install the CRDs before the main chart (separate spire-crds chart)
- Server needs persistent storage (SQLite by default) or entries and CA state are lost, prod usually wants external SQL
- Charts are beta and the controller API is v1alpha1, use released versions not git main
- Recheck versions before quoting (latest seen : v1.15.5, chart spire 0.30.4)
- A typo in a selector like
k8s:nsmeans silently no SVID - Parent ID must match the agent that really runs the workload
- SVIDs are short-lived, apps must reload certs via the Workload API or a library
- Not a secret store, that is Vault or External Secrets
entry show, and when no SVID shows up, look at selectors, parent IDs and the agent logs.Networking and traffic
MetalLB
1.Why MetalLB ?
On bare metal a LoadBalancer Service stays EXTERNAL-IP <pending> forever: no cloud controller hands out IPs.
- NodePort is ugly (high ports, clients must know node IPs)
- Cloud LBs (ELB, GCP LB) are not available on-prem, in homelabs or at the edge
- MetalLB = IP allocation from your pools + announcement to the network (L2 or BGP)
Fills in type: LoadBalancer, the Service type after NodePort.
2.Components
Installs into metallb-system, idle until you create a pool and an advertisement.
- Two jobs : assign IPs, then advertise them
- Works with IPv4, IPv6 and dual stack
- Config is Custom Resources only
- controllerDeployment. Watches Services, picks an IP from an IPAddressPool, writes it to the Service status.
- speakerDaemonSet. Announces the IP via ARP/NDP or BGP. Uses memberlist (7946 TCP/UDP) to spot dead nodes.
- frr-k8sDefault BGP backend. Wrapper around FRR, adds BFD and IPv6.
3.L2 Mode Flow
Controller allocates, speaker announces. Only the leader node answers ARP/NDP.
4.BGP Mode
Every node peers with your routers and announces the Service IP. Traffic is spread across nodes (ECMP).
- Needs a BGP-capable router
- Backends : FRR-K8s (default), native, legacy FRR (deprecated)
| Aspect | L2 (ARP/NDP) | BGP |
|---|---|---|
| Needs | Same LAN | BGP router |
| Load balancing | No, one node per IP | Yes (ECMP) |
| Failover | New leader, brief gap | Route withdrawn, BFD speeds it up |
| Setup | Very low | Higher (ASNs, peers) |
| Use | Homelab, small LAN | Datacenter, larger scale |
BGP = real load spreading across nodes.
1.Config Objects (CRDs)
Before v0.13.0 config was a ConfigMap. Now it is CRDs in metallb-system. Convert old config with quay..
| Kind | apiVersion | Purpose |
|---|---|---|
| IPAddressPool | v1beta1 | IP ranges to allocate |
| L2Advertisement | v1beta1 | Announce via ARP/NDP |
| BGPPeer | v1beta2 | Router to peer with |
| BGPAdvertisement | v1beta1 | Announce via BGP |
| BFDProfile | v1beta1 | BFD timers |
kind: IPAddressPool
metadata: {name: first-pool, namespace: metallb-system}
spec:
addresses: [192.168.10.0/24]
---
kind: L2Advertisement
metadata: {name: example, namespace: metallb-system}
spec:
ipAddressPools: [first-pool]
- Both use
apiVersion: metallb.io/ v1beta1 - A Service can pick a pool with the annotation
metallb.io/ address-pool - A pool with no advertisement is unreachable
- An L2Advertisement with no
ipAddressPoolsannounces every pool
2.Commands
| Command | Description |
|---|---|
| kubectl apply -f metallb-frr-k8s.yaml | Install, FRR-K8s backend (recommended). metallb-native.yaml is L2 only |
| helm install metallb metallb/metallb -n metallb-system --create-namespace | Install with Helm, after helm repo add |
| kubectl get pods -n metallb-system | Check controller and speaker |
| kubectl get ipaddresspools -n metallb-system | List the pools |
| kubectl logs -n metallb-system ds/speaker | Debug announcements and leader election |
| kubectl describe svc <name> | Events show IP and announcing node |
3.Gotchas
- kube-proxy in IPVS mode needs
ipvs.strictARP: trueor L2 misbehaves (ipvs is deprecated since v1.35) - Install alone does nothing: no pool, no advertisement, no IP
- Pool IPs must be unused on your LAN (outside the DHCP range)
- Port 7946 (TCP/UDP) must be open between nodes for memberlist
- L2 is failover, not load balancing. Legacy FRR mode is deprecated
Networking and traffic
Gateway API CRDs
1.What is the CRD Bundle ?
One YAML file per release of kubernetes-sigs/. You apply it once and the cluster learns the Gateway API kinds.
- Defines the kinds, their schema and CEL validation
- Cluster-scoped : one copy, shared by every controller
- Versions apart from Kubernetes core, latest
v1.6.3 - Each CRD is annotated with its
bundle-versionandchannel
2.Standard vs Experimental
Two release channels, same version number. Experimental is a superset of Standard.
- Standard : only v1 kinds and fields
- Experimental : adds alpha fields, plus kinds in group
gateway.networking. x-k8s. io
| Standard | Experimental | |
|---|---|---|
| Stable | No breaking changes in a version | May break any release |
| File | standard- | experimental- |
| Use | Production | Trying features |
| Rollback | Safe (pin version) | Can fail if objects use experimental fields |
3.Versions and Graduation
| Kind | Role | Points to |
|---|---|---|
| GatewayClass | Names the controller | controller |
| Gateway | Listeners, the LB | GatewayClass |
| HTTPRoute | L7 host, path, header | Gateway |
| GRPCRoute | L7 gRPC method | Gateway |
| BackendTLSPolicy | TLS to backend | Service |
| TLSRoute | Route by SNI | Gateway |
| ReferenceGrant | Allows cross-ns refs | its own ns |
| ListenerSet | Extra listeners | Gateway |
| TCPRoute / UDPRoute | L4 port routing | Gateway |
Only GatewayClass is cluster-scoped, every other kind lives in a namespace.
1.Install : CRDs vs Controller
One kubectl apply registers the CRDs. The controller, installed apart, does the real work.
- Always apply with
--server-side - Install the CRDs first, then the controller
- Without a controller objects are stored, status stays empty
- Controller writes
AcceptedandProgrammedto status
kubectl get gatewayclass, ACCEPTED = a controller owns it.2.Upgrade and Uninstall
An upgrade is the same apply with a newer file. The risk is what the new version drops.
- A removed version (v1alpha2) must not be in
storedVersions - New required fields can reject old objects
- The controller must support the new bundle version
kubectl delete -f with the same file. Deleting a CRD deletes every object of that kind, so every Gateway and Route goes with it.3.Commands
| Command | Description | Example |
|---|---|---|
kubectl apply --server-side -f standard-install. | Install or upgrade | pin the v1.6.3 URL |
kubectl get crd | grep gateway. | List the CRDs | k8s.io, x-k8s.io |
kubectl get crd gateways.gateway.networking.k8s.io -o yaml | grep bundle | Bundle version | v1.6.3, standard |
kubectl get crd tcproutes.gateway.networking.k8s.io -o jsonpath='{.status.storedVersions}' | Versions in etcd | lists v1alpha2 = migrate first |
kubectl apply --server-side --force-conflicts -f <file> | Take over fields | after Helm owned them |
kubectl delete -f <same file> | Uninstall the CRDs | drops all Routes too |
4.Helm-Managed CRDs
Many controller charts ship the Gateway API CRDs. Helm treats them differently by folder.
| Helm action | crds/ folder | templates/ |
|---|---|---|
| install | Created | Created |
| upgrade | Skipped (stale) | Updated |
| uninstall | Kept | Deleted, with all objects |
metadata.= client-side apply, addannotations: Too long --server-side- Two charts, two bundle versions : every upgrade flips the CRDs
- One chart's experimental CRDs switch the channel for every controller
- Old admission webhook left over : remove it, CEL replaced it
templates/CRDs : addresource-policy: keep- v1.6.0 : ReferenceGrant spec now required
Networking and traffic
Gateway API
Gateway API : the role-based, portable successor to Ingress
1.Why Gateway API ?
Typed routing objects, split by who owns what. This topic is how routing behaves. The CRDs, channels and install are in the Gateway API CRDs section.
- Match header, query, method, not just host + path
- Redirect, rewrite, mirror : typed filters
- Canary is a
weighton backendRefs - Timeouts per rule, TLS Terminate or Passthrough
- ingress-nginx was archived 2026-03-24, no more fixes
Ingress is still supported but frozen. Gateway API is the upstream successor (SIG Network).
2.Resource Model
Each role edits its own object. parentRefs / gatewayClassName point up to the parent; backendRefs point down to the backend.
| Field | On | Points to |
|---|---|---|
| controllerName | GatewayClass | Controller (up) |
| gatewayClassName | Gateway | GatewayClass (up) |
| parentRefs | Route | Gateway (up) |
| backendRefs | Route rule | Service (down) |
| certificateRefs | Listener | Secret (TLS) |
| hostnames | Route | Listener hostname |
GatewayClass is cluster scoped, everything else is namespaced. A Route can list several parentRefs (two Gateways, or one listener by sectionName). Route and Listener hostnames must overlap.
3.Request Flow
The Gateway status gives the address. The listener handles TLS, then a Route rule picks the backend.
| Mode | Who decrypts | Route |
|---|---|---|
| Terminate | Gateway, cert in certificateRefs | HTTPRoute, GRPCRoute |
| Passthrough | Backend, Gateway reads SNI only | TLSRoute |
status.addresses = the LB IP clients hit.1.Matching and Filters
A rule = matches + filters + backendRefs, checked in that order.
| Field | Example | Note |
|---|---|---|
| matches | ||
| path | PathPrefix /api | Exact, Prefix, Regex |
| headers | env = canary | Exact (Core), regex (impl.) |
| queryParams | v = 2 | Exact (Extended) |
| method | POST | Extended |
| filters | ||
| RequestHeaderModifier | set, add, remove | Core filter |
| RequestRedirect | to https, 301 | Core, no backend |
| URLRewrite | /v1/x to /x | prefix or full path |
| RequestMirror | copy to shadow | reply dropped |
| rule | ||
| timeouts | request: 10s | backendRequest per try |
- Tie break : Exact path, longest prefix, method, most headers
- No rule matches : 404
2.Gateway + Canary Route
kind: Gateway
metadata: {name: my-gw, namespace: infra}
spec:
gatewayClassName: my-class
listeners:
- {name: http, port: 80, protocol: HTTP,
allowedRoutes: {namespaces: {from: All}}}
---
kind: HTTPRoute
metadata: {name: web, namespace: app}
spec:
parentRefs: [{name: my-gw, namespace: infra}]
rules:
- backendRefs:
- {name: web-svc, weight: 90}
- {name: web-canary, weight: 10}
- Operator creates the Gateway
- Controller sets
status.addresses - Dev creates the HTTPRoute in ns app
- allowedRoutes must permit ns app
- Controller sets Accepted / ResolvedRefs
parentRefs. The Gateway must allow it with allowedRoutes (default : same ns).
3.Status and ReferenceGrant
| On | Condition | Reasons when False |
|---|---|---|
| Gateway | Accepted | ListenersNotValid |
| Gateway | Programmed | Pending, AddressNotAssigned |
| Listener | ResolvedRefs | InvalidCertificateRef |
| Route | Accepted | NotAllowedByListeners |
| Route | ResolvedRefs | BackendNotFound, RefNotPermitted |
| Command | Shows |
|---|---|
kubectl describe gateway my-gw -n infra | Listener conditions, attachedRoutes |
kubectl describe httproute web -n app | status.parents per Gateway |
kubectl get referencegrant -A | Who may point across ns |
ingress2gateway print | Convert Ingress to Gateway YAML |
Networking and traffic
kgateway
1.What is kgateway ?
kgateway is a control plane that watches Gateway API resources and configures Envoy proxies to route traffic into your cluster.
Think "Ingress, but modern, with Envoy doing the work".
- Formerly Gloo Gateway (OSS), made by Solo.io
- CNCF Sandbox project (accepted 2025-03-04)
- Control plane is Go, talks to Envoy over xDS
- Default GatewayClass :
kgateway - Latest stable seen : v2.4.7
2.kgateway on Gateway API
Why Gateway API, and who owns what : the Gateway API section. Here is what kgateway does with each object.
GatewayParameters.| Object | kgateway does |
|---|---|
| GatewayClass | Class kgateway, made by the chart |
| Listener | Envoy listener : port, protocol, TLS |
| HTTPRoute | Envoy routes, pushed over xDS |
| TrafficPolicy | Extras, attached by targetRefs |
| Protocols | HTTP, gRPC, TCP/TLS |
3.Control Plane vs Data Plane
status flows back to the Gateway (Accepted, Programmed)
- ControllerPod in
kgateway-system. - EnvoyAuto-created per Gateway. Applies routes and policies.
4.kgateway CRDs
All in group gateway., version v1alpha1.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: httpbin}
spec:
parentRefs: [{name: http}]
hostnames: ["www.example.com"]
rules:
- backendRefs:
- {name: httpbin, port: 8000}
| Kind | Use it for |
|---|---|
| GatewayParameters | Customize generated proxy |
| TrafficPolicy | Auth, rate limit, CORS, retry, JWT |
| Backend | Lambda, static, GCP upstreams |
| BackendConfigPolicy | Upstream TLS, health checks |
| HTTPListenerPolicy | Listener settings, access logs |
| DirectResponse | Fixed reply, no backend |
| GatewayExtension | Reusable ext auth config |
5.Handy Commands
| Command | Description | Example |
|---|---|---|
helm upgrade -i kgateway-crds | Install kgateway CRDs chartGateway API CRDs first | Gateway API CRDs first |
helm upgrade -i kgateway | Install the control planeget pods -n kgateway-system | get pods -n kgateway-system |
kubectl get gatewayclass | Confirm class kgateway existsget gatewayclass kgateway | get gatewayclass kgateway |
kubectl get gateway,httproute -A | Check Gateways and routesget gateway http -n kgateway-system | get gateway http -n kgateway-system |
kubectl get svc,deploy -n <ns> | Find Envoy Deploy + Svcget svc -n kgateway-system | get svc -n kgateway-system |
kubectl describe httproute <name> | Read route statusdescribe httproute httpbin | describe httproute httpbin |
kubectl get trafficpolicy -A | List policiesget trafficpolicy -A | get trafficpolicy -A |
<chart> is oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds (and .../kgateway), with -n kgateway-system --version v2.4.7. Add --create-namespace on the first install.6.Gotchas
- Spell the class
kgatewayexactly or nothing programs. - Install Gateway API CRDs first.
- Do not use ports 19000, 8082, 9091 for listeners.
- Own CRDs are v1alpha1, so APIs can change.
- Cross namespace backendRefs need a ReferenceGrant.
- Route stuck ? Read HTTPRoute status first.
- Policies attach by
targetRefsor labeltargetSelectors. - TLS certs are Secrets (cert-manager can issue them).
- agentgateway is a separate Rust data plane for AI / MCP traffic.
- Quickstart chart version can lag the newest release.
Serverless
Knative Operator
Knative Operator : install Knative with two small YAML files
1.What is the Knative Operator ?
A controller that installs, upgrades and configures Knative Serving and Eventing for you.
- You write a KnativeServing or KnativeEventing CR
- The operator reconciles it into a full Knative
- Runs as two Deployments,
knative-operatorandoperator-webhook, in nsknative-operator(helm--create-namespace) - Part of the Knative project (CNCF graduated, TOC vote 2025-09-11)
Tip : a hand edit to a ConfigMap key the CR sets is reverted; a key the CR does not set stays. Edit the CR instead.
2.Why do we need it ?
- Many big manifests, right order
- Upgrades by hand, drift creeps in
- ConfigMaps get edited ad hoc
- HA and limits patch many Deployments
Fix : declare Knative in one CR. GitOps friendly.
Operator = one controller that owns every Knative Deployment, ConfigMap and RBAC rule it installs.
| Topic | Operator (CR) | Manual |
|---|---|---|
| Install | operator + 1 CR | CRDs, core, ingress |
| Upgrade | change spec.version | re-apply, fix drift |
| Config | spec.config | edit ConfigMaps |
| HA, limits | spec.workloads | patch each Deployment |
| Drift | operator reverts its keys | stays drifted |
| Extra part | yes, the operator | no |
Operator itself ?
3.The Two CRDs
Both use operator.. CR is named like its namespace.
| Kind | Namespace | Controls |
|---|---|---|
| KnativeServing | serving | Services, Revisions, autoscaler |
| KnativeEventing | eventing | Brokers, Triggers, Channels |
Serving answers HTTP requests. Eventing moves events to your code. Install either one, or both.
1.Key spec fields
- versionpins Knative, change it to upgrade
- config.<map>keys go into Knative ConfigMaps
- ingressistio, kourier, contour or gateway-api
- high-availabilityreplicas for control plane
- workloads / servicesreplicas, resources, nodeSelector, env
- registryimage rewrite, imagePullSecrets (air-gapped)
All of it lives under spec.
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: version: "1.24" ingress: kourier: enabled: true config: network: ingress-class: kourier.ingress.networking.knative.dev high-availability: replicas: 3
2.Networking layer
| Layer | spec key | Note |
|---|---|---|
| Istio | istio.enabled | install Istio first |
| Kourier | kourier.enabled | light, set class |
| Gateway API | gateway-api | CRDs + Gateway |
| Contour | contour | Envoy based |
Tip : turn on one networking layer at a time; the operator does not enforce it.
3.Upgrades
One minor at a time: a skip sets VersionMigrationEligible False. One minor down passes.
4.Handy Commands
| Command | What it does |
|---|---|
helm install knative-operator | Install the operator (add --create-namespace) |
kubectl get knativeserving -o yaml | status.version and conditions (the list shows only TARGET CLUSTER) |
kubectl logs -f deploy/ | First stop if not Ready |
kubectl describe knativeserving | See status.conditions |
- Create the namespace first
- Never hand-edit Knative ConfigMaps
- No networking layer means no routes
- Delete CRs before removing the operator
- Operator is one more thing to upgrade
- Keep the CR in ns
knative-serving - Ready Unknown, IngressNotConfigured ? no ingress yet
- spec.version must be one it ships
Upgrade ? bump spec.version
HA ? spec.high-availability.replicas
Serving ? request-driven apps, scale to zero
Eventing ? Brokers and Triggers
Serverless
Knative
1.What is Knative ?
Knative adds serverless behavior to Kubernetes. It is a CNCF graduated project (TOC vote 2025-09-11, announced 2025-10-08). Latest is v1.24.0.
- Serving : runs HTTP apps that autoscale, even down to zero pods.
- Eventing : routes CloudEvents between producers and consumers.
- Two independent parts, install either one alone.
- Plain Deployment + Service + Ingress + HPA means many YAMLs and no scale to zero.
Editing the template makes a new immutable Revision. Rollback and traffic split are cheap.
2.Serving : How a Request Flows
The activator holds requests when a Revision has zero pods, wakes it up, then forwards. It also stays in the path while spare room is short: with the default target-burst-capacity of 211, until three idle Pods are ready.
- controllerReconciles Service, Revision, Route.
- activatorBuffers requests at zero pods and on bursts.
- autoscalerReads metrics, sets replicas (KPA).
- queue-proxySidecar in every pod. Counts concurrency.
- webhookValidates and defaults resources.
3.Autoscaling
Default autoscaler is KPA, driven by concurrency or RPS. Scale to zero is on by default.
- Annotations
autoscaling.andknative. dev/ min-scale max-scale. - Cold start : first request waits for a pod. Use min-scale 1 for latency.
- HPA class: serving-hpa.yaml by YAML; the Operator always installs it. CPU needs metrics-server.
| Topic | Deployment + HPA | Knative Service |
|---|---|---|
| Scale to zero | No | Yes (default) |
| Scale signal | CPU / memory | Concurrency or RPS |
| Traffic split | Ingress or mesh tricks | Built in per Revision |
| Rollback | ReplicaSets, manual | Immutable Revisions |
| Objects to write | Deployment, Service, Ingress, HPA | One Service |
| Cold start | None | Yes |
1.Eventing : Broker and Trigger
Producers do not know consumers. Triggers decouple them.
- SourcePingSource, ApiServerSource, KafkaSource. Turns things into CloudEvents.
- BrokerEvent mailbox in one namespace.
- TriggerFilter + subscriber.
- ChannelLower-level pipe with Subscription.
2.CRD Cheat Sheet
| Kind | apiVersion | What it is |
|---|---|---|
| Service (ksvc) | serving. | Whole app, Config + Route |
| Configuration | serving. | Pod template, makes Revisions |
| Revision | serving. | Immutable snapshot |
| Route | serving. | URL to Revisions, percent split |
| Broker | eventing. | Event ingress |
| Trigger | eventing. | Filter + subscriber |
| InMemoryChannel | messaging. | Dev-grade channel |
| KnativeServing | operator. | Operator CR for Serving |
| KnativeEventing | operator. | Operator CR for Eventing |
3.Install Steps
1.kubectl Commands (Knative)
| Command | Description |
|---|---|
| kubectl apply -f serving-crds.yaml | Install Serving CRDs, then serving-core |
| kubectl apply -f serving-default-domain.yaml | Magic DNS via sslip.io (needs a LoadBalancer IP; YAML installs only) |
| kubectl apply -f operator.yaml | Install the Knative Operator |
| kubectl get ksvc -A | List services, URL, ready state |
| kubectl get revisions,routes,configurations | Objects a ksvc created |
| kubectl get pods -n knative-serving | Control plane health |
| kubectl describe ksvc <name> | Conditions show why not Ready |
| curl -H "Host: <url>" http://<INGRESS_IP> | Call a service with no DNS |
| kubectl get brokers,triggers,sources | Inspect eventing objects |
2.Minimal Knative Service
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: hello spec: template: metadata: annotations: autoscaling.knative.dev/max-scale: "5" spec: containers: - image: "ghcr.io/knative/helloworld-go\ @sha256:05c2c89a7a143b85682a69d9aa\ 9f1d0f2a0ec857ba83bd13b8317f2e5ddb13de"
3.Gotchas and Fit with Core K8s
- No networking layer: the ksvc still gets a URL, but stays not Ready (IngressNotConfigured).
- Revisions are immutable. Edit the Service, not the Revision.
- Gateway API networking is beta in install docs.
- Native brokers (Kafka, RabbitMQ) beat channel-based ones.
- A ksvc makes a Deployment, a Service and an Ingress-like route.
- Control plane lives in knative-serving and knative-eventing.
- Config is in ConfigMaps like config-autoscaler, config-network, config-domain.
Observability
OpenTelemetry Operator
1.What is the OTel Operator ?
A Kubernetes operator that deploys and manages OpenTelemetry Collectors and injects auto-instrumentation into your pods, all from YAML.
- Hand-writing Collector Deployments, ConfigMaps and Services drifts.
- Tracing normally means changing every app, per language.
- Many Collector replicas double-scrape Prometheus targets.
Operator = controller + CRDs + admission webhooks. The webhooks need TLS certs, so cert-manager is the documented prerequisite.
2.The Moving Parts
Collector and Instrumentation CRs do most of the work. The operator keeps them in sync.
Drift detected, operator updates it back.
- ManagerWatches CRs, creates workloads, ConfigMaps, Services.
- WebhooksValidate CRs and mutate pods for injection.
- Collector CRMode, image, receivers, processors, exporters, pipelines.
- Instrumentation CRExporter endpoint, propagators, sampler, language images.
- Target AllocatorSplits Prometheus targets across replicas.
- OpAMPBridge CRConnects Collectors to an OpAMP server for remote config and health.
3.Collector Modes
The CR picks how the Collector runs. The operator creates a Service <name>-collector.
| Mode | Shape | When to use |
|---|---|---|
| deployment | Stateless gateway | Central OTLP endpoint |
| daemonset | One per node | Node metrics, logs |
| statefulset | Stable pod names | Target Allocator |
| sidecar | Inside app pod | Per-pod isolation |
1.Auto-Instrumentation
No image change needed. A pod annotation opts the workload in, and a webhook mutates the pod.
| Annotation (pod template) | Value |
|---|---|
| instrumentation.opentelemetry.io/ inject-java | "true" |
| ...-nodejs / python / dotnet / go | "true" |
| ...-apache-httpd / nginx / sdk | same pattern |
| Other values | "name", "ns/name" or "false" |
- OTEL_SERVICE_NAMEworkload
- OTEL_EXPORTER_OTLP_ENDPOINTCR exporter
- OTEL_PROPAGATORSfrom the CR
- OTEL_TRACES_SAMPLERCR sampler
- OTEL_RESOURCE_ATTRIBUTESpod, node, ns
2.Target Allocator
Shards Prometheus scrape targets across Collector replicas so nothing is scraped twice. It can discover ServiceMonitor and PodMonitor CRs.
- Enabled with
spec..targetAllocator - Usually statefulset mode (stable pod names); daemonset needs the per-node strategy.
- It rewrites scrape_configs to
http_sd_configs. prometheusCR.picks up the monitor CRs.enabled: true
consistent-hashing (default)
least-weighted (fewest targets)
per-node (daemonset mode)
prometheus receiver in the Collector config. The allocator only hands out targets, the Collectors still scrape.1.Handy Commands
| Command | What it does |
|---|---|
kubectl apply -f opentelemetry-operator. | Install via manifest from GitHub releases |
helm install opentelemetry-operator open-telemetry/opentelemetry-operator | Install via Helm (repo open-telemetry) |
kubectl get opentelemetrycollectors -A | List Collector CRs |
kubectl get instrumentations -A | List Instrumentation CRs |
kubectl describe pod <app-pod> | Check init container + OTEL_* env |
2.Quick YAML
# Collector (v1beta1)
kind: OpenTelemetryCollector
spec:
mode: deployment
config:
receivers: { otlp: ... }
exporters: { debug: {} }
service: { pipelines: ... }
# Instrumentation (v1alpha1)
kind: Instrumentation
spec:
exporter:
endpoint: http://demo-collector:4318
propagators: [tracecontext, baggage]
sampler: { type: parentbased_traceidratio }
java:
image: <registry>/autoinstrumentation-java
- Collector CR: mode plus a normal Collector config.
- Instrumentation CR: where to send, how to propagate, how to sample.
- Endpoint is the
<name>-collectorService. - Apply both, then annotate the pod template.
- Annotation goes on the pod template, not the Deployment metadata.
- Create the Instrumentation CR first. Restart existing pods.
- Webhook cert not ready blocks CR and pod creation.
- Versions are 0.x. Read release notes before upgrading.
- Metrics : remote write or scrape into Prometheus.
- Logs : OTLP to Loki at
/otlp. - Traces : OTLP to Tempo on 4317 (gRPC) or 4318 (HTTP).
- Profiles : Pyroscope (OTLP profiles are alpha).
Observability
Prometheus and ServiceMonitors
1.What is Prometheus ?
A pull-based time-series database. It scrapes /metrics from discovered targets into a local TSDB, answers PromQL and sends alerts to Alertmanager.
- metrics-server : for
kubectl topand HPA. Not a source. - kube-state-metrics : a source of object state.
- Plainscrape 1m, timeout 10s, rules 1m, retention 15d
- Prometheus CRscrape 30s, rules 30s, retention 24h if unset
First query you run : up == 0.
2.Data model
- Series = name (
__name__) + labels. - New label value = new series.
- Targets add
job,instance. Scrapes addup,scrape_*. - Base units, plural suffix, counters end in
_total.
| Type | Behaves | Series | Query with |
|---|---|---|---|
| Counter | only goes up, resets to 0 on restart | _total | rate(), increase() |
| Gauge | goes up and down | the value | the value, avg_over_time() |
| Histogram | cumulative buckets, aggregatable | _bucket{le}, _sum, _count | histogram_ over rate |
| Summary | quantiles computed in the client | {quantile} | read it, cannot be aggregated |
Native histograms : one composite sample, far fewer series. Stable since 3.8.0, still opt-in with scrape_.
3.The scrape loop
relabel_configs pick targets and build __address__. metric_ run on every sample (not up). A breached sample_limit fails the whole scrape. Vanished series get a staleness marker; lookback is 5m.
4.PromQL essentials
xinstant,x[5m]range vector. Rate first, then aggregate :sum(rate(x[5m])), never rate a sum.rateneeds two samples : range of 4x the scrape interval or more (a convention). At 1m,[1m]is usually empty.histogram_must keepquantile leinby.absent(x)alerts on vanished metrics.
| Starter query | Shows |
|---|---|
up == 0 | targets that failed their last scrape |
sum by (job) ( | request rate per job |
histogram_ | p90 latency, classic histogram |
absent( | 1 when the api job has no series at all |
5.Where Kubernetes metrics come from
| Source | Port, path | Measures |
|---|---|---|
| node-exporter (DaemonSet) | :9100/metrics | node CPU, memory, disk, network |
| kube-state-metrics | :8080/metrics | object state from the API |
| kubelet (no Service : the operator makes one) | :10250/ | the kubelet's own metrics |
| cAdvisor (in the kubelet) | :10250/ | per-container usage |
| kube-apiserver | :6443/metrics | request rates and latencies |
| metrics-server | reads kubelet /metrics/ | latest CPU, memory : not a source |
rate first, then aggregate.1.Why an operator ?
- Config becomes CRDs (Operators).
- Instance CRDs become StatefulSets.
- Scrape CRDs : one gzipped config in Secret
prometheus-<name>. config-reloader:/-/reload, no restart.
Install : bundle.yaml (kubectl create, into default), kube-prometheus-stack (Helm), kube-prometheus (jsonnet).
Also : Probe, ThanosRuler, AlertmanagerConfig (v1alpha1).
2.The CRDs
| Kind | Version | Short | Role |
|---|---|---|---|
| Prometheus | v1 | prom | instance |
| PrometheusAgent | v1alpha1 | promagent | scrape, remote write only |
| Alertmanager | v1 | am | instance |
| ServiceMonitor | v1 | smon | scrape Services |
| PodMonitor | v1 | pmon | scrape Pods |
| ScrapeConfig | v1alpha1 | scfg | raw jobs, outside targets |
| PrometheusRule | v1 | promrule | rules |
3.ServiceMonitor anatomy
selectorpicks Services by label;namespaceSelector: own namespace by default.endpoints[].port= Service port name, not a number or Pod port name. OrtargetPort(container port name or number).path/metrics,intervalfrom the CR.relabelingsbefore the scrape,metricRelabelingsafter.job= Service name (default). Job :serviceMonitor/<ns>/<name>/<i>.- PodMonitor selects Pods directly : DaemonSets, sidecars, no Service (named ports : Services).
kind: ServiceMonitor
metadata: {labels: {release: kps}}
spec:
selector: {matchLabels: {app: web-app}}
endpoints:
- {port: web, interval: 30s}
4.How a ServiceMonitor gets picked
- Two levels :
serviceMonitorSelectormatches labels, andserviceMonitorNamespaceSelectorthe namespace. - kube-prometheus-stack wants
release: <release>. No label : silently ignored. Or setserviceMonitorSelectorNilUsesHelmValues: false(also pods, probes, scrape configs, rules).
| Value | Resource selector | Namespace selector |
|---|---|---|
{} | all objects | all namespaces |
| omitted (null) | nothing | own namespace only |
matchLabels | labelled objects | labelled namespaces |
5."My ServiceMonitor is not picked up"
- Labels match ? Namespace matched ?
- Rejected ?
InvalidConfigurationWarning Event. - In the generated config ?
- Discovered ? None : no Service. Dropped : port.
- RBAC : list/watch Services, Pods, Endpoint(Slice)s.
- Scraped ?
up == 0and its error.
| Command | Does |
|---|---|
kubectl get prometheus,servicemonitors,podmonitors,prometheusrules -A | list objects |
kubectl -n monitoring get secret prometheus-k8s -ojson | jq -r '. | in the config ? |
kubectl get events -n <ns> --field-selector involvedObject. | rejected ? |
kubectl -n monitoring port-forward svc/ | UI : Targets, SD pages |
promtool check config prometheus.yml | validate a config |
kubectl apply --server-side --force-conflicts -f monitoring. | CRDs, server-side |
{} = all, omitted = none.1.Recording rules and PrometheusRule
- Precomputes an expression every group
interval, stored as a new series. - Name :
level:metric:operations, e.g.job:http_.requests:rate5m
- PrometheusRule objects, picked by
ruleSelector+ruleNamespaceSelector. NilruleSelector= no rules. - Rendered into ConfigMaps
prometheus-<name>-rulefiles-<n>.
Test with promtool test rules. Alerting rules (for, keep_firing_for) : Alertmanager Routing.
2.Storage : head, WAL, blocks, retention
- Writes : in-memory head + WAL. Every 2h the head becomes a block.
- Compaction merges blocks (to 10% of retention or 31d). Retention deletes whole blocks, up to 2h late.
- Local disk, no NFS. Size retention at most 80 to 85% of the volume.
- Disk = retention s x samples/s x 1 to 2 bytes.
| Layer | Default retention | Watch out |
|---|---|---|
| Plain Prometheus | 15d | time or size, first to trigger wins |
Prometheus CR | 24h when nothing is set | no volumeClaimTemplate = emptyDir |
| kube-prometheus-stack | 10d | on emptyDir unless a volumeClaimTemplate is set |
Config-file retention since 3.8.0 (reloadable, flags deprecated). The operator uses it only for 3.11.0+ (3.8 to 3.10 : unit bug).
3.Cardinality
- Series count, not sample count, drives memory.
- Most metrics under 10 label combinations; check any over 100. No user IDs or emails as labels.
- Find :
/,api/ v1/ status/ tsdb promtool tsdb analyze. - Limit :
sample_limit, metric relabeldroporlabeldrop.
4.High availability and scale
| Option | What it does |
|---|---|
replicas: 2 | twin Pods, prometheus_; dedupe later |
shards: N | hash of __address__; no resharding |
| Agent | scrape + remote write; no queries, rules |
Remote write : a queue per endpoint reads the WAL (slow receiver, scraping goes on). About 25% more memory; down over about 2h = data lost. RW 2.0 still experimental. Tempo remote-writes span metrics here; KEDA runs PromQL.
5.Three ways to run it
| Way | CRDs | Discovery | Rules |
|---|---|---|---|
plain prometheus chart | no | prometheus.io/* annotations, by relabeling | ConfigMap |
| kube-prometheus-stack (Helm) | yes | ServiceMonitor and friends (13 out of the box) | 35 PrometheusRule objects |
| kube-prometheus (jsonnet) | yes | same | kubernetes-mixin |
- The operator ignores
prometheus.: leaving the plain chart drops those targets. Add a ScrapeConfig.io/ scrape - Helm never upgrades CRDs. Flux defaults :
install.crds: Create,upgrade.crds: Skip. Set bothCreateReplace.
| Command | Does |
|---|---|
promtool check rules rules.yml | lint rules |
promtool test rules tests.yml | unit-test rules |
promtool tsdb analyze <dir> | cardinality |
curl localhost:9090/ | top series |
curl localhost:9090/ | true version |
No operator : a community chart rendered once, applied by Flux. Annotation targets; most app metrics by remote write from the OTel gateway. Rules in a ConfigMap, promtool-tested.
Pitfall : K3s repeats apiserver and etcd series in the kubelet job, so the head grows until an OOMKill. Raising limits for duplicated series is the wrong fix. Do better : drop them at scrape and alert on head series.
Observability
Alertmanager Routing
1.Two halves
- Prometheus evaluates alerting rules and sends the firing alerts. Loki's ruler can send too (Kubernetes Events as a Signal).
- Alertmanager deduplicates, groups, inhibits, silences and routes them to receivers (Slack, PagerDuty, webhook, ...).
2.Alerting rules
| Field | Default | Meaning |
|---|---|---|
expr | required | PromQL; one alert per output series |
for | 0s | time in pending before firing |
keep_firing_for | 0s | keep firing after it clears (anti-flap); 2.42.0+ |
labels | none | identity and routing keys |
annotations | none | text; may change freely |
- No
$valuein labels : each evaluation is a new alert, sofornever completes. foronincrease(x[15m]) > 0is not "bad for that long" : one failure holds it true.
- alert: PodCrashLooping
expr: increase(
kube_pod_container_status_restarts_total[15m])
> 3
for: 5m
keep_firing_for: 10m
labels: { severity: warning }
annotations:
summary: "{{ $labels.namespace }}/\
{{ $labels.pod }} restarting"
On Kubernetes it lives in a PrometheusRule, selected by ruleSelector (Prometheus and ServiceMonitors). Restarts : kube-state-metrics.
3.The send loop
- Resent at most every 1m (
--rules.) withalert. resend-delay EndsAt = now + 4 x max(eval interval, resend-delay). - Pending is never sent; resolved is resent for 15m. Send to every replica, never via a load balancer.
EndsAt passes : a dead Prometheus looks like "all clear". A Watchdog catches it.4.Inside Alertmanager
- The dispatcher runs once per alert. A new group waits
group_wait. - Each flush runs the notifier stages in order. Inhibited and silenced alerts are dropped there (red).
- A sent group goes into the nflog, so other replicas dedup against it.
5.The three timers
| Timer | Default | What it does |
|---|---|---|
group_wait | 30s | delay the first message of a new group so related alerts join |
group_interval | 5m | pace for updates; also the send timeout, max( plus the peer wait |
repeat_interval | 4h | resend an unchanged group; rounded up to a multiple of group_interval |
amtool check-config says SUCCESS for repeat_interval < group_interval; only the server logs a warning at load. resolve_timeout does nothing for Prometheus alerts (they always carry EndsAt).
group_wait, updates every group_interval, repeats every repeat_interval.1.The routing tree
- Every alert enters the root
route: no matchers allowed, areceiverrequired. - It walks the children depth-first and stops at the first match, unless that child sets
continue: true. No child matches : the current node handles it. - Unset settings inherit from the parent.
group_by: ['...']means every label (no grouping). - First match wins : a catch-all above specific routes eats everything.
2.Matchers
- Operators
=,!=,=~,!~.matchers:replaces deprecatedmatchandmatch_re(still parsed). - RE2 regexes are anchored :
=~ "prod"missesproduction. - In YAML, quote a matcher holding
{ } [ ] , # | > : - Since 0.27 a UTF-8 parser runs in fallback mode; strict mode is opt-in (
utf8-strict-mode).
3.Inhibition and silences
- Inhibition mutes target alerts while a source alert fires and the
equallabels match. A label missing on both sides counts as equal. - The source must arrive first, so a very short
group_waitcan leak the target. - A silence is matchers plus a time window, made with
amtoolor the UI. Silences and the notification log are saved to disk every 15m and kept 120h.
| Way to mute | Trigger | Lifetime | Use for |
|---|---|---|---|
| Inhibit rule | another alert firing | config | critical mutes warning, node down mutes its Pods |
| Silence | matchers, by hand | start and end time, saved | maintenance, known issues |
mute_ | the clock | config | quiet hours on a route |
null receiver | route match | config | drop on purpose (InfoInhibitor) |
4.Receivers
- A receiver holds integrations; one with none is a
nullreceiver. - No RESOLVED in Slack or email unless
send_resolved: true. *_filefields (api_url_file,routing_key_file) read a mounted Secret (ConfigMaps and Secrets).- Go text templates (
.CommonLabels,.Alerts.Firing) :check-configparses but never runs them, so render them.
| Integration | send_resolved default |
|---|---|
| Slack | false |
| false | |
| webhook | true |
| PagerDuty | true |
Rocket.Chat is false in code, though the docs say true.
| Command | What it does |
|---|---|
amtool check-config alertmanager.yml | parse the config, print SUCCESS |
amtool config routes test --config. | receivers a label set reaches |
amtool alert add TestAlert severity=warning --annotation=summary=test | inject a test alert |
amtool silence add alertname=X -d 2h -c "maintenance" | silence for 2h (default 1h) |
amtool silence expire $(amtool silence query -q alertname=X) | end those silences now |
amtool template render --template. | render against example data |
amtool tests a tree before any rule fires.1.Operator objects
AlertmanagerCR (CRDs) : StatefulSetalertmanager-<name>, Servicealertmanager-operated(web9093,tcp-meshandudp-mesh9094). One replica : gossip off.- Config : Secret
alertmanager-<name>or AlertmanagerConfigs, merged intoalertmanager-<name>-generated(Operators). - AlertmanagerConfig is
v1alpha1(API Machinery), validated by a webhook (Admission). Namespace selector nil = own namespace.
- Operator v0.94.1 defaults to Alertmanager v0.34.0, the inhibition un-mute bug. Set
spec.version. limits.passes a flag Alertmanager lacks : crash-loop.maxPerSilenceBytes
2.Team routes : AlertmanagerConfig
- The operator appends
namespace="<ns>"to each top route and both inhibit sides (removes nothing from routes), forcescontinue: true, renames receivers, prepends the routes. - Alerts with no
namespace(node, cluster-wide) never reach a team route.
| Strategy | OnNamespace (default) | OnNamespace | None |
|---|---|---|---|
| Effect | namespace matcher added | same, except objects in Alertmanager's own namespace see all alerts | nothing injected |
3.High availability
- Replicas gossip silences and the nflog on 9094, TCP and UDP (TCP only with
--cluster.).tls-config - Each waits
position x 15sand skips if a peer already sent. Broken gossip fails open : duplicates. - Allow 9093 from Prometheus, 9094 TCP and UDP between replicas (Network Policy in Depth).
4.Watch the watcher
Watchdogalways fires (vector(1)). Route it to a heartbeat service : when pings stop, the pipeline is dead.- Both distributions ship it unwired. Set
group_intervalon its route too, orrepeat_interval: 1mbecomes 5m. - Self-monitoring :
alertmanager_,notifications_ failed_ total{reason} alertmanager_,config_ last_ reload_ successful alertmanager_; on Prometheus,notifications_ suppressed_ total prometheus_.notifications_ alertmanagers_ discovered
| Setting | kube-prometheus v0.19.0 | kube-prometheus-stack 92.1.1 |
|---|---|---|
| replicas | 3 | 1 (emptyDir) |
root group_by | [namespace] | ['namespace'] |
repeat_interval | 12h | 12h |
| receiver | Default | 'null' |
| Watchdog | empty receiver | routed to null |
| Command | Does |
|---|---|
kubectl -n monitoring get secret alertmanager-main-generated -o jsonpath='{. | merged config |
amtool cluster show | gossip status |
curl -XPOST localhost:9093/-/ | reload |
runbook but the template prints runbook_url. Do better : a Watchdog to a dead man's switch, scrape it, one label name everywhere.Observability
Loki and Log Pipelines with Alloy
1.Labels, streams, chunks
kubectl logsshows one container (Pods). Loki keeps every Pod's lines, found by label.- A stream = one tenant + one unique label set. Its lines are compressed into chunks; a small TSDB index maps label sets to chunks. Both live in object storage.
- A query picks streams by label, then greps, parses and aggregates the lines.
- Cardinality matters : 15 label names per series by default; under 100,000 active streams, even for a huge tenant.
2.Write path and read path
- The distributor hashes tenant + labels onto the ring. Quorum
floor(RF / 2) + 1: RF 3 needs 2 writes. - Out-of-order lines only within 1h of the stream's newest line (older docs say 2h), else
too_far_behind. - The compactor keeps one index per tenant per day and applies retention.
3.Labels vs structured metadata vs line
Structured metadata (Loki 3.0+) : kept with each line, not indexed, still queryable. Trace IDs go here (Tempo).
| Index labels | Structured metadata | Line | |
|---|---|---|---|
| indexed | yes (TSDB) | no | no |
| defines a stream | yes | no | no |
| good for | bounded values | pod, trace ID, user ID | the rest |
| query | {app="x"} | | trace_id="abc" after the selector | |= "abc", parsers |
| needs | any schema | tsdb + v13 | any |
4.Deployment modes
| Mode | Targets | Scale | Status |
|---|---|---|---|
| Monolithic | -target=all | up to about 20 GB/day | Helm chart default since 12.0.0 |
| Simple scalable | write, read, backend | about 1 TB/day | deprecated : Loki 4.0 will refuse to start |
| Microservices | one per component | very large | supported |
The chart moved to grafana-community on 2026-03-16. Chart 17.0.0 refuses minio.; the MinIO subchart goes on 2026-10-31.
5.Schema and retention
- New installs :
tsdb,v13,period: 24h. Structured metadata is on, so Loki will not start unless the active period isv13+tsdb. - A schema change cannot be rolled back : add a new period starting at a future date.
- Retention is off by default. The compactor runs it :
retention_enabled,delete_(required),request_ store retention_period. - Markers sit on local disk : a PVC (Storage) or 3.7's
deletion_.marker_ object_ store_ prefix
index/.1.What is Alloy ?
- Grafana's collector : an OTel Collector distribution with native Prometheus, Loki and Pyroscope pipelines.
- It replaces two retired agents : Grafana Agent (EOL 2025-11-01) and Promtail (EOL 2026-03-02, not part of Loki 3.7).
alloy convert --source-format=promtailmigrates. Positions and metric names change : dashboards break.
2.Alloy syntax in five lines
- Block :
<component..name> "<label>" { attr = value } - Export :
discovery.. References build a graph, so file order does not matter.kubernetes. pods. targets //comments, backtick raw strings for regexes.alloy fmt,alloy validate.--stability.levelgates experimental components.- UI on
127.0.0.1:12345(the chart sets0.0.0.0). Reload :/-/reloador SIGHUP.
3.A Kubernetes log pipeline
loki.source.file "pods" {
targets = discovery.relabel.pods.output
forward_to = [loki.process.pods.receiver]
file_match {
enabled = true // else the glob is literal
}
}
loki.process "pods" {
stage.cri {}
forward_to = [loki.write.default.receiver]
}
- A glob
__path__(/) needsvar/ log/ pods/*$1/*. log file_match.alloy validatestays quiet. - Set
alloy.. Persistmounts. varlog: true storagePath:/tmp/alloyloses positions. - No clustering on a log-tailing DaemonSet.
loki_write_dropped_entries_total . WAL experimental, off.
4.File tailing vs API tailing
| loki.source. | loki.source. | |
|---|---|---|
| reads | /var/log/pods on the node | kubelet log endpoint, through the API server |
| needs | hostPath, usually root (Pod Security) | read on pods/log (RBAC), no privileges |
| runs as | DaemonSet (DaemonSets) | Deployment, clustered |
| restart safety | positions file of offsets | positions file of last timestamps |
| cost | node disk, inotify | kubelet CPU and network |
| node logs | yes | no |
5.loki.process stages that matter
| Stage | Does |
|---|---|
stage.cri | strip CRI prefix, join partials |
stage.json / logfmt | extract fields |
stage.labels | bounded field to a label |
stage. | high-cardinality field, unindexed |
stage.drop / multiline | drop noise, join stack traces |
6.Other ways in
- Events :
loki., run once or clustered, never per node (Events).source. kubernetes_ events - OTLP :
otelcol.toreceiver. otlp otelcol., endpointexporter. otlphttp http://. It appendsloki:3100/ otlp /v1/logs, so the full path fails. Collector setup : OTel Operator.
1.Query shape
- A log query is a stream selector, then a pipeline : line filters
|=,!=,|~,!~(this regex is not anchored, unlike label matchers), a parser (json,logfmt,pattern,regexp), label filters,line_format. - A parse failure adds
__error__(filter__error__=""). One matcher must not match empty :{app=~".*"}is rejected.
2.Metric queries
Like PromQL : wrap a log query in rate, count_over_time, bytes_rate; unwrap a field for quantile_; sum by.
| Query | Gives |
|---|---|
{namespace="shop", container="api"} |= "error" != "timeout" | error lines, minus timeouts |
{job="ingress"} | json | status >= 500 | 5xx lines, parsed |
sum by (container) ( | errors per second, per container |
{job="x"} | trace_ | one trace's lines (metadata) |
quantile_ | p99 latency per route |
3.OTLP and its labels
- OTLP logs go to
/otlp/v1/logs. Loki 3.7.8 indexes 18 resource attributes (docs say 17, the code wins), dots as underscores :service_name,k8s_pod_name... The rest is structured metadata. - Grafana now advises demoting
k8s.pod.nameandservice.: replace the global list, or per tenant setinstance. id ignore_defaults: trueand list what to index. - A bare per-tenant
structured_rule loses : defaults are prepended, first match wins.metadata
4.Limits and discard reasons
| Reason | HTTP | Knob (default) |
|---|---|---|
rate_limited | 429 | ingestion_rate_mb 4, burst 6 |
per_ | 429 | 3MB |
stream_limit | 429 | max_ 5000 |
line_too_long | 400 | max_line_size 256KB |
too_far_behind | 400 | the 1h window |
max_ | 400 | 15 |
Counted in loki_. "too many outstanding requests" is the query queue : frontend max_ 2048, scheduler max_ 32000.
5.Running it
| Command | Does |
|---|---|
logcli query --since 1h --limit 100 '{namespace="shop"} |= "error"' | LogQL from a shell |
logcli series '{}' --since=1h --analyze-labels | find high-cardinality labels |
loki -config. | check a config and exit |
curl http:// | readiness, says ready |
alloy validate config.alloy | schema errors, not runtime ones |
kubectl debug -it loki-0 --image=busybox --target=loki | a shell : no shell in the image since 3.6.0 |
/var/log/pods over OTLP (OTel Operator), not Alloy. Pitfall : the OTLP default labels, k8s_pod_name included, against current advice. Do better : demote pod names to structured metadata, keep a few bounded labels.
rate or count_over_time to get numbers, and keep OTLP labels few.Observability
Kubernetes Events as a Signal
1.The Event object
- A namespaced API object that says why another object changed.
- The API reference : "informative, best-effort, supplemental data". Reasons and messages "may evolve with time".
- One stored object, two API views : core
v1andevents.k8s.io/v1(GA 1.19).v1beta1is gone since 1.25. - Short name
ev. Events about Nodes or PVs live indefault.
| core/v1 | events. | Meaning |
|---|---|---|
involvedObject | regarding | the object it is about |
message | note | text, max 1024 bytes if eventTime is set |
reason | reason | CamelCase cause, max 128 |
type | type | Normal or Warning |
reportingComponent | reportingController | who reported it (source is deprecatedSource) |
count | series.count | how many repeats |
lastTimestamp | series. | last seen |
2.Who writes Events
- Legacy recorder (core/v1) : kubelet and kube-controller-manager. A repeat bumps
count. - Newer recorder (events.k8s.io/v1) : kube-scheduler and kube-proxy. Repeats become a
series. - Flux 2.9.x writes core/v1. Controllers record Events while they reconcile (who is who).
create on events can write one (RBAC). For "who did what", use the API server audit log.3.Folded, throttled, dropped
| What | Recorder numbers |
|---|---|
| Legacy combine | from the 10th similar message within a sliding 600 s, prefix (combined from similar events): |
| Legacy spam | burst 25, then 1 per 300 s, per source, object and type |
| New series | 2nd repeat starts it; later counts written on a 30 min heartbeat or after 6 quiet min |
| Both | 1000-event queue per sink, 12 write tries |
EventRateLimit can limit Events per server, namespace, user or source and object. It is still alpha (since 1.13) and off by default.4.Retention
--event-ttldefaults to 1h. It is an etcd lease set on every create and update, so an Event vanishes an hour after its last write.- "No events" on an old Pod means expired, not healthy.
- No watch cache for Events : lists and watches hit etcd. Big clusters give Events their own etcd (
--etcd-servers-overrides=/).events#...
1.Three ways to look
kubectl events(stable 1.28) : sorted by time,--for kind/name,--types=Warning,--watch.kubectl get eventsis not sorted : add--sort-by=.lastTimestamp.kubectl describe(1.36+) shows Events for one exactly named object only. Add--show-eventsfor several.
2.Field selectors
- A closed list :
involvedObject.,kind .name,.namespace(and four more),reason,reportingComponent,source,type,metadata.,namespace metadata.name. - Operators
=,==,!=only. A comma means AND. - Anything else,
message=...included, returns "field label not supported".
3.Reasons worth knowing
| Reason | From | Type | Nearest metric |
|---|---|---|---|
FailedScheduling | scheduler | Warning | kube_ |
Pulling / Pulled | kubelet | Normal | none |
Failed (start or pull) | kubelet | Warning | ..._ |
BackOff | kubelet | Warning (restart), Normal (image pull) | ..._ (CrashLoopBackOff or ImagePullBackOff) |
Unhealthy | kubelet prober | Warning | none (readiness : kube_) |
FailedMount | kubelet | Warning | none |
Evicted | kubelet | Warning | kube_ |
SystemOOM | kubelet, on the Node | Warning | none |
OOMKilling | node-problem-detector only | Warning | ..._ |
Metrics come from kube-state-metrics. More on the reasons : scheduling, image pulls, probes, volume failures.
4.Reading a failure story
5.Traps
BackOffandFailedeach mean two things : read the message. Atype=Warningfilter misses image-pullBackOff.- Scheduler Events read through core/v1 show
count: null. Readseries.count, which can lag up to 30 min. - Messages change between releases : 1.35 dropped the container name from
Created/Started, 1.37 added preemption text toFailedScheduling. Match the reason, not the message.
| Command | What it does |
|---|---|
kubectl events -A --types=Warning | All warnings, sorted by time |
kubectl events -n shop --for deployment/web --watch | One object, streaming |
kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp | Warnings, sorted |
kubectl get events -n shop --field-selector involvedObject. | One Pod |
kubectl get events -A --field-selector reason=FailedScheduling | One reason |
kubectl describe pods -l app=web --show-events | Several Pods (1.36+) |
kubectl events for a sorted view and field selectors to narrow it. Read the reason, not the message, and remember that some reasons mean two things and one of them is Normal.1.Why export ?
The 1h TTL means no history. kube-state-metrics deliberately exports no Event metrics (high cardinality, "transition signals"). If you want Events tomorrow, ship them somewhere today.
2.Three exporters
| kubernetes- | Alloy loki.source. | OTel k8s_events / k8s_objects | |
|---|---|---|---|
| Maintenance | no release since 2024-02, forks, no clear successor | active, v1.20.1 | active, alpha / beta |
| Count updates | ignored : first occurrence only | re-sent with the new count | new and updated |
| Late events | dropped after 5 s by default | kept | kept |
| Restart | startup replay discarded | positions file | storage extension |
| One instance by | leader election | single Deployment, or clustering (1.16+) | k8s_ |
| Sinks | about 20 receivers | Loki via loki.write | any OTel exporter |
The two Collector receivers go in a deployment-mode Collector (OpenTelemetry Operator). k8s_objects was k8sobjects before v0.151.0.
3.One collector, never a DaemonSet
- Each replica of a per-node agent that watches Events ships every Event.
- Use one Deployment, leader election, or Alloy clustering.
- Clustered on all namespaces, one Alloy peer collects : failover, not scale-out.
- Alloy syntax : Loki and Alloy.
loki.source.kubernetes_events "cluster" {
job_name = "integrations/kubernetes/eventhandler"
log_format = "logfmt"
clustering { enabled = true }
forward_to = [loki.write.default.receiver]
}
4.Alerting : metric first, Event second
- Prefer the state metric (Prometheus) :
kube_.pod_ status_ unschedulable == 1 - Events for causes with no status field :
FailedMount, probe flaps,FailedSchedulingdetail, Flux errors. - Loki ruler (LogQL, Prometheus rule format) with a count threshold : Events are lossy.
- No
ruler., no alerts (Alertmanager).alertmanager_ url
# ruler rule shape only, untested
count_over_time({job=...} | logfmt
| type="Warning"
| reason="FailedMount" [10m]) > 0
| Want | Use |
|---|---|
| CrashLoopBackOff | ..._ |
| OOMKilled | ..._ |
| why a volume failed to mount | FailedMount Event |
5.Flux events
- Each Flux controller writes a Kubernetes Event and POSTs it to notification-controller.
- That drops repeats for 5 min, then sends via
Alert,Provider:notification..toolkit. fluxcd. io/ v1beta3 - Normal is
info, Warning iserror. Read :flux events --for Kustomization/<name>.
events_discarded, raise the age limit and QPS.Observability
Tracing with Tempo
1.Traces, spans, context
A span is one timed piece of work. A trace is the tree of spans that share one 16-byte trace ID.
- Span : 8-byte span ID, parent span ID (none for the root), name, start and end.
- Kind (server, client, producer, consumer, internal), status (unset, ok, error), attributes, events, links.
- Resource says who made it :
service.name,k8s.pod.name.
00 - 4bf92f3577b34da6a3ce929d0e0e4736 - 00f067aa0ba902b7 - 01
version - trace ID (16 bytes) - parent ID, the caller's span (8 bytes) - flags
Lowest flag bit = sampled. tracestate carries vendor keys. SDKs default to propagators tracecontext,baggage.
2.The pipeline
- The app makes spans : SDK, or an agent injected from an
InstrumentationCR (OTel Operator). - Each outgoing call carries
traceparent. The next service adds a child span. - Head sampling happens here, in the SDK.
Alloy syntax lives in Loki and Alloy. Platform parts emit spans too : kgateway, Knative, CoreDNS (Cluster DNS).
- The SDK batches and exports OTLP to a node agent (DaemonSet Collector or Alloy) that adds
k8sattributesmetadata. - A gateway tier can tail-sample, then exports OTLP to Tempo's distributor on 4317.
- Tempo stores the spans. Grafana queries it on 3200.
3.Ports and protocols
| Port | What | Trap |
|---|---|---|
| 4317 | OTLP gRPC | Receivers bind localhost since Tempo 2.7 : set 0.0.0.0:4317 |
| 4318 | OTLP HTTP /v1/traces | Injected Java, Node.js, Python agents (operator v0.160.0) send http/protobuf here |
| 3200 | HTTP API (queries, /ready) | Was 80 before 2.8 |
| 9095 | Internal gRPC | server. |
Pitfall : an Instrumentation CR points agents at :4317. The Java agent defaults to http/protobuf and the operator passes the endpoint, not a protocol. HTTP at a gRPC port : no crash, no failed probe, no telemetry, for months. Do better : 4318 for HTTP, and alert when spans stop.
4.Sampling
| Head | Tail | |
|---|---|---|
| Decides | At the root, from the trace ID (parentbased_) | After seeing most spans : keep errors and slow traces |
| Cost | Cheap, but blind to which traces will fail | Every span of a trace must reach one instance |
| Where | SDK or Instrumentation CR | Collector tail_sampling processor (beta) |
- Tail defaults :
decision_wait: 30s,num_traces: 50000. Policies likestatus_code,latency,probabilistic. - Scale with two tiers : a
loadbalancingexporter withrouting_key: traceIDin front. - Sampling breaks links : an exemplar or log line can point at a dropped trace.
- It shrinks span metrics : with 10% sampling, expect roughly 10% of the request rate.
traceparent, apps export spans over OTLP to a collector, and the collector samples and forwards them to Tempo. Mind the ports : 4317 is gRPC, 4318 is HTTP.1.Components
Tempo stores only traces, indexes nothing up front, and keeps Parquet blocks in object storage : cheap.
| Component | Job | Modes |
|---|---|---|
| Distributor | Receives OTLP, Jaeger, Zipkin. Applies limits, shards by trace ID | both |
| Kafka-compatible log | Durable write-ahead log | microservices |
| Block-builder | Kafka to Parquet blocks in object storage | microservices |
| Live-store | Recent traces, queryable within seconds | both |
| Query frontend | Shards a query into jobs | both |
| Querier | Runs jobs on live-stores and object storage | both |
| Backend scheduler, worker | Compaction and retention. Only one scheduler should run | both |
| Metrics-generator | Span metrics and service graphs | optional |
2.Write path and read path
- Kafka acks before the distributor answers : it is the durability point, so RF is 1 (2.x clusters used 3).
- Three consumers read Kafka. Queriers read live-stores for recent data and object storage for the rest.
- Monolithic (
-target=all, the default) needs no Kafka and runs everything in one process. The generator remote-writes to Prometheus.
3.Storage and retention
- 3.1 writes vParquet5, and will not start if told to write vParquet3.
localstorage on a PVC (Volumes) is for tests and monolithic mode.- Retention key :
backend_, default 336h (14 days).worker. compaction. block_ retention - The chart's
tempo.retention: 24hreaches only the scheduler key, which deletes nothing : a default chart 3.1.0 install keeps 14 days. Settempo.too.backendWorker. compaction. block_ retention
4.From 2.x to 3.x
| 2.x | 3.x |
|---|---|
| ingester | live-store plus block-builder |
| compactor | backend scheduler plus worker |
| RF3 in clusters | RF1, Kafka for durability |
local-blocks processor | live-store serves recent data |
| flat overrides | refused unless overrides. |
No downgrade from 3.x. TraceQL metrics history starts at the upgrade.
5.Installing on Kubernetes
grafana-community/: a monolithic StatefulSet, 3.x since chart 3.0.0, persistence off by default.tempo tempo-distributedneeds Kafka. The Tempo Operator v0.22.0 still targets 2.x.- Tempo has no auth : put a proxy in front if anything outside the cluster can reach it.
| Command | What it does |
|---|---|
helm install tempo grafana-community/ | Monolithic Tempo |
kubectl -n observability port-forward svc/tempo 3200 | Reach the API |
curl -s localhost:3200/ | Readiness (probes) |
curl -s -G localhost:3200/ | TraceQL search |
tempo-cli migrate config --mode=monolithic old.yaml > new.yaml | 2.x to 3.x config |
1.TraceQL
{ conditions }selects spans,|chains a pipeline. A trace comes back if a spanset survives.- Intrinsics use a colon :
span:duration,trace:duration,span:status.
- Attributes use a dot :
span.,http. status_ code resource.. Bareservice. name durationmeansspan:duration. - Regex is fully anchored. Structure :
>>descendant,>child,~sibling.
| Query | Finds |
|---|---|
{ resource. | Failed spans in frontend |
{ resource. | Errors anywhere below frontend |
{ trace:duration > 5s } | Slow traces |
{ span. | Server errors |
{ resource. | Request rate by status (metrics) |
{ span:name = "GET /:endpoint" } | quantile_ | p99 latency (metrics) |
({ status = error } | rate()) / ({ } | rate()) | Error ratio (3.1 arithmetic) |
2.TraceQL metrics
- Spans to series :
rate(),quantile_,over_ time() by(). - GA in 3.0. Alerting on them is experimental. 3.1 adds arithmetic.
3.Metrics from traces
The metrics-generator turns spans into series and remote-writes them to Prometheus (needs --web.). Its processors do nothing until listed in overrides.. Spans that ended over 30s ago are not counted.
| Series | Labels | Answers |
|---|---|---|
traces_ | service, span_name, span_kind, status_code | Request and error rate |
traces_ | same | Latency histogram |
traces_ | client, server, connection_type | Calls between services |
traces_ | same | Failed calls |
Cardinality : span_name is usually the biggest driver; raw URLs or user IDs as dimensions explode it. The docs list ..._ and ..._ service-graph series that Tempo never emits : do not alert on them.
4.Correlations in Grafana
- Each jump is set on its own data source.
serviceMapdraws the service graph from Prometheus series. Trace to profiles (tracesToProfiles) is in Pyroscope. - Trace to logs : default time shift is 0, start with
-2s/2s. With Loki, the link shows only when a configured tag is on the span. - Exemplars need three switches :
send_exemplars: trueon the generator, Prometheus--enable-feature=exemplar-storage(else silently dropped), GrafanaexemplarTraceIdDestinations. - Trace IDs go in Loki structured metadata, not labels (Loki and Alloy).
A monolithic Tempo from a rendered chart (Flux) on a local volume, a two-tier OTel Collector, no tail sampling, the generator feeding a RED dashboard. Pitfall : exemplars are sent, but Prometheus runs without exemplar-storage and Grafana has no exemplarTraceIdDestinations, so metric to trace is quietly broken. Do better : set all three switches and click one exemplar through to its trace.
Observability
Continuous Profiling with Pyroscope
1.The fourth signal
- Continuous profiling samples running code all the time, in production, and stores the stacks. Later you ask : which function used the CPU or memory at 03:12 ?
- A one-off
pprofgrab only answers "now, on this one instance". - A hot Pod in
kubectl top(Metrics Server) is the cue to open its profile.
Overhead : the docs say about 2 to 5%. That is the vendor's own estimate, with no published method.
| Question | Signal |
|---|---|
| Is it slow or failing ? | metrics, Prometheus |
| What happened ? | logs, Loki |
| Which request, which hop ? | traces, Tempo |
| Which line of code ? | profiles, Pyroscope |
2.Profiles and flame graphs
- A pprof profile is gzip-compressed protobuf : a list of samples. Each is a stack of locations plus one value per sample type (CPU nanoseconds, bytes allocated).
- Pyroscope names a type
<name>:<sample_. CPU time :type>:<sample_ unit>:<period_ type>:<period_ unit> process_.cpu:cpu:nanoseconds:cpu:nanoseconds
- Width is the share of the total. Depth is the call stack. The x axis is not time.
- Self = time in the function itself. Total = with its callees.
- Alloc rising and never falling suggests a leak.
3.Profile types
| Type | Answers | Go endpoint |
|---|---|---|
| CPU | where CPU time goes | / |
| wall | where time goes, waiting included | none (Java, .NET, Node.js) |
| alloc | who allocates, and how much | / |
| inuse | what is live right now | /debug/pprof/heap |
| goroutine | how many goroutines, and where | / |
| mutex, block | lock and channel contention | off until runtime. / SetBlockProfileRate |
4.Four ways in
- Code change : pull needs pprof exposed, SDK push adds a library, eBPF and OTLP need none.
- Privileges : eBPF needs root and host PID, OTLP needs root for eBPF. Pull and SDK push need none.
- Maturity : pull, eBPF and SDK push are GA. OTLP is Alpha.
| Pull | eBPF | SDK push | OTLP | |
|---|---|---|---|---|
| Profile types | CPU, memory, goroutine, mutex, block | CPU only | richest per language | CPU |
| Custom labels | Pod labels | Pod labels | yes, dynamic | resource attributes |
service_name.1.v2 storage
Pyroscope 2.0 (2026-04-20) made v2 storage the default : no ingesters and no local disks on the write path. Durability comes from object storage.
2.Components
| Component | State | Job |
|---|---|---|
| distributor | stateless | validates, routes by tenant, then by service_name |
| segment-writer | stateless | buffers, writes a segment every 500ms, client waits until durable and indexed. A StatefulSet, no volumes |
| metastore | stateful | Raft index in 6-hour windows. Run 3 (1 failure) or 5 (2 failures) |
| compaction-worker | stateless | merges segments into blocks |
| query-frontend | stateless | plans the read from metastore metadata |
| query-backend | stateless | reads object storage directly, no cache layer |
3.Single binary vs microservices
- Single binary (
-target=all, the default) runs every role, with a one-member metastore. It may use the localfilesystembackend on a PVC. - Microservices need object storage;
filesystemis single node only.
v2 since chart 2.0.2. The bare binary still defaults to v1-v2-dual : writes go to v2, old v1 data stays readable. v1 (ingesters) is deprecated, and v1 knobs like pyroscopedb.max-block-duration do nothing on v2.
4.Defaults that bite
| Setting | Default | Do |
|---|---|---|
pyroscope. | false : profiles lost on restart | enable it |
extraArgs. | debug | set info |
retention vs querier. | 31 days kept, 7 queryable, older clipped | raise the lookback |
querier. | 1d | one query covers a day at most |
| ingestion limit | 4 MB/s, burst 2 MB, then HTTP 429 | use Alloy 1.19+ (retries 429) |
| Command | What it does |
|---|---|
helm -n observability install pyroscope grafana/pyroscope --set pyroscope. | single binary with a PVC |
kubectl -n observability port-forward svc/pyroscope 4040 | open the UI |
curl -s localhost:4040/ | readiness |
profilecli query series --query='{service_ | list series |
profilecli upload --extra-labels=service_ | upload a pprof file |
1.Annotation pull
- Annotate the Pod template.
.portor.port_nameis required : the Helm doc's named-port fallback is not in the template. - These keys are the chart's embedded Alloy. Plain Alloy reads
pyroscope..io/ service_ name service_nameis the main label (UI, distributor); a missing one isunspecified.
# checkout Deployment, pod template annotations: profiles.grafana.com/cpu.scrape: "true" profiles.grafana.com/cpu.port_name: http-metrics profiles.grafana.com/memory.scrape: "true" profiles.grafana.com/memory.port_name: http-metrics profiles.grafana.com/service_name: "checkout" # containers[].ports: name http-metrics, 8080 # the app imports net/http/pprof
2.eBPF DaemonSet
pyroscope.ebpfsamples every CPU at 19 Hz (since Alloy 1.11.0; 97 before) and sends every 15s.- Root in the host PID namespace (
privilegedandhostPID, or a capability list), so its namespace needs theprivilegedlevel (Pod Security). - One Pod per node (DaemonSet), local Pods by
spec.nodeName, tolerations for control-plane nodes.
- CPU only (off-CPU sampling exists, off by default).
- Stripped binaries show module names only; very shallow stacks mean no frame pointers.
- Never set a CPU limit on a profiler : throttling drops samples when the node is busiest.
- Alloy syntax is taught in Loki and Alloy.
3.From a slow span to its code
- Wrap the tracer provider (
otelpyroscope.in Go). Samples in the local root span getNewTracerProvider( tp) span_id,trace_id; the span getspyroscope..profile. id - CPU (Go, Ruby, .NET, Python); CPU or wall (Java). Spans under about 10ms may have no samples.
- The Tempo data source's
tracesToProfiles(datasourceUid,profileTypeId,tags) adds Profiles for this span (span start minus 60s to end plus 60s). - Map
service.name(from the OTel Operator or SDK) toservice_name. No id or tags, no link (others : Tempo).
4.OTel profiles and the gotchas
OTel Profiles : public Alpha since 2026-03-26, proto still v1development; Pyroscope calls OTLP ingest "suitable for development and testing". Relabel the eBPF profiler's process. to service_name at ingest.
| Symptom | Cause | Fix |
|---|---|---|
| no profiles from an annotated Pod | no .port or .port_name | add one |
| a new "service" every rollout | service_name built from the Pod name | use namespace and container |
| no memory profiles | eBPF is CPU only | use pull or an SDK |
| empty mutex and block | off in the Go runtime | call SetMutexProfileFraction / SetBlockProfileRate |
| nothing older than 7 days | 7-day query lookback (data kept 31) | raise max-query-lookback |
local-path volume (one node), the chart's Alloy scraping only itself, and a privileged eBPF Alloy DaemonSet profiling every Pod's CPU, all rendered charts under Flux CD v2. Pitfalls : CPU limits throttle both Alloys; an eBPF service_name built from Pod names churns per rollout; traces-to-profiles left unwired. Do better : no CPU limit on profilers, a stable service_name per workload, span profiles.
service.name to service_name so a slow span leads to its flame graph.Part 3Advanced Concepts
Parts 1 and 2 cover what to run. Part 3 looks inside: the loops behind every object, the API you can extend, the checks on every request, identity, sizing and scaling, how images, packets and volumes reach a Pod, and how clusters run from Git.
- Control plane, in the Architecture section
- Desired state, in the Deployments and Scaling section
- Rollouts, in the Rollouts and Probes section
- Every object has a loop behind it
- The API can learn new kinds
- Every write is checked before it is saved
- Every image can be traced and verified
- Policy decides who acts and where Pods run
- Autoscalers size Pods and replicas
- GPUs, volumes and clusters are objects too
- A Git repo can drive the whole cluster
Extending the API
Controllers and Reconciliation
1.What is a controller ?
A loop that keeps one kind of object true. You write the desired state in spec. The controller moves the world toward it and reports in status.
- Read the desired state (spec) from the API server.
- Observe the current state.
- Act to close the gap, then report back in status.
- Most built-in controllers act only through the API server. The Job controller never runs a container. It asks for Pods, and kubelets run them.
2.Level, not edge
- Reconcile from what the object looks like now, never from the event that woke you.
- spec.replicas goes 2 5 3 ? The system does not have to pass through 5.
- Your controller may have been down while things changed. Edge triggers and watches only make the loop faster.
for { desired := get(); current := observe(); makeChanges(desired, current) }
3.The control loop
- One pass handles one key,
namespace/name. It is idempotent : create if missing, update if different, do nothing if converged. Safe when it runs twice, runs late or reads a stale cache. - Open world : users, the HPA and the garbage collector change things too.
- A child's watch event queues its owner's key, via ownerReferences (Owners and GC).
- Outside systems (cluster autoscaler, cloud routes) : desired state from the API, status back to it.
Each pass compares whole states. A missed event does no harm.
4.Spec in, status out
# kubectl get deploy web -o yaml (trimmed) metadata: generation: 7 # +1 on a spec change resourceVersion: "184213" # opaque spec: replicas: 3 # desired, written by you status: # observed, via /status observedGeneration: 7 # == generation availableReplicas: 3 conditions: - type: Available status: "True" reason: MinimumReplicasAvailable lastTransitionTime: "2026-10-06T14:02:11Z"
5.Status rules
- generation : set by the API server. Deployments bump it on spec or annotation changes, CRDs with a status subresource on any change except metadata and status.
- observedGeneration : the generation the controller acted on. Lower means the status is stale.
- conditions : keyed by type. status True, False or Unknown, a CamelCase reason (required), lastTransitionTime = last flip. Missing means Unknown.
- Write status only through /status. PUT and POST on the main resource ignore it.
- 409 Conflict : someone wrote first. Re-read and retry (return the error to requeue).
6.Commands
| Command | Shows |
|---|---|
kubectl get deploy -o custom-columns=NAME:.metadata.name, | OBSERVED < GEN : not caught up yet |
kubectl get deploy web -o jsonpath='{range . | Each condition with its reason |
kubectl wait --for=condition=Available deploy/web --timeout=120s | Blocks until the condition is True |
kubectl rollout status deploy/web | observedGeneration first, then replicas |
kubectl get pods -w --output-watch-events | The event stream an informer gets |
kubectl events --for deployment/web | Events recorded while reconciling |
1.Inside a client-go controller
client-go gives you the informer. You write the handlers, the queue wiring and reconcile.
- Informer = Reflector + RealFIFO (DeltaFIFO before 1.33) + Indexer, one per type per process via a SharedInformerFactory.
- Reflector : LIST, then WATCH from its resourceVersion. On 410 Gone it relists (API Machinery).
- Lister : reads the cache, never the API server.
factory.Start(), thencache., only then the workers.WaitForCacheSync() - Handlers only Add keys, the owner's key for a child (
metav1.). Deletes may arrive as aGetControllerOf() DeletedFinalStateUnknown.
2.The workqueue
- Fair (FIFO) and stingy : a key added many times before a worker gets it runs once.
- Two workers never hold the same key at once.
- Re-added while processing ? It runs again after Done().
- N workers = N goroutines calling Get().
| Call | When |
|---|---|
| Done(key) | Always, in a defer |
| Forget(key) | Success : reset its backoff |
| AddRateLimited(key) | Error : retry with backoff |
| AddAfter(key, d) | Recheck after d |
Default limiter : max of per-key 5ms x 2n (cap 1000s) and a 10 qps, burst 100 token bucket.
3.kube-controller-manager
kube-controller-manager (kcm) is one binary with 49 built-in loops in v1.37, each with its own workers (Architecture). --controllers defaults to * (every on-by-default loop), and -name turns one off. Three are off by default : bootstrap-signer, selinux-warning and token-cleaner.
| --controllers name | Reads, then manages | Tuning flag (default) |
|---|---|---|
| deployment-controller | Deployment ReplicaSets | --concurrent-deployment-syncs (5) |
| replicaset-controller | ReplicaSet Pods | --concurrent-replicaset-syncs (5) |
| daemonset-controller | DaemonSet one Pod per eligible node | --concurrent-daemonset-syncs (2) |
| job-controller | Job Pods, then marks the Job finished | --concurrent-job-syncs (5) |
| endpointslice-controller | Service EndpointSlices | --concurrent-service-endpoint-syncs (5) |
| namespace-controller | Terminating ns deletes all inside | --concurrent-namespace-syncs (10) |
| garbage-collector-controller | ownerReferences dependents of gone owners | --concurrent-gc-syncs (20) |
| node-lifecycle-controller | Node Leases node health | --node-monitor-grace-period (50s) |
4.Leader election on a Lease
- Replicas race to update one
coordination.Lease. resourceVersion lets exactly one write win.k8s. io/ v1 - The leader keeps updating renewTime. Standbys wait leaseDurationSeconds after they last saw it change (own clock).
- kcm defaults :
--leader-elect=true, lease 15s, renew deadline 10s, retry 2s. - Lost the Lease ? kcm exits on purpose and the kubelet restarts it.
holderIdentity is <hostname>_<uuid>. Not fencing : keep reconcile idempotent.
Extending the API
Custom Resources (CRDs)
1.What is a CRD ?
A CustomResourceDefinition teaches kube-apiserver a new kind, with its schema, validation, defaults and versions. The API server serves and stores the objects itself.
- The CRD is cluster-scoped (
apiextensions.). Its metadata.name must bek8s. io/ v1 <plural>.<group>. - spec.scope (Namespaced or Cluster) is for the custom objects, not the CRD.
- A CRD only stores data. A controller acts on it (Operators).
Use the kind once the CRD is Established.
2.A CRD in YAML
apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: crontabs.stable.example.com # plural.group spec: group: stable.example.com scope: Namespaced # of the CronTabs names: kind: CronTab plural: crontabs # URL segment shortNames: ["ct"] categories: ["all"] # kubectl get all
# then, an object of the new kind
apiVersion: stable.example.com/v1
kind: CronTab
metadata: {name: nightly}
spec: {cronSpec: "0 2 * * *", image: "backup:1.4"}
versions:
- name: v1
served: true
storage: true # exactly one
subresources:
status: {}
scale:
specReplicasPath: .spec.replicas
statusReplicasPath: .status.replicas
additionalPrinterColumns:
- name: Image
type: string
jsonPath: .spec.image
priority: 1 # only in -o wide
schema:
openAPIV3Schema: {...} # structural
conversion:
strategy: None # the default
3.Field by field
| Field | Rule / default | Purpose |
|---|---|---|
| names.singular, names.listKind | default lowercased kind, <Kind>List | CLI alias, kind of list responses |
| names.shortNames, names.categories | optional, lowercase | kubectl get ct, kubectl get all |
| versions[].served, storage | required, exactly one storage: true | serve over REST, version written to etcd |
| versions[].schema.openAPIV3Schema | structural | validation, pruning, defaulting |
| versions[].subresources | status: {} and/or scale | /status for controllers, /scale for kubectl scale and HPA (labelSelectorPath) |
| versions[].additionalPrinterColumns | none = NAME, AGE | kubectl get columns |
| versions[].selectableFields | max 8 | --field-selector on custom fields |
| versions[].deprecated | default false | warning header on requests |
| spec.conversion.strategy | None (default) or Webhook | how objects convert between versions |
| status.storedVersions | API server adds, migration trims | every version ever stored |
1.Structural schema and pruning
- Structural (a must in v1) : a
typeon every node, no type or default inside allOf, anyOf, oneOf, not. - Pruning drops unknown fields (API warns, kubectl rejects).
x-kubernetes-preserve-unknown-fieldsopts a subtree out. - default: on request, on read and after a webhook patch.
nullable: truekeeps a null. x-kubernetes-list-type(atomic by default, set, map) andx-kubernetes-list-map-keystell server-side apply how to merge a list.
Dashed : admission. On read : storage-version defaults, then conversion.
2.CEL validation rules
image: type: string maxLength: 256 # bounds the CEL cost x-kubernetes-validations: - rule: "self == oldSelf" # transition message: "image is immutable" replicas: {type: integer, default: 1}
A rule also takes messageExpression, reason, fieldPath, optionalOldSelf.
x-kubernetes-validationsholds CEL rules (Stable 1.29).selfis the node the rule sits on.- Uses
oldSelf? It is a transition rule, run only on updates where old and new both exist. optionalOldSelf: true(Stable 1.33) runs it on create too.- Ratcheting (Stable 1.33, on) passes an update if the invalid part did not change.
3.Versions and conversion
- Exactly one version has
storage: true. Each has its own schema, subresources and columns. - None (default) only rewrites apiVersion. A renamed or split field needs a Webhook.
- status.storedVersions lists every version ever stored. Drop a version only once it has left the list.
- v1.37 : StorageVersionMigration is GA and on by default. It rewrites every object at the storage version.
Changing storage: true never rewrites old objects. Real example : Gateway API CRDs (channels, v1beta1 to v1, storedVersions).
4.Commands
| Command | Does |
|---|---|
kubectl wait --for=condition=Established crd/<name> | Block until the new endpoint is served |
kubectl api-resources --api-group=stable.example.com | Kinds, shortNames and scope the CRD added |
kubectl explain crontabs.spec --api-version=stable.example.com/v1 | Field docs from the published OpenAPI v3 schema |
kubectl apply --dry-run=server -f my-crontab.yaml | Full server check (schema, CEL, admission), nothing saved |
kubectl get ct -o wide | Printer columns, plus those with priority > 0 |
kubectl get crd <name> -o jsonpath='{.status.storedVersions}' | Versions that may still have objects in etcd |
- Deleting a CRD deletes every object of that kind.
- With
statuson, a normal update ignores .status. Use--subresource=status. - A failing conversion webhook can block reads, updates, even deletes.
- Most RBAC roles miss new kinds. Add Roles.
- *.k8s.io and *.kubernetes.io groups need the
api-approved.annotation.kubernetes. io - CEL over budget rejects the CRD. Bound it with maxItems, maxProperties, maxLength.
Extending the API
Operators
1.What is an operator ?
An operator is CRDs + a custom controller. It turns a human operator's runbook (install, upgrade, backup, failover) into a reconcile loop that runs forever.
- k8s.io : "software extensions to Kubernetes that make use of custom resources to manage applications and their components." The idea came from a CoreOS blog post.
- Every operator is a controller, not every controller is an operator. The Deployment controller is generic. CloudNativePG knows Postgres : promotion, WAL archiving, backups.
2.What it automates
- Deploy an app on demand
- Take and restore backups
- Upgrades with schema or config changes
- Publish a Service for apps that cannot use the Kubernetes API
- Simulate failure to test resilience
- Leader election for apps that have none
3.Anatomy (k8s.io SampleDB)
Deploy it like any app : CRDs plus a controller Deployment, outside the control plane.
- Users touch only the CR :
kubectl edit SampleDB/.example-database - You write .spec (a change bumps metadata.generation). The operator writes .status.conditions with observedGeneration.
- Children carry an ownerReference, so a StatefulSet change re-runs reconcile.
- Delete the SampleDB : a finalizer takes a snapshot, then the StatefulSet and volumes go.
Every write goes through kube-apiserver. Arrows to the children are shorthand.
4.Capability levels
| In these notes | Level* |
|---|---|
| CloudNativePG | 5 Auto Pilot |
| External Secrets | 4 Deep Insights |
| OTel Operator | 4 Deep Insights |
| cert-manager | 3 Full Lifecycle |
| Knative Operator | 2 Seamless Upgrades |
| Flux CD | 2 Seamless Upgrades |
* OperatorHub level, self-declared by the publisher in the CSV (metadata.). Read the docs before you trust it.
5.Write one, or not ?
| You need | Reach for | Why |
|---|---|---|
| Install / upgrade a stateless app | Helm chart + Flux HelmRelease | Levels 1-2, no Go code to maintain |
| App reads a config file | ConfigMap | k8s.io : a file a Pod reads is not a CR |
| Logic on built-in kinds | Plain controller, --resource=false | No new API needed |
| Validate or default fields | CRD schema + CEL, or admission policy | No webhook (CRDs, Admission) |
| Parent-to-children mapping | Metacontroller | One JSON webhook, it does the watching |
| Non-declarative API | A stand-alone API | k8s.io : CRDs and aggregation fit declarative only |
| It already exists | Install it | Someone else carries the pager |
| Backup, failover, ordered upgrades | Write an operator | Level 3+ is why operators exist |
1.One reconcile pass
Red dashed : a step failed, Result ignored, retry with backoff. Each pass reads the current state, never "what changed". Finalizer and ownerReference rules : Owners, Finalizers and GC.
2.The calls (controller-runtime v0.25)
func (r *SampleDBReconciler) Reconcile(ctx context.Context,
req ctrl.Request) (res ctrl.Result, err error) {
var db dbv1.SampleDB
if err = r.Get(ctx, req.NamespacedName, &db); err != nil {
return res, client.IgnoreNotFound(err) } // gone : nil
if db.DeletionTimestamp != nil { // final backup first
controllerutil.RemoveFinalizer(&db, fin)
return res, r.Update(ctx, &db) }
if controllerutil.AddFinalizer(&db, fin) {
return res, r.Update(ctx, &db) }
sts := child(&db) // *appsv1.StatefulSet, name + namespace
_, err = controllerutil.CreateOrUpdate(ctx, r.Client, sts,
func() error { // set sts.Spec, then own it
return ctrl.SetControllerReference(&db, sts, r.Scheme) })
if err != nil { return res, err }
meta.SetStatusCondition(&db.Status.Conditions, ready(&db))
res.RequeueAfter = time.Hour // periodic work
return res, r.Status().Update(ctx, &db)
}
Builder : For(&dbv1., a child change requeues. Requeue: true is deprecated, reconcile. stops retries.
3.Pick a toolkit
| Tool (language) | Latest | You write | Used here / notes |
|---|---|---|---|
| controller-runtime (Go) | v0.25.2 | Reconcile(ctx, req) (Result, error) | Flux CD, CNPG, ESO |
| Kubebuilder (Go) | v4.16.0 | Types, markers, Reconcile | CNPG, ESO, OTel |
| Operator SDK (Go, Ansible, Helm) | v1.42.3 | Go via Kubebuilder, a playbook or a chart | OTel plugins. Helm type : level 2 max |
| Java Operator SDK (Java) | v5.6.2 | A Reconciler class (fabric8 client) | Quarkus extension |
| Kopf (Python 3.11+) | 1.44.6 | @kopf.on.create, timer, daemon | Feature-complete, stable |
| kube-rs (Rust) | 4.2.0 | async fn reconcile(obj, ctx) | In kube::runtime |
| Metacontroller (any) | v4.17.2 | A JSON sync webhook | API still v1alpha1 |
| OLM v1 operator-controller | v1.12.0 | A ClusterExtension : package + channel | OperatorHub.io catalog, cluster-admin |
4.Build, test, ship (Kubebuilder)
| Command | Does |
|---|---|
| kubebuilder init --domain <domain> --repo <go-module> | Scaffold the project and a Kind e2e suite |
| kubebuilder create api --group db --version v1alpha1 --kind SampleDB | CRD Go type, controller, envtest suite |
| make manifests generate | CRD YAML + ClusterRole from markers, DeepCopy |
| make install run | CRDs in, controller runs on your laptop |
| make test | envtest : etcd + kube-apiserver from setup-envtest (KUBEBUILDER_ASSETS) |
| make test-e2e | Kind cluster + Ginkgo, deleted on pass |
| make docker-build docker-push IMG=<image> && make deploy IMG=<image> | Push image, deploy CRDs, RBAC, Deployment |
5.Gotchas & good to know
- Finalizer deadlock : remove the operator before its CRs and they hang in Terminating.
- ownerReferences cannot cross namespaces (OwnerRefInvalidNamespace events).
- envtest has no controller-manager : no GC, no Pods. Assert ownerReferences, .status.
- Keep
--leader-electon before replicas > 1. - Upgrade the operator first. It only knows operands up to its own version.
- Operator down : the app keeps serving, only automation pauses.
- OLM v1 installs, then upgrades along the channel.
API machinery
API Machinery and Server-Side Apply
1.Resources, kinds and URLs
- Resource vs kind : the URL uses the resource (
deployments), the body the kind (Deployment). - Core group :
/api/v1, apiVersionv1. Named :/, apiVersionapis/ GROUP/ VERSION apps/v1. - Identity = group + resource + namespace + name. v1beta1 and v1 are two views of one stored object, so names clash.
/api and /apis list what is served. Aggregated discovery (stable since 1.30) returns every group in 2 requests. Schemas at /openapi/v3.2.A request, step by step
- Every request : authentication (401), audit and impersonation, APF (429), authorization (403, RBAC and Service Accounts).
- Writes : mutating admission, object schema validation (422), validating admission (Admission Control), then etcd. Stale resourceVersion : 409.
- Unknown fields are dropped with a Warning.
fieldValidation=Strictmakes them a 400. - Reads skip admission and come from the watch cache. While it is cold, watches and unpaginated or filtered lists get 429 with
Retry-After.
3.resourceVersion and conflicts
- resourceVersion (RV) : a string, new on every write. Pass it back unmodified. Compare as integers only within one group + resource (required since 1.35).
- PUT sends
metadata.. Stale ? 409 Conflict. GET again, reapply your change, retry.resourceVersion - Patches skip this check unless they carry the RV or a JSON Patch
testop.
| Field | Changes on |
|---|---|
| resourceVersion | Every write, status and labels too |
| generation | Desired state (spec) changes |
| observedGeneration | Set in status by the controller. Equal to generation : status is current |
How controllers use observedGeneration : Controllers and Reconciliation.
1.List, then watch
- A LIST returns the items and the collection's RV. WATCH from there, not from an item's RV.
- etcd keeps about 5 min of history. Older RV : 410 Gone. Clear the cache, re-list.
- Bookmarks carry only a fresh RV for reconnects. No interval promise.
- Pages :
limit=Nreturnsmetadata.continue. All pages share one RV. Tokens expire in about 5 min. - Streaming list (WatchList, beta, on) : one WATCH with
sendInitialEvents=truesends the state as ADDED events, then a BOOKMARK.
2.Ways to write an object
| Method | HTTP / Content-Type | Conflicts | Use for |
|---|---|---|---|
| update | PUT, the whole object | 409 if metadata.resourceVersion is stale | Read-modify-write, you write the retry loop |
| JSON Patch | PATCH application/json-patch+json | Only via a test op or the RV | Exact path ops, compare-and-set |
| JSON Merge Patch | PATCH application/merge-patch+json | None, last write wins | Small partial edits |
| Strategic Merge Patch | PATCH application/strategic-merge-patch+json | None | Built-ins only, never CRDs. Superseded by SSA |
| Server-Side Apply | PATCH application/apply-patch+yaml ?fieldManager= | 409 on another manager's field unless force=true | Declarative config, controllers, create-or-update |
3.Server-Side Apply (SSA)
- Stable since 1.22. The body is your partial intent : it creates the object if missing, patches it otherwise.
managedFieldslists each manager, its operation (Apply or Update) and the fields it owns.- Apply over another manager's field : 409 unless
force=true. Update never conflicts, it takes ownership. Same value : shared. - A field you drop is removed only if no one else owns it. Controllers should always force.
- Lists merge by
x-kubernetes-list-type(atomic, set, map). CRD lists are atomic unless marked (Custom Resources (CRDs)).
- A renamed manager stays on as a second owner.
- Drop replicas before the HPA owns it : it resets to 1.
- Clear managedFields with
[{}], not[].
1.API Priority and Fairness
- Stable since 1.29. Each apiserver has 600 seats : 400 inflight + 200 mutating.
- Priority levels split them by
nominalConcurrencyShares, and lend and borrow. - FlowSchemas match by ascending
matchingPrecedence. First match wins. - A flow = FlowSchema + distinguisher (ByUser, ByNamespace). Shuffle sharding picks its queue.
- A LIST costs seats by object count. exec and logs bypass APF.
- Edits to suggested objects revert each minute unless annotated
apf..kubernetes. io/ autoupdate-spec: "false" - Replicas under one ServiceAccount are one flow.
2.Deprecation and version skew
| API level or component | Rule |
|---|---|
| Alpha API | Off by default. Can be removed in any release without notice |
| Beta API | New betas off by default since 1.24. Still served 9 months or 3 releases after deprecation, whichever is longer |
| GA API | Can be deprecated, never removed within v1 |
| kube-apiserver (HA) | Instances within 1 minor of each other |
| controller-manager, scheduler | Not newer than kube-apiserver, up to 1 older |
| kubelet, kube-proxy | Not newer than kube-apiserver, up to 3 older |
| kubectl | Within 1 minor of kube-apiserver, either way |
3.Commands
| Command | Shows |
|---|---|
kubectl api-resources -o wide | Discovery as a table : APIVERSION, NAMESPACED, KIND, VERBS |
kubectl get pods -w --output-watch-events | List, then watch, printing each event type |
kubectl apply --server-side --field-manager=ci-cd -f deploy.yaml | SSA as manager ci-cd (default kubectl). A conflict fails with 409 |
kubectl apply --server-side --force-conflicts -f deploy.yaml | Take the conflicting fields, become sole owner |
kubectl get deploy web -o yaml --show-managed-fields | managedFields : who owns which field, Apply or Update |
kubectl get flowschemas,prioritylevelconfigurations | APF config |
API machinery
Owners, Finalizers and Garbage Collection
1.Owners and dependents
The garbage collector (GC) deletes a dependent once all its owners are gone. The link lives on the dependent. Deployment, ReplicaSet, StatefulSet, DaemonSet, Job and CronJob write it for you :
ownerReferences: # on the Pod - apiVersion: apps/v1 kind: ReplicaSet name: nginx-deployment-6b474476c4 uid: <owner uid> controller: true # at most one blockOwnerDeletion: true # default false
Arrows point from dependent to owner. Labels pick what a controller manages, owners pick what gets cleaned up.
2.Same namespace, or cluster-scoped
- A reference has no namespace field, so the owner must sit in the dependent's namespace, or be cluster-scoped.
- Owner in another namespace : the ref counts as absent, so the GC may delete your dependent.
- Cluster-scoped dependent with a namespaced owner : never collected (v1.20+). Both raise a Warning event,
OwnerRefInvalidNamespace.
3.Cascading deletion : three modes
Deleting an owner picks the order things disappear in. kubectl --cascade maps to DeleteOptions..
| Mode, kubectl flag | Owner gets | What happens |
|---|---|---|
| Background (default) --cascade=background | no finalizer | Owner goes at once. The GC deletes the dependents afterwards. Everyday deletes. |
| Foreground --cascade=foreground | foregroundDeletion | Owner stays (deletionTimestamp set) until the GC has deleted each dependent with blockOwnerDeletion=true. For teardown that must not lose the owner first. |
| Orphan --cascade=orphan | orphan | The GC strips the owner uid from each dependent, then the owner goes. Dependents keep running, so you can swap a controller without killing its Pods. |
A raw API DELETE of a batch/v1 Job defaults to orphan. orphanDependents is deprecated. blockOwnerDeletion only matters for foreground, and the plugin that polices it (OwnerReferencesPermissionEnforcement) is off by default in v1.37.
4.Other collectors
| Who, knob | What it collects |
|---|---|
Job TTL controllerttlSecondsAfterFinished (unset = keep) | A finished Job and its Pods, N seconds after it finishes (0 = at once). Stable v1.23 |
PodGC--terminated-pod-gc-threshold (12500) | Succeeded / Failed Pods above the count |
- Images, 5 min : disk over imageGCHighThresholdPercent (85), least recently used first, down to imageGCLowThresholdPercent (80). imageMinimumGCAge 2m, imageMaximumGCAge 0 (off).
- Containers, 1 min : MinAge 0, MaxPerPodContainer 1, MaxContainers -1 (flags deprecated).
5.Commands
| Command | What it does |
|---|---|
kubectl get pod <pod> -o jsonpath='{. | Lists the owners. "controller":true marks the controller |
kubectl delete deployment nginx-deployment --cascade=foreground | Owner stays (foregroundDeletion) until its ReplicaSet and Pods are gone |
kubectl patch job pi --type=merge -p '{"spec":{"ttlSecondsAfterFinished":600}}' | Makes a finished Job eligible for TTL cleanup |
1.Finalizers : deletion gates
A finalizer is a key in metadata.. While any key is left, the object stays.
- DELETE sets
deletionTimestampand returns 200 OK (202 only for the deprecatedorphanDependents=false). Terminating : keys can only be removed, no undelete. - Custom keys need a domain,
example.(CRs only warn). Bare :com/ cleanup kubernetes,orphan,foregroundDeletion. --force --grace-period=0skips graceful termination, not finalizers.
2.In your controller
- Add your key before making anything external.
- deletionTimestamp set ? Clean up (idempotent).
- Remove only your own key, then Update.
- Failed ? Return the error and retry. The key stays.
- RBAC : plain
updateon the kind.<kind>/only gates setting blockOwnerDeletion on children.finalizers
More : Controllers and Reconciliation, Operators.
- Namespace : an aggregated APIService is down (
NamespaceDeletionDiscoveryFailure), or CRs outlived their operator (NamespaceFinalizersRemaining). - PVC : a Pod still mounts it (pvc-protection).
- CR : its operator was uninstalled before it.
3.Built-in finalizers you will meet
| Key | On (added by) | Gone when |
|---|---|---|
| kubernetes.io/pvc-protection, kubernetes.io/pv-protection | PVC, PV (StorageObjectInUseProtection) | No Pod uses the PVC, the PV is unbound |
| kubernetes.io/pv-controller (in-tree), external-provisioner.volume.kubernetes.io/finalizer (CSI) | PV with reclaim Delete (v1.33) | Backend volume deleted |
| foregroundDeletion, orphan | Any owner (API server) | GC is done with the dependents |
kubernetes (in spec.finalizers) | Namespace, on create | All content gone, Pods first |
| batch.kubernetes.io/job-tracking | Job Pods (v1.26) | Job controller counted the Pod |
4.Stuck in Terminating ?
A finalizer is a promise that someone will clean up. Find the someone before you delete the promise.
5.Commands
| Command | What it does |
|---|---|
kubectl get pvc -n shop -o custom-columns=NAME:. | What is Terminating, and which key holds it |
kubectl get ns shop -o jsonpath='{range . | Why a namespace is stuck. On a DiscoveryFailure, fix the APIService that kubectl get apiservices shows as False |
kubectl describe pvc data-postgres-0 -n shop | Used By lists the Pods that keep pvc-protection in place |
kubectl patch widgets. | Last resort, drops ALL keys. Only if the controller is gone for good and you did its cleanup |
API machinery
Admission Control
1.What is admission control ?
The API server's last gate before etcd. Admission plugins can change a write, then reject it.
- Runs in kube-apiserver after authn and authz. Writes only : create, update, delete, connect.
- Mutating plugins may change the object. Validating plugins only say yes or no.
- One rejection, in either phase, fails the request.
- Only new writes are checked. Existing objects are left alone.
Any reject in either phase ends the request with an error.
2.Inside admission, in order (v1.37)
- Built-in mutators first : ServiceAccount, DefaultStorageClass, LimitRanger ...
- Then each bound MutatingAdmissionPolicy (CEL).
- Then mutating webhooks, serial, any order.
- Object changed ? Built-ins run again, and
reinvocationPolicy: IfNeededhooks may too. - Schema check, built-in validators (PodSecurity ...), ValidatingAdmissionPolicies, validating webhooks (parallel), ResourceQuota last. One deny is enough.
3.Built-in plugins
Compiled into kube-apiserver, 25 on by default in v1.37. Change the set with --enable-admission-plugins / --disable-admission-plugins (order does not matter).
| Plugin | Type | What it does |
|---|---|---|
| NamespaceLifecycle | M | Rejects objects in missing or terminating namespaces. default, kube-system, kube-public cannot be deleted |
| LimitRanger | M + V | Applies LimitRange defaults, enforces the namespace min / max |
| ResourceQuota | V | Rejects a write that would push the namespace past its quota |
| ServiceAccount | M + V | Defaults the SA to default, rejects a missing SA, adds the API token volume |
| PodSecurity | V | Pod Security Standards from namespace labels (stable v1.25), see Pod Security |
| DefaultStorageClass | M | New PVC with no class gets the default one. Create only, newest default wins |
M = mutating, V = validating.
4.Picking a mechanism
| Mechanism | Runs in, logic in | Good for |
|---|---|---|
| ValidatingAdmissionPolicy | kube-apiserver, CEL (GA v1.30) | Deny / warn / audit rules, no server to run |
| MutatingAdmissionPolicy | kube-apiserver, CEL (GA v1.36) | Simple defaults : labels, sidecars, PSA labels |
| Validating webhook | your HTTPS server, any language | Complex rules, external data. Called in parallel |
| Mutating webhook | your HTTPS server, any language | Complex mutation, external APIs. Called serially |
| Kyverno (v1.19) | webhooks, YAML + CEL | Policy library, generate / cleanup, image checks |
| OPA Gatekeeper | webhooks, Rego | ConstraintTemplate + Constraint, audits old objects |
1.Webhooks : how a call works
- Register in a Mutating or ValidatingWebhookConfiguration (v1).
- The API server POSTs an AdmissionReview. The reply echoes
request.uidand setsallowed. - Changes return as a base64 JSONPatch.
warnings[]reach the client as Warning headers (code 299).
- clientConfig : a Service (port 443, path / by default) or an https URL.
- caBundle : CA for the serving cert, valid for
<svc>.<ns>.. Filled by cert-manager's cainjector, see cert-manager.svc
2.Webhook settings
| Field | Values | Default | Notes |
|---|---|---|---|
| failurePolicy | Fail, Ignore | Fail | Errors and timeouts only. allowed: false always denies |
| timeoutSeconds | 1 to 30 | 10 | On timeout, failurePolicy decides |
| sideEffects | None, NoneOnDryRun | required | Legacy v1beta1 Some / Unknown fail dry-run |
| matchPolicy | Exact, Equivalent | Equivalent | Also catches other group / versions of the resource |
| namespaceSelector | label selector | all | Leave out kube-system and the webhook's own namespace |
| objectSelector | label selector | all | Opt-in hooks only : whoever writes the labels can skip it |
| matchConditions | up to 64 CEL | none | Called only if all are true (GA v1.30) |
| reinvocationPolicy | Never, IfNeeded | Never | Mutating only. IfNeeded needs an idempotent hook |
Matching : rules (operations, apiGroups, apiVersions, resources, scope), then the two selectors and matchConditions. Since v1.37 (beta, on by default), TokenReview, SubjectAccessReview and other virtual auth resources skip webhooks.
3.ValidatingAdmissionPolicy : CEL, no server
# demo : at most 5 replicas per Deployment apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: max-replicas.example.com spec: matchConstraints: resourceRules: - apiGroups: ["apps"] apiVersions: ["v1"] operations: ["CREATE", "UPDATE"] resources: ["deployments"] validations: - expression: "object.spec.replicas <= 5" --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: max-replicas-demo.example.com spec: policyName: max-replicas.example.com validationActions: [Deny] # [Warn, Audit] to trial matchResources: namespaceSelector: matchLabels: kubernetes.io/metadata.name: demo
- GA since v1.30. Does nothing until a binding names it in
policyName. failurePolicy defaults to Fail. validationActions: Deny, Warn, Audit. Never Deny and Warn together.paramKind+paramRefpass a ConfigMap or any CR to CEL asparams.parameterNotFoundAction(Allow / Deny) is required.- CEL sees
object,oldObject(null on CREATE),request,namespaceObject,variables,authorizer. Type errors :status..typeChecking
reinvocationPolicy is required. Binds the same way.Manifest-based admission (beta v1.37) : staticManifestsDir loads webhooks and policies from disk, enforced from API server start, even with etcd down. Names end in .static.k8s.io.
4.When a webhook goes down
- namespaceSelector : skip kube-system, own ns.
- 2+ replicas behind the Service.
- Mutating hooks fail open, a validator checks.
- Small timeoutSeconds : mutating hooks are serial.
- Break glass :
kubectl deletethe Validating / MutatingWebhookConfiguration
Security
RBAC and Service Accounts
1.Two questions per request
Every API call is checked twice. Authentication asks who you are. Authorization asks if that identity may do this verb on this resource.
- Authentication returns only strings : username, UID, groups, extra.
- Bad credential : 401. Nobody allows : 403, the default deny.
- Authorizers run in order, e.g. Node, RBAC, Webhook. The first allow or deny wins.
- RBAC allows or has no opinion. It never denies.
- Admission runs only after an allow.
Members of system:masters skip RBAC and webhook authorizers.
2.Users come from outside
There is no User object. You cannot add a normal user with an API call.
- X.509 certCN = username, each O = a group
- OIDC / JWTtoken from an identity provider
- Webhookan outside service checks the token
- Auth proxya proxy in front sets the user
- Nothingsystem:anonymous, group system:unauthenticated
Every authenticated request also gets the group system:authenticated.
3.Inside a rule
- apiGroups
""= the core group - resourcespods, secrets, pods/log (get), pods/exec (create)
- verbsget, list, watch, create, update, patch, delete, deletecollection
- specialescalate, bind, approve, sign, attest, impersonate (beta : impersonate-on:MODE:VERB)
- resourceNamesonly these objects
- nonResourceURLs/healthz, ClusterRoleBinding only
4.Four RBAC kinds
| Kind | Scope | Holds | Effect |
|---|---|---|---|
| Role | Namespaced | rules[] | Permissions usable only inside its namespace |
| ClusterRole | Cluster | rules[] or aggregationRule | Reusable. The only way to cover nodes, nonResourceURLs or all namespaces |
| RoleBinding | Namespaced | subjects[] + roleRef (Role or ClusterRole) | Grants the role only in the binding's namespace |
| ClusterRoleBinding | Cluster | subjects[] + roleRef (ClusterRole only) | Grants the role in every namespace and cluster-wide |
All in rbac.authorization.k8s.io/v1. No deny rules, permissions only add up. Turn it on with --authorization-mode=...,RBAC or an --authorization-config file (stable since v1.32).
5.Subjects, bindings, roles
- A RoleBinding can reuse one ClusterRole in many namespaces, granting only in its own.
- roleRef is immutable. Delete and recreate, or
kubectl auth reconcile. - A ServiceAccount subject uses apiGroup
"". A ClusterRoleBinding must set its namespace. - Users are just strings. Recreate a deleted name and it inherits its old bindings.
1.ServiceAccounts
The only identity stored as an API object. Namespaced, core v1.
- Every namespace gets a
defaultSA. It may only do discovery and read ClusterTrustBundles. - A Pod picks one with
serviceAccountName(else default). A missing SA gets the Pod rejected. automountServiceAccountToken: falseon the SA or the Pod skips the token. The Pod wins. Use it when the app never calls the API.
- usersystem:serviceaccount:shop:log-reader
- groupssystem:serviceaccounts and system:serviceaccounts:shop
kind: Pod
spec:
serviceAccountName: log-reader
automountServiceAccountToken: false # no token
Identity outside the cluster, with mTLS : see SPIFFE, SPIRE and Pod Certificates.
2.Bound tokens in a Pod
- Mount : /var/run/secrets/kubernetes.io/serviceaccount holds token, ca.crt and namespace.
- Lifetime : asks for 3607s, and the API server may extend it to a year. Renewed past 80% of TTL or 24h.
- Revoke : not per token. Delete the Pod or the SA; it still works up to 10 s (cache).
- Legacy Secret tokens : not auto-made since v1.24. A hand-made one never expires.
3.Default roles
- cluster-admin : everything. Bound to system:masters and kubeadm:cluster-admins.
- admin : most namespaced objects, Roles too. Cannot write ResourceQuota or the Namespace.
- edit : read / write most objects and Secrets, run Pods as any SA. Cannot see Roles.
- view : read-only. No Secrets, no Roles.
admin, edit and view have no default binding. They aggregate any ClusterRole labelled rbac. (or -admin, -view), so CRDs can extend them. view flows into edit, edit into admin.
4.Risky permissions
| Permission | Why |
|---|---|
| get / list / watch secrets | All three return the data, legacy SA tokens too |
| create pods | Mount any Secret, run as any SA in the namespace. See Pod Security |
| escalate, bind | Write or bind roles you do not hold. Without them the RBAC API blocks that |
| impersonate | Act as anyone, even group system:masters |
| get nodes/proxy | Exec in every Pod on the node, no audit |
| create serviceaccounts/ | Mint tokens for existing SAs |
5.Commands
| Command | Does |
|---|---|
kubectl auth whoami -o json | Username, groups and extra, as the cluster sees you |
kubectl auth can-i get pods --subresource=log -n shop | Ask the authorizer. The exit code is the answer |
kubectl auth can-i list pods -n prod --as=system:serviceaccount:dev:foo | Ask as someone else. Needs impersonation rights |
kubectl auth can-i --list -n shop | Everything you may do in a namespace |
kubectl create rolebinding log-reader -n shop --role=pod-log-reader --serviceaccount=shop:log-reader | Grant a Role (or --clusterrole) in one namespace |
kubectl create token log-reader -n shop --duration 10m | Short-lived token through TokenRequest |
Security
Pod Security
1.Standards vs admission
Pod Security Standards (PSS) define three levels. Pod Security Admission (PSA) enforces them per namespace.
- PSA is the built-in
PodSecurityvalidating admission plugin. GA since v1.25, on by default. - It runs after mutating admission, so it judges the final Pod. See Admission Control.
- Pods comply through
securityContextand, since v1.35,hostUsers: false. - PodSecurityPolicy : removed in v1.25. Use PSA, a third-party admission plugin, or both.
A Pod that passes restricted also passes baseline.
2.One label per mode
Label the namespace pod-security.. The modes are independent.
| Mode | On a violation | Checks |
|---|---|---|
| enforce | Rejects the Pod (403) | Pods only |
| audit | Audit annotation, allowed | Workloads + Pods |
| warn | Client warning, allowed | Workloads + Pods |
Common rollout : enforce baseline, warn and audit restricted.
- -version
v1.37orlatest(default). Pinenforce-versionso an upgrade cannot start rejecting Pods. - No labelsCluster defaults from
PodSecurityConfiguration. Built in : privileged / latest. - Exemptusernames, runtimeClasses, namespaces skip all three modes. Avoid controller service accounts.
3.Deployment path vs Pod path
enforce checks Pods only : a bad Deployment is stored, its Pods are not.
4.The checks, field by field
| Baseline bans | Allowed |
|---|---|
| hostNetwork / PID / IPC | unset or false |
| privileged | unset or false |
| hostPath volumes | not used |
| ports[*].hostPort | unset or 0 |
| appArmorProfile.type | unset, RuntimeDefault, Localhost |
| seccompProfile.type | anything but Unconfined |
| Restricted adds | Allowed |
|---|---|
| allowPrivilegeEscalation | false |
| capabilities | drop ALL, add only NET_BIND_SERVICE |
| runAsNonRoot | true (pod, or every container) |
| runAsUser | non-zero or unset |
| seccompProfile.type | set : RuntimeDefault or Localhost |
Restricted volumes : configMap, csi, downwardAPI, emptyDir, ephemeral, persistentVolumeClaim, projected, secret. A rejection names the check, for example allowPrivilegeEscalation != false or runAsNonRoot != true.
1.A Pod that passes restricted
apiVersion: v1
kind: Pod
metadata: {name: app, namespace: team-a}
spec:
securityContext: # pod level
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
seccompProfile: {type: RuntimeDefault}
containers:
- name: app
image: busybox:1.28
command: ["sh", "-c", "sleep 1h"]
securityContext: # container level
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: {drop: ["ALL"]}
volumeMounts: [{name: tmp, mountPath: /tmp}]
volumes: [{name: tmp, emptyDir: {}}] # scratch
2.securityContext, two levels
spec. sets defaults for every container. containers[*]. wins when both set a field.
| Field | Level | If unset |
|---|---|---|
| runAsUser | both | image USER |
| runAsNonRoot | both | no check |
| seccompProfile | both | Unconfined * |
| appArmorProfile | both | Runtime |
| fsGroup | pod | owners untouched |
| allowPrivilegeEscalation | container | true |
| readOnlyRootFilesystem | container | false |
| capabilities | container | runtime default |
| privileged | container | false |
* RuntimeDefault if kubelet seccompDefault: true (GA v1.27, off by default). ** on AppArmor nodes. fsGroup, supplementalGroups, sysctls : pod only.
3.User namespaces : hostUsers: false
- GA in v1.36. Root in the Pod maps to an unprivileged host UID range, 65536 IDs per Pod by default.
- PSA (policy v1.35+) skips runAsNonRoot and runAsUser : root inside passes restricted.
- Not with hostNetwork, hostPID, hostIPC or volumeDevices.
- Nodes : Linux 6.3+, containerd 2.0+ or CRI-O 1.25+, runc 1.2+ or crun 1.9+.
4.Commands
| Command | Does |
|---|---|
kubectl label --overwrite ns team-a pod-security. | Enforce restricted, pinned to v1.37 |
kubectl label --dry-run=server --overwrite ns <ns> pod-security. | Running Pods that would fail |
kubectl apply --dry-run=server -n <ns> -f <file> | Runs admission, stores nothing |
kubectl debug -it <pod> --image=<nonroot-image> --target=<container> --profile=restricted | Restricted-safe debug container |
kubectl exec <pod> -- cat / | Expect 0, a high host UID, 65536 |
5.Gotchas & Good to know
- enforce never blocks a Deployment. Read the warnings and the ReplicaSet events.
- New enforce labels do not evict running Pods. Dry-run the label first.
- runAsNonRoot with an image USER of root, or a name : CreateContainerConfigError. Set a numeric runAsUser.
- An explicit appArmorProfile RuntimeDefault fails on nodes without AppArmor.
- PSA checks debug containers too : use
--profile=restricted. - hostPID sees node processes, hostNetwork Pods may skip NetworkPolicy, privileged or hostPath reach the node's Secrets.
Identity and trust
OIDC Authentication for the API Server
1.No users, only strings
- Kubernetes has no user objects. Every authenticator returns four attributes : username, UID, groups, extra. They mean nothing until RBAC matches them.
- Success adds group
system:authenticated. Anonymous :system:anonymous, groupsystem:unauthenticated.
- First success wins; the API server does not guarantee the order. Docs : ServiceAccount tokens plus at least one method for people.
- See RBAC, API Machinery. Workloads : SPIFFE.
2.Ways in, and when each one ends
| Method | Set with | Who uses it | Ends when |
|---|---|---|---|
| X.509 client cert | --client- | break-glass | at expiry, or rotate the CA |
| ServiceAccount token | --service- | Pods | bound ones die with their Pod or Secret |
| Bootstrap token | --enable- | node join | delete the token Secret |
| Static token file | --token- | not recommended | edit the file, restart |
| OIDC / JWT | --oidc-* or --authentication- | people | at exp. No revocation |
| Webhook | --authentication- | a service | after the cache TTL (2m) |
| Authenticating proxy | --requestheader- | a front proxy | at the proxy |
X.509 : CN = user, each O = a group. v1.37 has no certificate revocation. Issuing them : cert-manager, Private PKI.
3.What an ID token is
- issequals the issuer URL exactly, trailing slash too
- subthe subject : stable but opaque
- audone entry must match an audience
- expexpiry;
iat,nbfchecked too
kubelogin sends the id_token; its aud is the client id. A Keycloak access token usually shows aud: account, so it fails the audience check. See Keycloak.
Also mapped : email, preferred_username, groups. IdP-signed; the API server caches the jwks_uri keys from {issuer}/, checks offline. Issuer : https, not --service- (fatal, 1.30+).
4.One kubectl request
- kubectl runs the exec plugin named in kubeconfig.
- kubelogin returns a cached ID token, refreshes it, or opens a browser login.
- kubectl sends
Authorization: Bearer <id_token>. - The authenticator matches
iss, checks the signature with cached JWKS, thenaudandexp. - Claims become username, groups, uid and extra, with prefixes.
- RBAC allows it, or the request is authenticated but
Forbidden.
5.Why OIDC for people
- One identity per person, groups managed in the IdP, short-lived tokens.
- Not a shared admin cert : k3s's
CN=system:admin,O=system:mastersskips RBAC and cannot be revoked short of rotating the CA. Keep it offline.
No OIDC at the API server: people use a client-cert context, and SSO users reach RBAC only through a web UI that impersonates their email and groups. Pitfall : empty claims make the UI act as its own ServiceAccount, often cluster-admin. Do better : validate claims with CEL before impersonating, and keep client certs for break-glass.
1.Flags or a file, never both
- Legacy
--oidc-*flags : supported, not deprecated. One issuer, one audience. --authentication-:config AuthenticationConfigurationinapiserver., GA since 1.34, gate removed in 1.37.config. k8s.io/ v1
- Any
--oidc-*flag plus any config file, even one with onlyanonymous, stops the API server : mutually exclusive. - The removed
Structuredgate on 1.37 : unrecognized feature gate.Authentication Configuration
| Topic | Flags | File |
|---|---|---|
| Issuers, audiences | one each, audience = --oidc- | up to 64 issuers; many audiences with MatchAny |
| Username prefix | <issuer># unless the claim is email; - disables | no default : prefix required with claim |
| Claim rules, maps | --oidc-; claim names only | claim or CEL, plus uid, extra |
| Change | restart the API server | hot reload |
| Signing algs | --oidc-, default RS256 | all known algs (RS, ES, PS) |
| IdP CA | --oidc- (a path) | certificateAuthority (PEM), see Private PKI |
2.A minimal v1 file
apiVersion: apiserver.config.k8s.io/v1
kind: AuthenticationConfiguration
anonymous:
enabled: true
conditions: [{path: /livez},
{path: /readyz}, {path: /healthz}]
jwt:
- issuer:
url: https://auth.example.com/realms/demo
audiences: [kubernetes]
claimValidationRules:
- expression: 'claims.azp == "kubernetes"'
claimMappings:
username: {claim: preferred_username,
prefix: "oidc:"}
groups: {claim: groups, prefix: "oidc:"}
uid: {claim: sub}
userValidationRules:
- expression: "!user.username.startsWith('system:')"
message: no system names
apiserver.k8s.io is the wrong group : no kind "AuthenticationConfiguration" is registered. username.claim: email rejects a token with email_verified: false.
3.Claims to user
- CEL :
claimsin claim rules and mappings,userin user rules. An empty username fails. - Prefixes stop IdP
system:names and clashes. File : no defaulting,prefix: ""= none. - Flags,
--oidc-unset, claim notusername- prefix email:<issuer>#(<claim value> subdefault). Groups prefix : no default (notoidc:as the CLI docs say). - Email (1.33) :
claims.emailin the username needsclaims.in it, an extra or a claim rule :email_verified claims.?email_verified..orValue(true) == true
4.Reload
- Bad parse or validation : the old config stays.
- A valid file applies once every new issuer answers discovery within 1 minute, else failed to update authentication config.
anonymouscannot change. Write atomically.- 10s success cache : a token works up to 10s after a reload.
Check apiserver_ (kubectl get --raw /metrics).
5.On k3s
- File on every server node, mode 0600.
- In
/etc/, a list item underrancher/ k3s/ config.yaml kube-:apiserver- arg - authentication-.config=<path> - Drop
oidc-*entries and the oldStructuredgate.Authentication Configuration systemctl restart k3sonce; later edits reload.- Keep an
anonymousblock : with this flag k3s dropsanonymous-, so anonymous is enabled.auth=false
Never the k3s ServiceAccount issuer https://.
kube-apiserver-arg with its own anonymous block.1.Exec credential plugins
- kubeconfig
users[].:user. exec command,apiVersion(client.),authentication. k8s.io/ v1 args,installHint,provideClusterInfo. interactiveMode(Never,IfAvailable,Always) is required in v1.- The plugin prints an
ExecCredentialwithstatus.tokenandexpirationTimestamp. kubectl caches it until expiry or a 401.
kuberc credentialPluginPolicy: Allowlist (entries keyed command; name deprecated in 1.36) refuses other plugins.
users:
- name: oidc
user:
exec:
apiVersion: client.authentication.k8s.io/v1
command: kubectl
interactiveMode: IfAvailable
args: [oidc-login, get-token,
--oidc-issuer-url=<issuer>,
--oidc-client-id=kubernetes,
--oidc-extra-scope=groups]
2.kubelogin
kubectl oidc-login: int128/kubelogin v1.36.4 (not the AKS tool). Versions do not track Kubernetes.- Auth code + PKCE on
127.0.0.1:8000(or18000). Cache :~/.kube/(disk; keyring optional).cache/ oidc-login
- IdP client : public with PKCE, no secret on laptops. Register
http://andlocalhost:8000 http://exactly.localhost:18000 - Ask for the
groupsscope; its mapper : Keycloak.
3.RBAC for OIDC groups
- Bind
kind: Groupwith the prefixed name. RBAC rules : RBAC and Service Accounts. - Check with
kubectl auth whoami: the final username and groups, after prefixes.
$ kubectl auth whoami
ATTRIBUTE VALUE
Username oidc:alice
Groups [oidc:k8s-viewers system:authenticated]
Groups can vanish after a refresh (kubelogin may not resend extra scopes) : run whoami after one too.
4.Operations
| Situation | What happens | What to do |
|---|---|---|
| Logout | works until exp, plus the 10s cache | short tokens, end the IdP session, oidc-login clean |
| Removed from a group | takes effect at the next refresh | keep tokens short |
| IdP down | issued tokens work; logins, refreshes fail | a break-glass cert, offline |
| API server up before the IdP | tokens fail, oidc: authenticator not initialized, 10s retries | wait : certs and SA tokens still work |
- The API server process must reach the issuer (host network on k3s). An in-cluster issuer is a bootstrap loop;
discoveryURL(httpsonly, keys still fromjwks_uri) does not fix it cheaply.
5.Commands
| Command | Does |
|---|---|
kubectl auth whoami | who the API server thinks you are |
kubectl auth can-i list pods --as=oidc:alice --as-group=oidc:k8s-viewers | test RBAC for an OIDC identity (impersonation) |
kubectl oidc-login setup --oidc-issuer-url=<issuer> --oidc-client-id=<client> | log in, print the ID token claims |
kubectl oidc-login clean | delete the local token cache (disk, keyring) |
curl -s <issuer>/ | check issuer and jwks_uri |
kubectl auth whoami. Tokens cannot be revoked : keep them short, keep a break-glass cert.Identity and trust
Keycloak and Single Sign-On
1.What single sign-on is
Keycloak is an OpenID Connect and SAML identity provider (IdP). Apps send the browser to it, it checks the user (password, OTP, passkey or a brokered IdP) and returns signed tokens. One login leaves a session cookie at Keycloak that every app in the realm reuses : that is single sign-on. Keycloak 26.8.0, CNCF incubating.
Neighbours : the API server trusting these tokens is OIDC for the API server; what a groups claim grants is RBAC.
2.The model
| Object | What it is | Watch out |
|---|---|---|
| Realm | A tenant : own users, keys, issuer .../ | master is for admins. No /auth since 17 |
| Client | One app that asks for tokens | Its clientId shows up as azp in tokens |
| Public / confidential | No secret (browser, CLI) / a secret or key for /token | Public : PKCE, tight redirect URIs |
| Client scope | Bundle of mappers. Default : always. Optional : on request | New realm defaults skip old clients |
| Protocol mapper | Puts a claim in the ID token, access token or userinfo | Group Membership, Audience, ... |
| User, group, role | Identities, memberships, realm and client roles | Groups nest, roles can be composite |
3.Authorization code with PKCE
Paths under https://<host>/
| Endpoint | Path |
|---|---|
| Discovery | /.well-known/ |
| Authorization | /protocol/ |
| Token | .../ |
| Userinfo | .../ |
| Logout | .../ |
| JWKS keys | .../ |
Grants : implicit SHOULD NOT be used, password grant MUST NOT be used (Keycloak, citing RFC 9700). Client credentials only for a client acting for itself. PKCE is optional per client until you set its method to S256.
4.Tokens and lifetimes
The basic claims are on OIDC for the API server. Keycloak adds :
azp: the client the token was issued to.audon a plain user token isaccount(Audience Resolve mapper plus the defaultaccountroles). A service that checksaudneeds an Audience mapper.groupscomes from a Group Membership mapper. Full group path is ON in the admin console (/team). A mapper made by API or import withoutfull.pathacts OFF (team).
| New realm | Default |
|---|---|
| Access token | 5 min (master 60 s) |
| SSO Session Idle | 30 min |
| SSO Session Max | 10 h |
| Offline Session Idle | 30 days |
| Authorization code | 60 s |
microprofile-jwtwrites agroupsclaim from realm roles, not groups.
The ID token's exp equals the access token's. Since 26.8.0 every refresh token's exp is capped by the session idle timeout, and a per-client lifespan cannot outlive SSO Session Max.
1.Install paths
- Official operator (lockstep) : kustomize
?ref=26.8.0into akeycloaknamespace you create first, raw manifests, experimental Helm chart, OLM. It runs the server, not the database : bring PostgreSQL 14 to 18 (CNPG). - codecentric
keycloakxlags one release; defaultrelativePath: /authputs/authin the issuer. Plain manifests : you own clustering and probes.
2.The Keycloak CR and what it makes
apiVersion: k8s.keycloak.org/v2beta1 # since 26.6.0 kind: Keycloak spec: instances: 2 image: registry.example.com/keycloak-custom:26.8.0 db: {vendor: postgres, host: keycloak-pg} # + Secrets http: {httpEnabled: true} # TLS at the Gateway hostname: {hostname: auth.example.com} proxy: {headers: xforwarded} ingress: {enabled: false} # HTTPRoute instead features: {enabled: ["client-admin-api:v2"]}
The CR also makes a NetworkPolicy, a temporary <name>- Secret (replace it), a ServiceMonitor if metrics are on. Default memory: request 1700MiB, limit 2GiB. Never expose port 9000 (/health, /metrics).
3.Build time vs runtime options
| Option | Kind | Notes |
|---|---|---|
db | build | Default dev-file, not for production |
health-enabled, metrics-enabled | build | Default false |
features | build | --features replaces the stored list |
hostname, proxy-headers, http-enabled | runtime | Fine at each start |
spec.image (or startOptimized: true) starts --optimized : a build option passed at runtime with a different value exits with code 2. The stock image rebuilds at start. features.enabled must repeat the baked list.4.Realms and clients as code
KeycloakRealmImportcreates a realm once.KeycloakOIDCClient(v2alpha1, Preview) needsclient-admin-api:v2and a Secret<cr-name>-. Name =admin clientId. No mappers, scopes, PKCE, lifetimes.- Since 26.8.0 its Secret needs label
operator., elsekeycloak.org/ kind: KeycloakOIDCClient HasErrors.
5.Upgrades
update. | On an image change |
|---|---|
RecreateOnImageChange | Default. Scales to 0 first : short outage |
Auto | Rolls a newer patch, same major.minor |
Explicit | Rolls until you change spec. |
Schema migration is one way : back up first. Roll back by restoring a dump into a fresh database.
| Command | What it does |
|---|---|
| kubectl apply -k 'github.com/ | Operator, into ns keycloak |
| kubectl -n keycloak get keycloaks/ | Ready, HasErrors |
| kubectl -n keycloak get keycloakoidcclient <name> -o yaml | Client errors (only here) |
| kc.sh show-config | Baked values show (Persisted) |
1.Clients that work
- Redirect URIs match exactly, case-sensitive.
*only as the last character : then a prefix match. https://*.example.com/cbis a literal : it never matches.- Since 26.6.3
https://example.com*meanshttps://example.com/*. - A portless plain-http loopback URI (
localhost,127.0.0.1) matches any port, for CLIs (OIDC auth). - Since 26.7.3 a redirect URI carrying OIDC response parameters is refused.
aud: account is normalAdd an Audience mapper (often in its own client scope) when a proxy or gateway checks aud. Since 26.6.2 introspection checks it too.Post-logout redirect URIs are a separate list. A broken one does not break login, so it is easy to miss.
2.Ways to put SSO or a token check in front of an app
| Pattern | Code flow runs in | Good for | Weak spot |
|---|---|---|---|
| Native OIDC in the app | The app | Roles map to app roles | Configure each app |
| oauth2-proxy | The proxy, serves /oauth2/* | Mature, group checks | Extra hop per app; no gRPC ext_authz mode |
| Envoy ext_authz server | The auth server, inside ext_authz replies | One deployment for many hosts | In-memory sessions mean one replica |
| Gateway-native OAuth2 | Envoy oauth2 filter : kgateway OAuth2, Envoy Gateway SecurityPolicy | No extra service | Gateway-specific config |
| Gateway JWT validation | None : client brings a bearer token | APIs and machine clients | No browser login |
3.The ext_authz path
- A kgateway
GatewayExtensionholdsextAuth,oauth2orjwt; aTrafficPolicyattaches it to an HTTPRoute bytargetRefs.typeis deprecated, a no-op. - Across namespaces it needs a ReferenceGrant, or it silently fails to attach.
# oauth2-proxy flags --provider=keycloak-oidc --oidc-issuer-url=https://auth.example.com/realms/demo --code-challenge-method=S256 --reverse-proxy=true --cookie-refresh=4m # under the 5 min token --allowed-group=/admins # full path
The kgateway jwt gate answers 401 for a missing or bad token and 403 when its CEL rule misses. CEL rbac entries are ORed : put AND inside one expression, and use in for list claims.
4.The issuer is the public URL
- In-cluster clients that check
issmust use the public URL, notkeycloak-service..<ns>.svc - The JWKS may come from the in-cluster Service. With a 5 minute JWKS cache, a longer Keycloak outage breaks token checks.
- One Keycloak outage takes out every SSO app. Keep an escape hatch per app where one exists.
hostAliases or split DNS.aud is checked, pick the pattern by where the code flow runs, and use the public issuer URL everywhere.Identity and trust
OpenBao: Secrets as a Service
1.What is OpenBao ?
- The open-source fork of HashiCorp Vault, which is BSL 1.1 from 1.15.0. Forked from Vault's
release/branch (November 2023). 1.14.8 was the last MPL release.1.14.x - MPL-2.0, an OpenSSF Sandbox project. Latest v2.7.1.
- API compatible, not storage compatible.
X-Vault-*headers,vault:v1:ciphertext,vault.*metrics andvault.annotations stay. CLIhashicorp.com/* bao, envBAO_*.
| OpenBao 2.7.1 | Vault 2.1.2 | |
|---|---|---|
| Licence | MPL-2.0 | BSL 1.1 |
| CLI | bao | vault |
| Tokens | s. b. r. by default | hvs. hvb. hvr. |
| Cloud seals | plugins only, since 2.7 | built in |
| Namespaces | yes, since 2.3 | Enterprise only |
2.One request path
- Everything is a path :
secret/,data/ app database/,creds/ app pki/,issue/ web auth/,kubernetes/ login sys/.health - Policies are written against paths. No matching rule means no access.
- Capabilities :
create,read,update,patch,delete,list,scan(OpenBao only, 2.2, a recursive list),sudo. denyalways wins over every other capability.
path "secret/data/app/*" { capabilities = ["read"] }
3.Engines and auth methods
| Engine | What it does | See |
|---|---|---|
| KV v2 | Versioned values : secret/, versions secret/, keeps 10 | ESO |
| database | A real DB user per lease (v-openbaouse-...), dropped when it ends | CloudNativePG |
| transit | Encryption as a service, the key never leaves. aes256-gcm96, vault:v1:... | SOPS key |
| PKI | A CA that issues short-lived certificates on request | Private PKI |
| auth methods | approle, cert, jwt, oidc, kubernetes, userpass, token | OIDC, Keycloak |
| not built in | LDAP, Kerberos, RADIUS : plugins since 2.7. AWS, GCP, Azure engines : plugins, never built in | plugins |
4.Tokens, leases, TTLs
- serviceRenewable, revocable, accessor, children.
- batchLight. Not renewable, no accessor.
- periodicRenews forever within its period.
- rootFrom init, a root, or
generate-root.
- Default and max lease TTL 768h (32 days). Mounts, roles, tokens only go lower.
bao lease renew/revoke [-prefix].sys/renew,sys/revoke*gone in 2.5.4.
5.Sealed by default
- A new server is sealed : it cannot read its own data.
- Shamir : init makes 5 shares, threshold 3. Each restart needs 3 on every node.
- Auto-unseal (KMS plugin,
transit,static,kmip) : the node unseals itself. Init gives recovery keys : they authorizegenerate-root, never decrypt the root key.
GET /v1/sys/health : 200 active, 429 standby, 503 sealed, 501 uninitialized.
1.Seals in 2.7
- 2.7.0 moved
awskms,gcpckms,azurekeyvault,ocikms,alicloudkms,pkcs11into KMS plugins. - A same-named
plugin "kms"shadows the builtin : no seal migration. - Built in :
transit,static,kmip, Shamir. - Shamir to KMS is a real migration : downtime, backup, both seals up,
unseal -migrate.
plugin_directory = "/openbao/plugins"
plugin "kms" "awskms" { command = "openbao-plugin-kms-aws" }
seal "awskms" {
region = "us-east-2"
kms_key_id = "<kms-key-id>"
}
2.Storage
raft (Integrated Storage) : HA, 3 or 5 voters. postgresql : HA, the docs now suggest it to newcomers. pebbledb (2.7) : durable, not HA, chart standalone default since 0.30.0. file : removed in 2.7.0, a migrate source until 2.8.0.
3.Raft on a StatefulSet
- Each node needs a unique
node_id. Set it per Pod, or leave it unset and raft makes a UUID in its data dir. Never hardcode one id in a shared ConfigMap. - Quorum is a majority of voters : 2 of 3.
- Standbys forward writes and, since 2.5, serve reads themselves (eventually consistent).
service_labels each Podregistration "kubernetes" openbao-active. Without it : no labels, no active-only Service.- Pods, PVCs, stable names : StatefulSets.
4.The Helm chart (0.30.3)
| Setting | Default in 0.30.3 |
|---|---|
| Mode | Standalone with pebbledb. HA with server. and server. |
retry_join | Not in the default raft HCL : add it, or join by hand |
| Update strategy | OnDelete, so you choose the order |
authDelegator | On : OpenBao's ServiceAccount may call TokenReview |
| Injector | On, with HashiCorp's vault-k8s 1.7.6 image |
| TLS, liveness, audit storage | Off. Not a production default |
Optional : a CSI provider, and a snapshot CronJob (chart 0.22.0), aimed at the active Service since 0.23.5.
5.Upgrades, backups, audit
- Back up : a raft snapshot. No downgrade guarantee.
- Replace standbys one at a time (
-l openbao-active=false), the active Pod last. - Never fail over from a newer to an older version.
- Snapshots come only from the active node : a standby redirects. No
snapshot inspect, list withgunzip -c f.snap | tar tv. - Audit devices go in the config
auditstanza : the API refuses them since 2.3.2 unless an unsafe switch is set. If every audit device fails, requests fail.
| Command | What it does |
|---|---|
| helm install openbao openbao/ | 3-replica raft HA install |
| bao status | Sealed ? Active or standby ? |
| bao operator init | Init : Shamir 5 / 3 |
| bao operator raft list-peers | Voters, leader, state |
| bao operator raft snapshot save <file> | Back up, on the active node |
| bao operator unseal [-migrate] | Give one unseal share. -migrate in a seal migration |
1.Kubernetes auth
bao auth enable kubernetes bao write auth/kubernetes/config \ kubernetes_host=https://kubernetes.default.svc bao policy write app-read app-read.hcl bao write auth/kubernetes/role/app \ bound_service_account_names=app \ bound_service_account_namespaces=app \ policies=app-read ttl=1h
- OpenBao calls the TokenReview API, so its own ServiceAccount needs
system:. The chart sets it. A hand-written install must add it, or every login is "permission denied".auth-delegator - No reviewer token : the client's JWT is the bearer, and each client SA needs the binding instead.
- OpenBao re-reads its own SA token every minute, so short-lived projected tokens work (RBAC).
- A role must bind ServiceAccount names and namespaces.
- Since 2.7.1 and 2.6.4 the SA JWT is checked again on renewal : a revoked SA token can no longer renew.
app-read.hcl: a read-only policy onsecret/, plusdata/ app/* read,listonsecret/.metadata/ app/*
2.JWT auth, SPIFFE and people
- JWT auth checks a signed token against one issuer per mount (
oidc_,discovery_ url jwks_urlor fixed keys), like the API server (OIDC). No TokenReview call, no early revocation. - SPIFFE : a JWT-SVID from the SPIRE agent, posted to
auth/. Pinjwt/ login bound_subject=, or any SVID with the right audience gets the role. No SPIFFE-specific method (SPIFFE, SPIRE).spiffe://<td>/ ns/ <ns>/ sa/ <sa> - People : a second JWT/OIDC mount that trusts the IdP (Keycloak).
3.Ways to get a secret out
| Path | Secret lands as | Auth | Note |
|---|---|---|---|
| ESO | a Secret, per refreshInterval | ESO's SA | GitOps-friendly. The value sits in etcd |
| injector | sidecar files, /vault/ | the Pod's SA | vault., webhook failurePolicy: Ignore : webhook down, the Pod starts without secrets |
| CSI driver | files in a volume | the Pod's SA | provider: openbao, needs the CSI driver |
| direct login | in the app's memory | Kubernetes or JWT | best for leases and dynamic secrets |
| Flux SOPS | kustomize-controller decrypts | Kubernetes (Flux 2.9) | a transit key as SOPS master key |
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata: {name: openbao}
spec:
provider:
vault:
server: http://openbao.openbao:8200
path: secret
version: v2
auth:
kubernetes:
mountPath: kubernetes
role: external-secrets
serviceAccountRef:
name: external-secrets
namespace: external-secrets
- ESO's docs point to the
vaultprovider. A newerprovider.(camelCase) usesopenBao path, notmountPath.
- Lagging patch releases : renewal fixes land in patches. Do better : track the current minor's patches.
- Cloud KMS auto-unseal ties unsealing to cloud credentials. If a client cert supplies them, its CA now guards unsealing too (Private PKI). Do better : alert on that chain and keep a tested recovery path.
- Each auth mount fails its own way : Kubernetes auth needs the reviewer's
system:auth-delegatorbinding, or every login fails. - Snapshots must hit the active node : a standby redirects to the leader's
api_addr. Do better : the active Service, TLS end to end.
bound_subject.Identity and trust
Private PKI and Trust Distribution
1.Why a private CA ?
- For names and clients no public CA will vouch for : internal names, mTLS client certs, cloud trust anchors (AWS IAM Roles Anywhere).
- Let's Encrypt dropped client certs :
classiclost client auth on 2026-02-11, thetlsclientprofile ended on 2026-07-08. - CA/B Forum ballot SC-081 shrinks public TLS certs : 200 days (2026-03-15), 100 days (2027-03-15), 47 days (2029-03-15). Neither rule binds a private CA.
- An OIDC issuer on a private CA means the API server must trust that root too.
- A real client hits a public hostname : a public CA (Let's Encrypt).
- A workload proves who it is with a cert (mTLS, Roles Anywhere) : the private CA.
- Anything else : no cert at all, TLS ends at the gateway.
2.Hierarchy
Lifetimes are OpenBao's PKI guidance. Keep the root key outside the online signer. For contrast, one real cluster runs a 10-year root, no intermediate and 1-year leaves renewed 30 days early.
3.The cert-manager bootstrap
- A ClusterIssuer's CA Secret lives in the cluster resource namespace (default
cert-manager). - Give the root a real
subject: an empty issuer DN breaks RFC 5280 (BadConfig). - The self-signed issuer should sign only the root.
4.What the CA issuer does not do
cert-manager itself warns : CA issuers "are generally for cert-manager demos or for advanced users with experience and tooling for running a PKI".
| It does not | So you |
|---|---|
| check name constraints or path length when it issues | rely on clients that enforce RFC 5280 at the handshake |
| make CRLs or OCSP responses (it only embeds their URLs) | plan revocation elsewhere, or keep leaves short |
| rotate a CA Secret you made by hand | track its expiry yourself |
| re-issue leaves when the CA Secret changes | run cmctl renew |
| stop leaves outliving the CA | set the leaf duration yourself |
5.Offline root, online intermediate
- Make the root key and cert on an offline machine (
CA: true, up to 10 years). - Sign the intermediate there :
CA: true,pathlen:0, permitted DNS names, marked critical. - Load a Secret :
tls.crt= intermediate then root,tls.key= the intermediate key only (kubectl -n cert-manager create secret tls). - Point a
caClusterIssuer at it. Leaves getca.crt= the root. - Publish only the root to trust stores. Track the intermediate's expiry yourself.
1.Copy the root, never the key
- Copy only the public root into a dedicated ConfigMap or Secret. Not
tls.crt(it can hold intermediates), and nevertls.key. - A dedicated source lets you choose when trust changes roll out. Then fan it out to Pods, nodes and people.
2.Ways to distribute trust
| Option | Moves | Lands as | Key stays out ? |
|---|---|---|---|
| reflector | whole Secret or ConfigMap | same-named copy per namespace | no, copies tls.key |
trust-manager Bundle | chosen certs + public roots | ConfigMap per chosen namespace | yes |
| ClusterTrust | CA certs only | kubelet-written file in the Pod | yes |
kube-root-ca.crt | the cluster's own CA only | ConfigMap in every namespace | yes, not for your CA |
| node store | the root PEM | /etc/ssl/certs on each host | yes |
3.trust-manager
- v0.25.0 ships one CRD,
Bundle(trust.cert-manager.io/v1alpha1, cluster-scoped). Its target ConfigMap takes the Bundle's name. - Sources :
configMap,secret,inLine,useDefaultCAs(pin it : Debian trixie since v0.23.0). - Unset
namespaceSelector= all namespaces today, documented to change. - PKCS#12 target for Java (JKS is deprecated). Secret targets : off by default, grant read on all Secrets.
spec: # a Bundle : current + previous root
sources:
- useDefaultCAs: true
- configMap: {name: root-current, key: ca.crt}
- configMap: {name: root-previous, key: ca.crt}
target:
configMap: {key: ca-bundle.pem}
additionalFormats: {pkcs12: {key: bundle.p12}}
namespaceSelector: {matchLabels: {trust: corp}}
4.ClusterTrustBundle
- GA in v1.37 (
certificates.k8s.io/v1) : cluster-scoped, CA certs only, readable by every ServiceAccount.v1beta1: deprecated, never served by default. - Signer-unlinked : no colon, plain RBAC. Signer-linked : the name starts with the signer name (
/written as:) then a colon, e.g.example.com:mysigner:foo; writing needsatteston the signer (admissionctbattest). - Mount a projected source by
name, orsignerNamepluslabelSelector:{}matches all, unset none; the Pod will not start unlessoptional: true(docs wrong). - The kubelet keeps the file updated; trust-manager cannot write one yet. Pod Certificates signers are expected to publish here.
kind: ClusterTrustBundle # v1, no colon in name
metadata: {name: corp-root}
spec: {trustBundle: "-----BEGIN CERTIFICATE-----..."}
- clusterTrustBundle: {name: corp-root, path: ca.crt}
5.Runtimes read trust differently
| Runtime | Add a private root | Watch out |
|---|---|---|
| Debian, Ubuntu | a .crt in /usr/local/share/, then update-ca-certificates | other file extensions are silently ignored |
| Go | SSL_CERT_FILE or SSL_CERT_DIR | the pool is cached per process |
| Node.js | NODE_EXTRA_CA_CERTS | read only when the process starts |
| Java | keytool -importcert -cacerts, or a PKCS#12 truststore | restart the JVM |
Python requests : REQUESTS_CA_BUNDLE (certifi, not the OS store). k3s image pulls : registries.yaml tls.ca_file, restart k3s per node.
A root served over plain HTTP is fine only if people compare a published fingerprint; openssl x509 -noout -fingerprint prints SHA-1 without -sha256. Pitfall : reflector mirroring the CA Secret copies the root key too. Do better : publish the SHA-256 fingerprint out of band; fan out only ca.crt.
1.Rotating a root with overlap
- Same subject, different key : old leaves do not validate ("ECDSA verification failure").
- So the new root reaches every store first, and the old one stays until its last leaf expires.
- In OpenBao's words, the hard part is rotating root certificates. It can take months.
2.The rotationPolicy trap
Since cert-manager v1.18.0 a renewal makes a new private key by default (rotationPolicy: Always, locked since v1.20.0). On a CA that is a new root key : trust-manager reading that Secret swaps the root at once.
Always (default) | Never | |
|---|---|---|
| key at renewal | new key and key id, same subject | reuses tls.key; subject, key id unchanged |
| new leaves vs old root cert | fail | verify, until the old notAfter |
| trust stores | need the new root before leaves move | need the renewed cert before the old expires |
| key compromise | each renewal limits exposure | key lives as long as the hierarchy |
Never buys time, not freedom. OpenSSL and Go reject the chain once the old root cert passes notAfter. Fixes : two Bundle sources (current, previous), or Never plus the renewed cert in every store within renewBefore.3.OpenBao PKI as the issuer
- Root outside, intermediate in OpenBao (
internal: no key export;leaf_not_after_behaviorerrortruncate). Classic endpoints : onlypermitted_dns_domains(CEL role templates : full constraints). - cert-manager's
vaultissuer calls the sign path,pki_int/sign/<role>. Roles limit names (allowed_domains). serviceAccountRef: 10-minute tokens for a dedicated ServiceAccount, via a Role tocreateserviceaccounts/token.
kind: ClusterIssuer # cert-manager.io/v1
metadata: {name: bao-int}
spec:
vault:
server: https://bao.example.com:8200
path: pki_int/sign/internal
caBundleSecretRef: {name: bao-ca, key: ca.crt}
auth: {kubernetes: {role: cert-manager,
serviceAccountRef: {name: vault-issuer}}}
Same Vault API, so expected to work, but neither project documents or tests OpenBao with this issuer.
4.Issuer choices
CA issuer : no extra service, key in a Secret, no CRL. Vault issuer + OpenBao : key stays inside, CRLs, one more service. SPIRE : short-lived, SPIFFE IDs, not DNS names. PodCertificateRequest : native, you deploy the signer.
5.Commands and one real cluster
| Command | What it does |
|---|---|
| kubectl -n cert-manager get secret <ca> -o | show CA:TRUE, pathlen, dates |
| kubectl -n cert-manager get certificate <ca> -o | when it will renew, and re-key |
| cmctl renew -n <ns> <certificate> | re-issue now (leaves after a CA change) |
| openssl verify -CAfile root.crt -untrusted int.crt leaf.crt | check a chain |
- Pitfall : one online root that is also an outside trust anchor, for example a cloud IAM Roles Anywhere anchor behind SOPS decryption and OpenBao auto-unseal. When cert-manager renews it with a new key, the outside copy goes stale silently and everything behind it fails at once.
- Do better : an offline root with an online intermediate, alerts on CA expiry and on anchor drift, and a rehearsed rotation with two anchors overlapping.
- Deleting a CA Secret re-keys it at once. Guard it with RBAC and a backup.
Never needs the renewed cert in every store in time.Identity and trust
Pod Certificates
1.What are Pod Certificates ?
A Pod asks for an X.509 certificate by adding a podCertificate source to a projected volume. The kubelet makes the key, sends a PodCertificateRequest (PCR), waits for a signer, then writes key plus chain into the Pod's tmpfs volume and keeps it fresh. No Secret.
- Alpha 1.34, beta 1.35 and 1.36 (off by default). GA in v1.37 :
certificates.k8s.io/v1, gate on, not locked in 1.37.
kubernetes.io/kube-apiserver-client-pod , is documented as unimplemented.2.The flow
- No approval step. The signer sets one of
Issued,Denied,Failedvia/status, then it is immutable. - The stub is an empty PKCS#10 request : its signature proves the kubelet holds the key.
- The API server checks the answer : requested key, lifetimes, clock skew.
- Trust side : mount the signer-linked bundle with
signerNamepluslabelSelector: {}(Private PKI).
3.The volume
| Field | Notes |
|---|---|
signerName | Required. <domain>/<path> : exactly one /, a domain of at least two labels |
keyType | Required. RSA3072, RSA4096, ECDSAP256, ECDSAP384, ECDSAP521, ED25519 |
maxExpirationSeconds | Default 86400, min 3600, max 7862400. At most 86400 for kubernetes.io/ signers |
credentialBundlePath | Recommended. One PEM : PRIVATE KEY first, then the chain, leaf first |
keyPath, certificateChainPath | Separate files. Can be read mid-rotation and not match |
userAnnotations | Copied into the PCR as unverified. Untrusted input |
user | Alpha, gate AtomicWriteVolumeUserFields. Owner UID of the files |
securityContext: {fsGroup: 2000}
volumes:
- name: creds
projected:
sources:
- podCertificate: {keyType: ED25519,
signerName: coolcert.example.com/foo,
credentialBundlePath: credentialbundle.pem}
- clusterTrustBundle: {labelSelector: {},
signerName: coolcert.example.com/foo,
path: ca.pem, optional: true}
- There is no field to ask for DNS or IP SANs. The signer picks them.
- The docs example sets
defaultMode: 0644: with no file owner and nofsGroup, any UID in the container can read the key. WithfsGrouporuser, the mode is forced to0600. - Never mount with
subPath: the file never updates.
4.Who vouches
- NodeRestriction : a node asks only for a Pod bound to it, with matching Pod, ServiceAccount and node UIDs. Never for a mirror (static) Pod. No Pod phase check.
- Since 1.37 the Pod must mount a
podCertificatesource naming that signer, or the node passes an extra authorization check. - So a signer can trust
podName,serviceAccountName,nodeNameand their UIDs. NeverunverifiedUserAnnotations: any Pod author writes those.
1.The PodCertificateRequest
- ObjectNamespaced. The kubelet creates it with
generateName: req-and an ownerReference to the Pod (Garbage Collection). - SpecImmutable :
signerName,podName/podUID,serviceAccountName/serviceAccountUID,nodeName/nodeUID,maxExpirationSeconds,stubPKCS10Request,unverifiedUserAnnotations. - StatusOne update, then immutable : one condition,
certificateChain(leaf first),notBefore,notAfter,beginRefreshAt. - Deny reasons
UnsupportedKeyType,InvalidUnverifiedUserAnnotations. No short name :kubectl get pcrfails.
2.States
- Metric :
kubelet_podcertificate_states{signer_name, state}, six states. - Refresh starts at
beginRefreshAtplus 0 to 5 minutes of jitter, with a new key. - Files are swapped atomically. The app must notice (inotify or polling).
- KEP suggestion : 24 h certs refreshed at 12 h ride out a 12 h control plane outage.
3.The numbers
| What | Value |
|---|---|
| Default lifetime | 24 h |
| Minimum | 1 h |
| Maximum | 91 days |
kubernetes.io/ signers | at most 24 h |
| Refresh jitter | 0 to 5 min after beginRefreshAt |
| Refresh pass | every 1 to 2 min |
| Overdue event | 10 min past the jittered time |
| PCR deleted | 15 min after creation, any state |
| Cleaner runs | every 5 min |
Signer notBefore | within 5 min of the API server clock |
beginRefreshAt | 10 min inside notBefore and notAfter |
4.What goes wrong
- No signer answers : the Pod sits in
ContainerCreating, with a fresh key and PCR every 15 to 20 minutes. - Denied or Failed is permanent, on first issue and on refresh. Only a new Pod helps.
- Expired is quiet : the kubelet keeps serving the old files. One
CertificateExpiredevent, theexpiredstate. - One failing source blocks the whole projected volume write.
- A kubelet restart re-keys and re-requests everything.
- The app must reload the file. Kubernetes sends no signal.
5.Commands
| Command | Shows |
|---|---|
kubectl get podcertificaterequests -A | Recent PCRs. Only the last 15 to 20 minutes survive |
kubectl get podcertificaterequests -n <ns> <name> -o yaml | Condition, timestamps and chain |
kubectl explain pod.spec.volumes. | Field docs from the live server |
kubectl api-resources --api-group= | Is the API served ? |
kubectl get events --field-selector involvedObject.name=<pod> | Denied, Failed, CertificateOverdueForRefresh, CertificateExpired |
kubelet_podcertificate_states | Kubelet /metrics. Alert on it (Prometheus) |
beginRefreshAt, and a PCR lives only 15 minutes. Denied is permanent and expired is quiet, so alert on the kubelet metric and make the app reload.1.Who ships a signer (2026-10-08)
- Nobody in core. Upstream expects in-tree signers for Service DNS and SPIFFE client certs, no version given.
- cert-manager v1.21.2 : no PCR support. Its CSR signers serve CertificateSigningRequests.
- SPIRE v1.15.3 : no PCR signer.
- A toy reference signer (by the KEP author) and a community signer exist : third-party, unvetted.
A signer is a controller. What it needs (RBAC) :
| Right | Notes |
|---|---|
sign on signers | Name <domain>/<path> or <domain>/*. Checked when a status update changes the status |
update on status | On podcertificaterequests/status : write the answer |
attest on signers | Publish its ClusterTrustBundle |
2.Writing a signer
- Own a name :
example.com/<path>, neverkubernetes.io/. - Watch PCRs, filter on
spec.signerName. - Trust only API-checked fields. Deny with
InvalidUnverifiedorUserAnnotations UnsupportedKeyType. - Sign the stub's key. SANs by policy :
spiffe://<td>/ns/<ns>/sa/<sa>or<svc>.<ns>.svc. - Lifetime and
beginRefreshAtinside the rules, clock within 5 minutes. - Answer well inside 15 minutes.
- Publish the CA in a signer-linked ClusterTrustBundle, rotate with overlap (Private PKI).
Skew : a v1-only signer cannot serve kubelets older than 1.36. v1 dropped pkixPublicKey, proofOfPossession.
3.Compared
| PCR | CSR | |
|---|---|---|
| Scope | namespaced | cluster-scoped |
| Asker | the kubelet, for a Pod on its node | any user with create |
| Approve | none | an Approved condition |
| Content | empty stub, key only | the full CSR |
| Lives | 15 min after creation | 1 h approved or denied, 24 h pending |
| Use | workload identity | kubelet and user certs |
| Pod Certificates | SPIRE | |
|---|---|---|
| Delivery | projected volume, kubelet writes | Workload API socket via csi.spiffe.io |
| Proof | API server NodeRestriction | node and workload attestation |
| Formats | X.509 only | X.509 and JWT SVIDs |
| Rotation | kubelet swaps files, app reloads | pushed over the Workload API |
| Install | a signer | server, agents, CSI driver |
IDs and SVIDs : SPIFFE, SPIRE. cert-manager's csi-driver and csi-driver-spiffe also put certs in Pods. Replacing a mesh is a KEP non-goal.
4.Seen in practice
- A v1.37 cluster serves
certificates.k8s.io/v1PodCertificateRequests and ClusterTrustBundles with no flags, but nothing uses them until you deploy a signer. - Often the only ClusterTrustBundle is the one kube-controller-manager publishes for
kubernetes.io/.kube-apiserver-serving - Clusters on SPIRE lean on JWT-SVIDs (OpenBao logins, object storage STS; see OIDC). X.509-only Pod Certificates do not replace them.
- Where it fits : operators that take projected volumes but no CSI, like CloudNativePG's
projectedVolumeTemplate. A signer backed by your private CA avoids a SPIRE-to-Secret bridge.
sign, status update, attest). Pick Pod Certificates for kubelet-made X.509 without a CSI driver, SPIRE for JWTs and attestation.Supply chain security
Images, Registries and Pulls
1.A reference, part by part
- Shape :
[host[:port]/]path[:tag][@digest]. Tag and digest both given : only the digest is pulled. - First part is a host only with a
.or:, aslocalhost, or in uppercase. Elsedocker.io(+library/for one word). - The kubelet only adds
:latest. containerd maps short names to Docker Hub. CRI-O uses its search list and can refuse an ambiguous name.
| In the Pod | containerd pulls | Default policy |
|---|---|---|
| busybox | docker.io/ | Always |
| myregistry/ | docker.io/. or :, so Docker Hub) | IfNotPresent |
| registry:5000/ | registry:5000/: makes it a host) | IfNotPresent |
| nginx | docker.io/ | IfNotPresent |
| nginx:latest | docker.io/ | Always |
2.Behind one tag
- A multi-arch tag points at an index : one manifest per platform.
- The runtime picks the node's platform. None fits : "no match for platform in manifest".
- A manifest names one config and its layers, all by digest.
- Clients re-hash what they fetch and reject a mismatch ("unexpected commit digest").
- The config digest is the ImageID (
crictlIMAGE ID).
3.Things that point at an image
- OCI 1.1 : an artifact names the image digest in
subject, its kind inartifactTypeor the config type. GET /lists them. A 404 : fall back to the tagv2/<name>/ referrers/<digest> sha256-<hex>.- The kubelet and the CRI never read referrers. Checks live in admission, tools or runtime policy : Image Signing, Attestations, Image Policy.
4.imagePullPolicy
- Defaulted once, at create : Always when the parsed tag is
latest(no tag,:latest, even:latest@sha256:, not:Latest), else IfNotPresent. - Fixed on a Pod. Moving a template to
:latestlater keeps the old default. - Always : a registry round trip per container start, not a re-download. Needs the registry up.
- IfNotPresent keeps the old bytes of a moved tag.
5.What actually ran : imageID
imageIDcopies the CRIimage_ref(Architecture). On containerd : the index digest.- It can name another repo with the same bytes. Seen :
docker.io/ rancher/ mirrored-pause@sha256:f548e0e8... - Compare only the digest part, with
crane digestwithout--platform.
| Command | Shows |
|---|---|
kubectl get pod web -o jsonpath='{range . | The digest that ran, per container |
crane digest <image> / crane digest --platform linux/amd64 <image> | Index digest / one platform's manifest |
crane manifest <image> | jq -r '.mediaType, (.manifests | length)' | Index or not, and how many platforms |
oras discover <ref>@sha256:<d> | Referrers tree. Finds nothing for legacy .sig tags |
imageID to see what really ran.1.One container start, step by step
- Always skips the presence check. IfNotPresent and Never ask CRI
ImageStatus. - Never and no image :
ErrImageNeverPull. Present : pull records decide if this Pod may use it. - Else : credentials, per-Pod back-off, a
Pullingevent, the queue. - The
Pulledevent shows two times : "in 209ms (209ms including waiting)". The gap is time in the queue.
2.Queue, rate and back-off
- Default : 1 worker, queue of 10, 1 pull at a time.
maxParallelImagePulls>= 2 (stable v1.35) flips an unsetserializeImagePullsto false. An explicittruestops the kubelet.- Keep it at 31 or less before v1.37.0 (panic).
registryPullQPS5, burst 10 : a request rate. Over it :pull QPS exceeded.- Back-off per Pod UID + image, 10s doubling to 300s. A new Pod starts fresh.
3.Where pull credentials come from
- Order : Pod
imagePullSecrets, nodeconfig.json, then plugins. - The ServiceAccount's secrets are copied only at create, only if the Pod lists none.
- Only
dockerconfigjsonanddockercfgSecrets count :kubectl create secret docker-registry. - One
PullImageper matching credential, first success wins. No anonymous retry.
| Method | Scope | Rotation | Static Pods |
|---|---|---|---|
Pod imagePullSecrets | The Pod | Update the Secret | no |
ServiceAccount imagePullSecrets | New Pods of the SA, listing none | Update the Secret | no |
Node config.json | Every Pod on the node | Re-read within 5 minutes | yes |
| Exec credential provider | Pods whose image matches | Plugin's cache duration | yes |
| SA-token credential provider | The Pod's ServiceAccount | Short-lived, no Secret | only without a token |
4.Credential provider plugins
CredentialProviderConfig(kubelet.) :config. k8s. io/ v1 name= binary,matchImages,defaultCacheDuration.- Flags
--image-credential-provider-config(a directory since v1.34) and-bin-dir. - stdin in, stdout out, 1-minute timeout, cached in memory. A bad config stops the kubelet.
5.Pulling with the Pod's own identity
tokenAttributes(beta, on since v1.34) : the plugin trades a Pod-bound token for short-lived credentials.cacheTyperequired since v1.34.ServiceAccount: one SA, shared credentials.- NodeRestriction : grant the kubelet
request-serviceaccounts-token-audience(RBAC). - Required SA annotation missing : skipped, plain 401.
*.k8s.iomatchesx.k8s.io,*.iodoes not : one label per*. A path is a plain prefix :/imagesalso matches/images2.*in a path never matches : the v1.34 blog's*.myregistry.io/*matches nothing. Write*.myregistry.io.
6.Commands
| Command | Shows |
|---|---|
kubectl apply --dry-run=server -f pod.yaml -o jsonpath='{range . | Defaulted pull policy, nothing created |
kubectl get --raw / | The kubelet's effective image settings |
kubectl events --for pod/<p> | Pulling, Pulled (queue time), BackOff |
crictl images --digests / crictl inspecti <image> | Node images : DIGEST (index) vs IMAGE ID (config) |
1.The cached-image leak
- IfNotPresent or Never : any Pod can run a private image another tenant pulled there (#18787).
- Old fix :
AlwaysPullImages(admission, off by default). Skips subresources :kubectl debugescapes it. - New fix :
KubeletEnsureSecretPulledImages, beta and on since v1.35.
2.Pull records
- Files in
/, keyed by the image name as the Pod wrote it.var/ lib/ kubelet/ image_ manager/ - A Secret hash matches from any namespace. An SA matches by UID.
- Pulled anonymously, by node
config.jsonor a non-token plugin :nodePodsAccessible, open to any Pod. - No record under Never :
ErrImageNeverPull, same text as a missing image.
| Policy | By kubelet | Preloaded |
|---|---|---|
| NeverVerify | no check | no check |
| NeverVerify | checked | no check |
| NeverVerify | if not listed | if not listed |
| AlwaysVerify | checked | checked |
Red = the default. The allowlist ignores how the image arrived.
3.Turning it on safely
- Preloaded : on the node before the gate, from
crictl pull --creds USER:PASS <img>or a node image. Clean private images off first. - Spell it
NeverVerifyAllowlistedImages: the docs'AllowListedstops the kubelet. - Allowlist : no tag, only a trailing
/*, spelled like the Pods (busyboxis notdocker.).io/ library/ busybox - Never delete
image_manager/. Watch..._.mustpull_ checks_ total{result="mustAuthenticate"}
| Stop the leak | Always | Kubelet check | Image source policy |
|---|---|---|---|
| Runs in | admission | kubelet | admission |
| Registry needed on every start | yes | new credentials only | no |
kubectl debug containers | not covered | covered | per policy |
| Beaten by a credential-less mirror | yes | yes (inference) | no |
4.Mirrors and caches
- registry.k8s.io : no uptime SLA, backends move (blobs from
cdn.on 2026-10-07). Mirror what you run.registry. k8s. io - containerd
hosts.toml, no restart. Untrusted mirror :capabilities = ["pull"]. No line = full trust. - CRI-O :
pull-from-mirror = "digest-only", SIGHUP. - Digest pulls are safe via any mirror. A tag trusts its resolver.
- A cache with no auth challenge (Spegel) serves anyone, past both checks above.
5.Image volumes take the same path
volumes[].image(GA v1.36, Volumes) : same pull path, credentials, back-off and verification. Mounted read-only.- A failed volume pull holds back only the containers that mount it. The Pod stays Pending (the docs say Failed).
ImageVolumeWithDigest: alpha, first shipped in v1.36, empty on v1.37.1. Pin the reference by digest instead.
Supply chain security
Image Signing: Sigstore and Notation
1.What a signature proves
- Binds an identity to one digest, never a tag.
- Both tools resolve a tag and warn : use
@sha256:. - Sigstore : cosign, an OIDC identity, a public log.
- Notary Project : Notation, your own X.509 PKI.
- Verifying by tag checks whatever the tag points at now. See the tag-swap advisory in Crossplane.
- How a signature finds its digest (
subject, referrers) : Images, Registries and Pulls.
2.cosign with a key
cosign generate-key-pair: encrypted key file, or a KMS.- Sign and verify
IMAGE@sha256:...with--key. - Key mode still logs to public Rekor by default.
- Private : signing config without
rekorTlogUrls, verify with--insecure-ignore-tlog. - v3.0.6+ :
--keyplus identity flags is an error.
| Key URI | Holds |
|---|---|
cosign.key | file, plus cosign.pub |
awskms:///alias/x | AWS KMS, by alias |
gcpkms:// | Google Cloud KMS |
hashivault://, openbao:// | Vault transit (OpenBao v3.0.6) |
k8s://ns/name | a Secret : key, password, pub |
3.Keyless, step by step
- Fulcio : OIDC token to a 10-minute cert. Identity in the SAN, issuer in OID
1..3. 6. 1. 4. 1. 57264. 1. 8 - Signed : a DSSE in-toto Statement, predicate type
https://.sigstore. dev/ cosign/ sign/ v1
Bundle v0.3 = leaf cert + tlog entry + timestamp + DSSE envelope. The identity stays in public logs for good.
4.Reading the identity
- A reusable workflow puts its own path in the SAN. cosign pins the caller only by deprecated OID
.5(--certificate-github-workflow-repository). - The caller sits in OIDs
.12(source repo) and.18(build config) : that needs a policy engine.
| Signer | --certificate-oidc-issuer | SAN (--certificate-identity) |
|---|---|---|
| GitHub Actions | https:// | https:// |
| GitLab CI | https:// | https:// |
| Google SA | https:// | krel-trust@k8s-releng-prod. |
| Kubernetes SA | your cluster issuer | https:// (RBAC) |
| SPIFFE | your configured issuer | spiffe://... from sub (SPIFFE, SPIRE) |
5.Trust roots, Rekor v1 and v2
- Trust : TUF ships
trusted_root.json. Itstimestamp.jsonlives a week, so air-gapped verifiers pass--trusted-root. - Public default : Rekor v1, for the foreseeable future. v2 is opt-in (
signing-config create --with-default-rekor-v2). - Rekor v2 has no SET, so a TSA is mandatory. Shards rotate : never hardcode
log2025-1. - No revocation : short certs plus log monitoring. Private instance :
cosign trusted-root create.
1.Where signatures live
- cosign v3 : a bundle as an OCI 1.1 referrer, type
....sigstore. bundle. v0. 3+json - Legacy cosign (v2 default) : tag
sha256-<hex>., nosig subject, so the Referrers API never lists it. .sigtag and fallback indexsha256-<hex>differ. The index is a racy read-modify-write.--registry-referrers-mode=oci-1-1is not the v3 switch : experimental, legacy-format writes only.
| Format | Tag | subject | Listed | Written by |
|---|---|---|---|---|
| Legacy cosign | sha256-<hex>. | no | no | v2 default; v3 with 2 legacy flags |
| cosign v3 bundle | none | yes | yes | v3 default; v2.6 with a flag |
| Notation | none | yes | yes | notation |
| Fallback index | sha256-<hex> | n/a | n/a | both, without the API; notation v1.x always |
2.Verifying with cosign
- Keyless verify requires an identity and an issuer flag (or their
-regexpforms). - The regexps are unanchored : write
^...$. A.*accepted a stranger's bundle in the lab : any Sigstore user passes. - One identity per run. Always verify
IMAGE@sha256:....
$ cosign verify ghcr.io/ORG/REPO@sha256:<d> \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity-regexp '^https://github\.com/ORG/REPO/\.github/workflows/release\.yml@refs/tags/v[0-9.]+$'
3.What "verified" means
- cosign runs eight checks, in the order on the right.
- Key mode swaps cert, SCT and identity for the key, and needs no timestamp.
- The predicate type is not checked. An SBOM attestation from the pinned identity passes
cosign verify(see SBOMs, Provenance and Attestations). --insecure-ignore-tlogstill prints "verified offline". Never scrape stderr.
4.Old and new verifiers
- Old verifiers say
Error: no signatures found(exit 10) on a v3 signature. - If any bundle exists, v3 ignores legacy
.sigsignatures. - Floor : cosign v3.1.3+ or v2.6.5+. GHSA-fx35-mq7g-6g98 has no CVE ID, so CVE scanners miss it.
- CI lags :
cosign-installerv4.1.2 defaults to v3.0.6, Alpine 3.20 ships 2.2.1. Pin it and printcosign version.
| Signed with | < v2.6.0 | v2.6.0 to v2.6.2 | v2.6.3 to v2.6.5 | v3.x |
|---|---|---|---|---|
Legacy .sig | yes | yes | yes | yes, if no bundles |
| v3 bundle referrer | no | only with --new-bundle-format | yes (auto-detect) | yes |
| Both on one image | .sig only | .sig only | bundles only | bundles only |
5.Commands
| Command | What it shows |
|---|---|
cosign sign --yes IMAGE@sha256:<d> | Keyless sign, bundle pushed as a referrer |
cosign sign --key cosign.key IMAGE@sha256:<d>cosign verify --key cosign.pub IMAGE@sha256:<d> | Key mode, sign and verify |
cosign tree IMAGE | .sig and .att tags, plus referrers |
cosign bundle inspect b.sigstore.json | Cert extensions, SCTs, tlog, timestamps (no crypto checks) |
cosign version | Client version : check it in every CI image |
1.Notation : your own PKI
- Keys via plugins : azure-kv v1.2.1, AWS Signer.
notation sign: JWS (default) or COSE envelope, optional RFC 3161 countersignature.- v1.3.2 writes a referrer with
config.mediaType...(nonotary. signature artifactType) and thesha256-<hex>index.
2.trustpolicy.json, trust stores
- Stores hold root certs only, in
~/..config/ notation/ truststore/ x509/ - Policy pick : exact repo, then one
"*", else fail. - Identities need C, ST, O. No
serverAuthEKU, unlike cert-manager TLS certs. notation policy import trustpolicy.json, thennotation verify -vby digest.
{"version": "1.0", "trustPolicies": [{
"name": "app",
"registryScopes": ["example.com/app"],
"signatureVerification": {"level": "strict"},
"trustStores": ["ca:acme", "tsa:my-tsa"],
"trustedIdentities": [
"x509.subject: C=US, ST=WA, O=acme.io"]
}]}
3.Verification levels
auditpasses a wrong signer with exit 0.skipwith"*": v1.3.2 accepts it, the spec forbids it.- No TSA : a signature dies with its leaf cert.
- Unknown revocation fails
strict(OCSP 2s, CRL 5s).
E enforce (fails), L log and go on, - skipped.
| Check | strict | permis. | audit | skip |
|---|---|---|---|---|
| Integrity | E | E | E | - |
| Authenticity | E | E | L | - |
| Timestamp | E | L | L | - |
| Expiry | E | L | L | - |
| Revocation | E | L | L | - |
4.cosign or Notation ?
Notation v1.3.2 (2025-04-27) is current (v2 : alpha). Flux checks both on OCI sources. Cluster : Image Policy.
| Aspect | cosign v3.1 | Notation v1.3 |
|---|---|---|
| Root of trust | TUF trusted_root.json or --trusted-root | Root certs in trust stores |
| Identity check | SAN + issuer (OID .8), or a raw key | trustedIdentities x509 subject, or a plugin identity |
| Signing time | Rekor v1 SET, or RFC 3161 (must with Rekor v2) | Optional RFC 3161 countersignature |
| Transparency | Rekor + CT log by default | None built in |
| Revocation | None : short certs, log monitoring | OCSP first, CRL fallback |
5.Copy, then verify again
oras cp -r(preview) misses legacy.sig.cosign copy(deprecated v3.0.5) misses v3 bundles.regctl image copy --referrers --digest-tags SRC DSTcarries both. Keep the digest unchanged.- Re-verify at the destination. Notation also needs the new repository in
registryScopes.
6.Worked example : Kubernetes v1.37.1
- Images :
krel-trust@k8s-releng-prod., issuer Google, legacyiam. gserviceaccount. com .sig. - That
.sigholds krel-staging and krel-trust signatures : either passes. Pin the digest (Deployments). - Binaries : krel-staging, with flags deprecated and due to go in v4. No
.sigstore.jsonyet.
$ cosign verify registry.k8s.io/kubectl@sha256:b7cab618... --certificate-identity krel-trust@... --certificate-oidc-issuer https://accounts.google.com $ for f in kubectl kubectl.sig kubectl.cert; do curl -sSfLO "https://dl.k8s.io/release/v1.37.1/bin/linux/amd64/$f"; done $ cosign verify-blob kubectl --signature kubectl.sig --certificate kubectl.cert --certificate-identity krel-staging@... --certificate-oidc-issuer https://accounts.google.com
Supply chain security
SBOMs, Provenance and Attestations
1.Signed is not the same as true
An attestation is a signed claim about one artifact digest. The signature tells you who said it, never that it is right. Signing itself is in Image Signing; this chapter is about what gets signed.
- Signature + identity : a trusted signer and issuer, not "any signer".
- Subject : a
subject[].digestequals the digest you hold. - Type : the
predicateTypeyou expect (provenance, SBOM, VEX). - Policy on the predicate : builder, source repo, workflow, ref.
2.One claim, four layers
- Statement v1 :
_type,subject[](name + digest),predicateType,predicate. - DSSE signs the PAE string,
DSSEv1 28 application/, sovnd. in-toto+json <len> <body> payloadTypeis signed too. keyidis only a hint, never a security decision.- Then use the verified bytes only. Never re-parse.
- Referrers : Images and Pulls.
3.Predicate types
SBOMs, provenance, scans, VEX : all predicates. cosign v3.1.3 still writes Statement v0.1, actions/attest v1.
| Predicate | predicateType | cosign --type |
|---|---|---|
| SLSA provenance v1 | https:// | slsaprovenance1 |
| SLSA provenance v0.2 | https:// | slsaprovenance and slsaprovenance02 : the trap |
| SPDX | https:// | spdx, spdxjson |
| CycloneDX | https:// | cyclonedx |
| OpenVEX | https:// | openvex |
| cosign custom | https:// | custom : the default |
4.SLSA 1.2 : Build and Source tracks
1.2 adds the Source track. The Build track and the predicate https:// do not change.
| Level | Build track | Source track |
|---|---|---|
| L1 | Provenance exists. "Trivial to bypass or forge" | Version controlled |
| L2 | Hosted platform signs it : stops tampering after the build | History and provenance per revision |
| L3 | Isolated runs, signing secrets out of reach : stops it during | Technical controls on named refs |
| L4 | none (old v0.1 "SLSA 4" is gone) | Two-party review |
| Evidence | Provenance v1, signed by the platform | Source VSA |
5.Provenance v1, field by field
externalParametersare untrusted inputs : check them, reject fields you do not know.runDetails.is meant to decide the Build level on its own.builder. id resolvedDependencies: sources with digests, here the Git commit.materialsorinvocationmean v0.2 from an older generator, not a bad signature.
GitHub provenance v1, trimmed :
"buildType": ".../buildtypes/workflow/v1",
"externalParameters": {"workflow": {
"ref": "refs/tags/v1.4.0",
"path": ".github/workflows/release.yml"}},
"resolvedDependencies": [{"uri": "git+...",
"digest": {"gitCommit": "<sha>"}}],
"builder": {"id": "https://github.com/acme/
build-templates/.../oci-release.yml@..."}1.GitHub artifact attestations
actions/attest@v4: one action for provenance, SBOM and custom. The older two are wrappers.- Write on
id-token+attestations; onartifact-metadataonly for storage records,packagesfor GHCR. - Alone : Build L2. Built and attested inside a reusable workflow : L3.
- Private repos : no transparency log, GHE Cloud only.
2.Verifying provenance
gh attestation verify oci://: SLSA v1 + GitHub issuer by default.IMG@sha256:D -R org/repo - A reusable workflow signed it ? Add
--signer-workflow. - No TTY : success prints nothing. Gate on the exit code, keep
--format jsonas evidence. - slsa-github-generator, slsa-verifier : unmaintained since 2026-08-07 (tags predate the notice).
| gh | cosign | slsa-verifier | |
|---|---|---|---|
| Reads | API, OCI, disk | referrers, .att | generator, GCB |
| Default | SLSA v1 | custom | by builder |
| Identity | -R, signer flags | identity + issuer | --source-uri |
| Kept up | yes | yes | no |
3.BuildKit attestations
- Push with
--provenance=mode=max --sbom=true: oneunknown/unknownmanifest per platform. - SLSA v1 by default since v0.28.0 (Docker's docs say v0.2).
- Layers are bare, unsigned Statements : sign the predicate yourself.
- Read :
docker buildx imagetools inspect IMG --format '{{ json .Provenance.SLSA }}' mode=maxrecords build-arg values.--loaddrops attestations.
4.What an SBOM lists
- SPDX 2.3 is still the tools' default. SPDX 3.0.1 is JSON-LD and not field-compatible with it.
- syft's SPDX and Kubernetes' SBOM miss the CISA 2026 minimum elements. The signature goes outside, in an attestation.
| SPDX 2.3 | SPDX 3.0.1 | CycloneDX 1.7 | |
|---|---|---|---|
| Shape | JSON or tag-value document | JSON-LD graph | JSON, XML, Protobuf |
| Default in | syft, trivy, BuildKit, bom | syft opt-in (spdx-json@3.0) | syft and trivy CycloneDX output |
actions/attest | accepts JSON, not tag-value | rejects | accepts, needs serialNumber |
| grype / trivy | both; cross-tool can read 0 | grype yes, trivy fails | both; cross-tool loses findings |
5.Making and attaching SBOMs
syft scan registry:IMG@sha256:D -o spdx-json=s.json -o cyclonedx-json=c.: two formats, no daemon.json trivy image --format cyclonedxis an SBOM, not a scan, without--scanners vuln.- Signed :
cosign attest --type cyclonedx --predicate c.json IMG@sha256:D. Verify withcosign verify-attestation --type cyclonedx; without--typeit looks forcustom. cosign attest --statementis ignored for images. Onlyattest-blob --statementtakes a full Statement.- Unsigned :
oras attach --artifact-type application/. Fine to share, not evidence.vnd. cyclonedx+json IMG c.json
1.Rescan the SBOM, not the image
grype sbom:./x.jsonortrivy sbom x.json: stored SBOM vs today's DB. No pull, no rebuild.- Cross-tool rescans fail silently : Trivy found 0 in syft's SPDX, grype 0 in Trivy's SBOMs, against 48 to 52 direct. All exit 0 (2026-10-07).
- Check for a detected OS and a sane package count.
grype --fail-onexits 2, not 1 (1 = the scan broke).
2.VEX : saying "not affected"
- OpenVEX v0.2.0 :
vulnerability,products(digest PURL best),status.not_affectedneeds ajustificationorimpact_statement. - VEX removes findings, so authenticate it. Trivy's experimental
--vex ocilet a VEX signed by a throwaway key drop a CVE (48 to 45). - Same with
--sbom-sources oci: an unverified attached SBOM gave 9 findings, not 48. Who can push a referrer picks what is scanned (RBAC).
| --vex | Trivy | Grype |
|---|---|---|
| Formats | OpenVEX, CycloneDX, CSAF | OpenVEX, CSAF |
| Suppresses | not_affected, fixed | same, into ignoredMatches |
| Sig check | none on found VEX | local files only |
| Two files | --vex a --vex b or a,b | repeat --vex only |
$ vexctl create --product=pkg:oci/app@sha256:... \
--vuln=CVE-... --status=not_affected \
--justification=... --file=app.vex.json
3.Continuous scanning in the cluster
- Trivy Operator v0.35.0 runs scan Jobs and writes a VulnerabilityReport and SbomReport per container, owned by the workload (GC).
scannerReportTTL24h : reports are recreated daily against a fresh DB.- Standalone (default) : each Job downloads the DB. ClientServer : one shared server.
- Its scanner image is pinned by tag (
trivy:0.75.0) : mirror a verified digest.
$ kubectl get vulns -A -o wide $ kubectl get clustervuln,clustersbom
4.Worked example : Kubernetes itself
sbom.: SPDX 2.3 tag-value, 812 packages (among them 24 images, 508 Debian packages) plus 100 files. Go modules are ink8s. io/ v1. 37. 1/ release /source.- Signed like the binaries (krel-staging,
.sig+.cert) : verify as in Image Signing. provenance.json: Statement v0.1, SLSA v0.2, unsigned, 404 for every 2026-09-23 patch. Only level claim : "Level 1" (v1.23).- Rescan 2026-10-07 : grype 1306 matches (155 CVEs, all Debian); Trivy 0 (
os family none).
$ U=https://dl.k8s.io/release/v1.37.1 $ curl -sI $U/provenance.json # 404 $ curl -Ls $U/kubernetes-release.spdx -o k.spdx $ grype sbom:./k.spdx
5.Case study : the Trivy compromise (March 2026)
- Stolen credentials published trivy v0.69.4 through the real release pipeline, and force-pushed 76 of 77
trivy-actiontags and all 7setup-trivytags (CVE-2026-33634, in CISA KEV). - Pin actions by full SHA, transitively (trivy-action <= 0.30.0 called setup-trivy by tag). Pin the tool too : setup-trivy defaults to
latest. - Verify the binary with a tight identity. The signer moved from
private-trivytotrivyat v0.69.3.
Digest pins and full-SHA pins held, but only where every layer pinned.
Supply chain security
Enforcing Image Policy
1.Where a check can run
A policy covers only the requests and fields it sees. Webhooks and VAP syntax : Admission Control.
- CI checks manifests but only advises : anyone can
kubectl applyaround it. - Admission runs once per API write, before any pull.
- The runtime checks every real pull, never cached images.
2.The gap between admit and pull
- Admission verifies
app:1.0at digest A. The tag moves to B. A later pull gets B, never verified. - Fix : pin digests in Git, or let the verifier append one. All three keep the tag :
podinfo:6.9. 2@sha256:cab90e...
3.Every field that holds an image
A Pod-only rule admits the Deployment, then denies its Pods as ReplicaSet FailedCreate, as in Pod Security. kubectl debug --copy-to makes a new Pod, so pods rules see it.
| Field | Reached by | Trap |
|---|---|---|
| containers, initContainers | Pod CREATE | Examples often check containers only |
| ephemeralContainers | kubectl debug : UPDATE | pods, * miss it : list pods/ |
| volumes[].image.reference | Pod CREATE | Skipped by every admission verifier by default |
| spec.template.spec | Deployment, StatefulSet, DaemonSet, Job | Unchecked unless listed |
| spec.jobTemplate.spec.template.spec | CronJob | One more path for your CEL |
4.What built-in CEL can do
variables: # Pod form - name: images expression: >- object.spec.?containers.orValue([]).map(c, c.image) + object.spec.?initContainers.orValue([]).map(c, c.image) + object.spec.?ephemeralContainers.orValue([]).map(c, c.image) + object.spec.?volumes.orValue([]).filter(v, has(v.image)) .map(v, v.image.?reference.orValue('')) validations: - expression: variables.images.all(i, i.matches('@sha256:[a-f0-9]{64}$')) reason: Forbidden # 403, not 422
- Can :
startsWith,matches,all/exists. - Cannot : parse a reference, normalize
nginx, call a registry, resolve a tag. - End prefixes in
/: blocksregistry..example. com. evil. io - Forbid
:latest, even next to a digest.
5.Built-ins outside a webhook
- ImagePolicyWebhook (off by default) : Pods and debug containers only.
defaultAllow: falsefails closed. - Static image VAP (manifest-based, beta v1.37) : no
paramKind, so a hard-coded allow-list. Not deletable through the API. A broken file keeps the old one. - AlwaysPullImages : see Images, Registries and Pulls.
6.Commands
| Command | Shows |
|---|---|
kubectl debug -it <pod> --image=<ref>@sha256:<d> --target=<c> | A debug container : a separate request on the subresource |
kubectl describe rs <rs> | Late FailedCreate from a Pod-only rule |
kubectl get validatingadmissionpolicy <name> -o jsonpath='{. | CEL type errors |
kubectl get pod <p> -o jsonpath='{. | Whether digest pinning happened |
1.Kyverno ImageValidatingPolicy
policies., GA since 1.17 (v1 storage planned for 1.20). The Namespaced kind acts in its own namespace.
| spec field | What it does |
|---|---|
| matchConstraints | Webhook rules, copied verbatim : list pods/ yourself |
| matchImageReferences | A scope on the raw string : unmatched images are skipped, not denied |
| images | Extra extractors (CEL). None built in for image volumes |
| attestors, attestations | cosign (key, keyless, trustedRoot) or notary. Claims by intoto.type or referrer.type |
| validations | CEL : verifyImageSignatures, verifyAttestationSignatures (count > 0) |
| validationConfigurations | mutateDigest, verifyDigest, required : all default true |
2.One Pod through Kyverno
- Autogen covers controllers only when
resourcesis exactly["pods"]. Addingpods/turns it off : use two policies.ephemeralcontainers
3.An IVPOL that pins a release workflow
matchImageReferences: [{glob: "ghcr.io/kyverno/*"}]
attestors:
- name: github
cosign:
keyless:
identities:
- issuer: https://token.actions.githubusercontent.com
subjectRegExp: '^https://github\.com/kyverno/kyverno/\.github/workflows/.+@refs/tags/v.+$'
source: {repository: ghcr.io/kyverno/signatures}
validations:
- expression: (images.containers + images.initContainers)
.all(i, verifyImageSignatures(i, [attestors.github]) > 0)
- Anchor
subjectRegExp. Kyverno keeps its signatures in another repo. - cosign v3 bundles are auto-detected. cosign attestors check only in-toto.
trustedRootsits next tokeyless, not inside it (docs nest it).- IVPOL never reads the Pod's
imagePullSecrets.spec.credentials(Kyverno namespace Secrets) replaces the helpers, no merge.
4.Kyverno version traps
- 1.19.0 made all three real : upgrades can deny.
- 1.19.1 fixes GHSA-5cjf-wwfg-pj4c : a one-image exception exempted the whole resource.
verifyImages(ClusterPolicy) : deprecated 1.19, removed in 1.20 (late 2026).- 1.19 is tested only on Kubernetes 1.33 to 1.35.
5.Sigstore policy-controller
- Opt-in namespaces only.
no-match-policydefaults to deny, so does a typo. - CIP : every matching one must pass, one authority in each is enough.
ctlog= Rekor. - v0.15.1 cannot verify cosign v3 bundle image signatures :
bundleis attestations only. - Chart 0.10.8 runs v0.13.1. No cache.
6.Commands
| Command | Shows |
|---|---|
kubectl get ivpol <name> -o jsonpath='{. | Which controllers got variants |
kubectl get ivpol <name> -o jsonpath='{. | Stored defaults (all three true) |
kubectl get policyreports -n <ns> | Audit results |
kubectl label namespace payments policy. | Opts a namespace in |
kubectl patch configmap/ | Warn on unmatched images |
1.Ratify behind Gatekeeper
- External data : a Gatekeeper
Provider, called from Rego withexternal_data. Responses cached 3m. - Ratify v1.4.6 is the stable line (
config.).ratify. deislabs. io/ v1beta1 - v2.0.0-beta.2 (
Executor) says "do not use in production" and renamed its API between betas. - The Gatekeeper chart fails open (
Ignore, 3s). The sample constraint covers only Pods indefault.
2.Checks on the node
- CRI-O :
policy.jsonper Pod namespace (/) withetc/ crio/ policies/<ns>. json sigstoreSigned. - containerd 2.x :
bindirverifiers (exit 0 allows), only on transfer-service pulls, the CRI default since 2.1. - Legacy
registry.mirrors/configs/authsordiscard_silently fall back to local pulls with no verifiers (kind's default did). Check :unpacked_ layers = true journalctl -u containerd | grep -i 'Falling back to local image pull' ctr image pull --localskips verifiers too : test withcrictl pull.
| CRI-O | containerd | |
|---|---|---|
| Keyless identity | Exact email SAN only : no GitHub workflow URIs | Whatever your binary does |
| cosign v3 bundles | No (container-libs issue 388) | Your binary |
| Cached images | Re-checked at create when a policy path is set | Never checked |
| Pod shows | SignatureValidationFailed | ErrImagePull, then ImagePullBackOff |
| Warn mode | None : test on a canary pool | None |
3.Failure and caching
Fail: a registry or Rekor outage blocks admission.- Kyverno with
Ignorestill denied on registry errors (observed once).[Audit]still mutates. - Caches : Kyverno 60m, Gatekeeper 3m, policy-ctrl none.
- Private Sigstore : Kyverno
trustedRoot/tuf, policy-controllerTrustRoot, Ratify verifier parameters.
4.Digest mutation and GitOps drift
- Kyverno (autogen), policy-controller and Ratify rewrite templates to
tag@sha256: Argo CD and Flux see drift. - Pin digests in Git, so there is nothing to rewrite : the
digeststrategy of Argo CD Image Updater, or a FluxImagePolicywithdigestReflectionPolicy(Flux). - Or hide the diff :
ignoreDifferencesin Argo CD.
5.Check in CI too, as advice
kyverno apply ivpol.yaml --resource rendered.yamlruns only the validating path : tag-only images failverifyDigest. A policy file without.yamlis skipped with exit 0.policy-tester -policy cip.yaml -image REFchecks one image against one CIP.
6.Which engine does what
| VAP | Kyverno 1.19 | policy-ctrl 0.15.1 | Ratify + GK | CRI-O | containerd | |
|---|---|---|---|---|---|---|
| Signatures | no | cosign, notary | cosign .sig only | notation, cosign | sigstore | your binary |
| Tag to digest | no | mutateDigest | always | Assign | n/a | n/a |
| kubectl debug | if listed | if listed | digest only | sent by default | at pull | at pull |
| Image volumes | if CEL reads | extractor | no | no | yes | yes |
| Pod templates | your rules | autogen | yes | mutate only | n/a | n/a |
| Unmatched | your CEL | skipped | deny | by scopes | default | allowed |
| Trial mode | [Warn, Audit] | [Audit] or [Warn] | mode, no-match : warn | dryrun, then warn | none | none |
| On failure | Fail | Fail | Fail | Ignore | pull fails | pull fails |
Scheduling
Advanced Scheduling
1.What does kube-scheduler do ?
It gives a node to each Pod with no spec.nodeName.
- Filter : drop each node that breaks a hard rule.
- Score : rank the rest on soft rules. The highest total wins, ties at random.
- Bind : write
spec.nodeNamethrough the API server. - 0 nodes fit : try preemption, else Pending. The events name the failed rule.
- A Pod fits if its request fits the node's allocatable minus what is already requested.
- Effective request : the larger of the summed app + sidecar containers and the biggest init container, plus RuntimeClass overhead.
- Usage and limits are ignored. No requests = fits anywhere. QoS : Quick Revision.
2.One Pod through the scheduler
The scheduling framework (stable v1.19) chains plugin hooks.
- Scheduling cycle : serial, one Pod at a time, from PreFilter to Permit.
- PostFilter runs only when 0 nodes fit. Preemption lives there.
- Binding cycle : PreBind, Bind, PostBind. May overlap the next Pod.
- A failure at Reserve or later runs Unreserve and sends the Pod back to a queue.
- backoffQ : 1 s first, 10 s max. QueueingHints (GA v1.34) requeue only on an event that might help, like a new node.
3.Hard rules filter, soft rules score
| Control | Hard (Filter) | Soft (Score) | Plugin, weight |
|---|---|---|---|
| nodeSelector | every label must match | - | NodeAffinity, 2 |
| Node affinity | required... | preferred..., weight 1 to 100 | NodeAffinity, 2 |
| Pod (anti-)affinity | required..., with a topologyKey | preferred..., weight 1 to 100 | InterPodAffinity, 2 |
| Taints | NoSchedule, NoExecute | PreferNoSchedule | TaintToleration, 3 |
| Topology spread | DoNotSchedule (default) | ScheduleAnyway | PodTopologySpread, 2 |
| Requests | must fit free allocatable | LeastAllocated (default) | NodeResourcesFit, 1 |
| Priority | no filter : queue order, preemption | - | PrioritySort, DefaultPreemption |
| spec.nodeName | skips the scheduler | - | the kubelet admits or rejects it |
Also scoring : BalancedAllocation 1, ImageLocality 1. A soft rule only adds to a weighted sum, so it can lose.
4.Node affinity
- nodeSelector : the node must carry every listed label.
- requiredDuringScheduling... is hard. preferred... is soft, it adds its weight.
- Terms are ORed, matchExpressions in a term ANDed. With nodeSelector too, both must hold.
- Operators : In, NotIn, Exists, DoesNotExist, Gt, Lt (integer labels).
- IgnoredDuringExecution : relabeling a node never moves a running Pod.
5.Pod affinity and anti-affinity
- Matches labels of Pods already running in a domain.
topologyKeynames it and is required. kubernetes.= per node,io/ hostname topology.= per zone.kubernetes. io/ zone - Own namespace only, unless namespaces or namespaceSelector widen it.
- Costly. Not recommended above several hundred nodes.
- Trap : required anti-affinity on hostname with more replicas than nodes leaves the extras Pending.
1.Taints and tolerations
- A taint on a node repels. A toleration permits, it never attracts.
- NoSchedule blocks new Pods, PreferNoSchedule discourages, NoExecute also evicts.
- tolerationSeconds : time allowed under NoExecute. Unset = forever.
- operator : Equal (default) or Exists. Exists with no key matches all.
- Dedicated nodes : taint and label them. Pods get the toleration and required affinity on the label.
| Built-in taint | Effect | Set when |
|---|---|---|
| not-ready | NoExecute | Ready is False |
| unreachable | NoExecute | Ready is Unknown |
| memory-pressure | NoSchedule | MemoryPressure |
| disk-, pid-pressure | NoSchedule | DiskPressure, PIDPressure |
| network-unavailable | NoSchedule | network not set up |
| unschedulable | NoSchedule | kubectl cordon, drain |
| out-of-service | NoExecute | by hand, dead node |
| control-plane | NoSchedule | kubeadm control plane |
Keys start with node.kubernetes.io/, control-plane with node-role.kubernetes.io/. Pods tolerate not-ready and unreachable for 300 s, so they leave a dead node after about 5 min.
2.Topology spread
- maxSkew (> 0) : allowed gap to the least-loaded domain. topologyKey : the node label that makes domains.
- whenUnsatisfiable : DoNotSchedule (default) or ScheduleAnyway.
- labelSelector : which Pods count. Match the Pod's own labels.
- None set ? Soft defaults : hostname skew 3, zone skew 5. Only checked at scheduling time.
3.Priority and preemption
- PriorityClass (cluster-scoped) : yours go up to 1000000000. One may be globalDefault, else 0.
- Admission writes
spec.priority. Higher = earlier in activeQ. preemptionPolicy Never : jumps the queue, never evicts. - Preemption : delete lower-priority Pods on one node to make room, set
status.(a hint).nominatedNodeName - Best effort : PDBs may break, QoS ignored, preemption never frees DRA devices.
4.Skip, hold or swap it
- spec.nodeName : no filters, NoSchedule ignored. Pin with nodeSelector on kubernetes.io/hostname instead.
- schedulingGates (GA v1.30) : the Pod waits at SchedulingGated until every gate is gone.
- Profiles : several schedulerNames in one kube-scheduler. Pods pick with
spec..schedulerName - Descheduler : evicts Pods that now break a rule, kube-scheduler places them again.
- DRA (GA v1.34) : a ResourceClaim asks for a GPU. Device and node are picked together.
5.Why is it Pending ?
FailedScheduling : 0/N nodes are available: <reason>, ...
| Reason says | Fix |
|---|---|
| Insufficient cpu / memory | lower requests, add nodes |
| untolerated taint(s) | unnamed since v1.35 : describe node |
| node affinity/selector | label nodes or fix the rule |
| pod anti-affinity rules | preferred, or spread instead |
| topology spread constraints | raise maxSkew, add domains |
| were unschedulable | kubectl uncordon |
| unbound immediate PVCs | fix the PVC (Storage) |
| SchedulingGated (status, no event) | the gate owner removes it |
6.Commands
| Command | Shows |
|---|---|
kubectl events --for pod/<pod> --types=Warning | That Pod's warnings |
kubectl get events -A --field-selector reason=FailedScheduling | All scheduling failures |
kubectl describe node <node> | Taints, Allocated resources |
kubectl get nodes -L topology.kubernetes.io/zone | Zone as a column |
kubectl taint nodes <node> k=v:NoSchedule | Add a taint, a trailing - removes it |
kubectl get priorityclasses | VALUE, GLOBAL-DEFAULT |
Scheduling
Dynamic Resource Allocation
1.Why DRA ?
- Device plugin : only says how many,
limits: nvidia.com/gpu: 2. No overcommit, no sharing between containers, no pick by attribute, config per node. - DRA : says which. The scheduler picks specific devices from the driver's ResourceSlices by CEL on attributes and capacity. The kubelet has the driver prepare them through CDI.
- Status : GA since v1.34, gate locked on since v1.35. v1.37 serves only
resource.by default.k8s. io/ v1
| Aspect | Device plugin | DRA |
|---|---|---|
| Asks with | integer in limits | a ResourceClaim |
| Picks device | kubelet | scheduler, by CEL |
| Inventory | node allocatable | ResourceSlices |
| Sharing | none | containers, Pods |
| Config | per node | per claim |
| Node setup | Allocate RPC | Prepare + CDI |
| Partitions | static | partitionable (beta) |
| Health | drops from count | device taints (GA) |
Both can run. One resource name on one node comes from the plugin or DRA, not both.
2.The object model
Like PVC and StorageClass (Storage) : the claim asks, the class categorizes, the slice is the inventory.
- DeviceClass (cluster) : CEL selectors, optional config. Made by an admin or a driver chart.
- ResourceSlice (cluster) : devices per pool. The driver owns it and overwrites manual edits.
- ResourceClaim : the request. Status has allocation, reservedFor.
- ResourceClaimTemplate : one claim per Pod, deleted with the Pod (Owners and GC).
- DeviceTaintRule (cluster) : taints devices by driver, pool, device. In v1 since v1.37.
3.One GPU, two containers
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata: { name: big-gpu }
spec:
spec:
devices:
requests:
- name: gpu
exactly:
deviceClassName: gpu.example.com
selectors:
- cel:
expression: device.capacity["gpu.example.com"]
.memory.compareTo(quantity("40Gi")) >= 0
---
apiVersion: v1
kind: Pod
metadata: { name: trainer }
spec:
resourceClaims:
- name: gpu
resourceClaimTemplateName: big-gpu
containers:
- name: main
image: ubuntu:24.04
command: [sleep, infinity]
resources: { claims: [ { name: gpu } ] }
- name: sidecar # same GPU
image: ubuntu:24.04
command: [sleep, infinity]
resources: { claims: [ { name: gpu } ] }
4.Wiring rules
- Each
spec.entry has a name and exactly one of resourceClaimName, resourceClaimTemplateName.resourceClaims[] - A named claim must already exist in the Pod's namespace, or the Pod stays Pending.
- The Pod list only makes it available. A container gets the device only if its
resources.claimsnames it.
5.Inside a request
devices.: a name plus exactly, or firstAvailable[] (ordered fallback subrequests).requests[] - exactly : deviceClassName, selectors, allocationMode ExactCount (default, count 1) or All, adminAccess, tolerations, capacity.
- constraints[] : matchAttribute, distinctAttribute. config[].opaque : driver + parameters.
device.driver, device., and device., a Quantity (use compareTo).
A missing key is an error that aborts allocation, not a skip. Guard with has(), or optional chaining with orValue().
1.From claim to device
- The kubelet plugin publishes ResourceSlices.
- resourceclaim-controller creates claims from templates.
- DynamicResources runs CEL, picks devices first-fit.
- It writes allocation + reservedFor, then binds the Pod.
- The kubelet calls NodePrepareResources (gRPC).
- The driver returns CDI device IDs.
- The runtime injects them into the containers.
- NodeUnprepareResources after the Pod stops.
- Last Pod gone : deallocated, template claim deleted.
The rest of the cycle : Advanced Scheduling.
2.Sharing a device
- Template : each Pod gets its own claim and devices.
- Shared claim : many Pods name one ResourceClaim, 256 at most (reservedFor).
- Containers : one Pod claim listed in several containers.
- How they share at runtime (time-slicing, MPS) is up to the driver and its opaque config.
3.Features in v1.37
| Feature | Key fields | Feature gate | Maturity |
|---|---|---|---|
| Core DRA | v1 kinds, pod spec.resourceClaims | DynamicResourceAllocation | GA 1.34, locked 1.35 |
| Prioritized list | requests[].firstAvailable[] | DRAPrioritizedList | GA 1.36, locked 1.37 |
| Admin access | exactly.adminAccess + namespace label | DRAAdminAccess | GA 1.36 |
| Device taints | devices[].taints[], exactly.tolerations[] | DRADeviceTaints | GA 1.37 |
| DeviceTaintRule | deviceSelector {driver, pool, device} | DRADeviceTaintRules | GA 1.37 |
| Extended resource | extendedResourceName : limits served by DRA | DRAExtendedResource | GA 1.37 |
| Device status | claim status.devices[] | DRAResourceClaimDeviceStatus | GA 1.37 |
| Partitionable devices | sharedCounters[], consumesCounters[] | DRAPartitionableDevices | Beta 1.36, on |
| Consumable capacity | allowMultipleAllocations, capacity.requests | DRAConsumableCapacity | Beta 1.36, on |
| Binding conditions | bindingConditions, bindsToNode | DRADeviceBindingConditions | Beta 1.36, on |
| Resource health | allocatedResourcesStatus | ResourceHealthStatus | Beta 1.36, on |
| PodGroup claims | PodGroup spec.resourceClaims | DRAWorkloadResourceClaims | Beta 1.37, off |
Alpha, off : list-type and derived attributes, node allocatable resources, optional node operations, compatibility groups, pool status.
4.Gotchas
- Preemption never frees a DRA device. A higher-priority Pod stays Pending.
spec.nodeNameskips the scheduler, so nothing allocates the claim. Pin with nodeSelector on kubernetes.io/hostname.- firstAvailable is decided per Pod. Replicas can get different subrequests.
- Only v1 is on by default. A v1beta1 or v1beta2-only driver needs kube-apiserver
--runtime-config. - The admin label is resource.kubernetes.io/admin-access="true", not resource.k8s.io.
5.Commands
| Command | Shows |
|---|---|
kubectl get deviceclasses | "No resources found" = on |
kubectl get resourceslices --field-selector spec. | One node's devices |
kubectl get resourceclaims -A | pending, allocated, reserved |
kubectl describe resourceclaim <name> -n <ns> | First stop, Pending Pod |
kubectl get pod <pod> -o jsonpath='{.status. | Generated claim names |
Try it : dra-example-driver (mock GPUs, gpu.example.com). NVIDIA driver : gpu.nvidia.com, mig.nvidia.com.
Resources and autoscaling
Requests, Limits and QoS
1.Reservation and ceiling
Basics are in Quick Revision. Here : under the hood.
- Scheduler : per resource, requests on the node plus the new Pod must fit
node.. Usage is ignored. A miss :status. allocatable FailedScheduling,insufficient cpu(plugins). - For scoring only, a container with no request counts as
100mCPU and200Mi, so zero-request Pods spread out. Fit still uses zero. - HPA utilization is a percentage of the request.
2.Units and traps
| CPU | Memory | |
|---|---|---|
| unit | cores, m = milli | bytes, Mi, Gi, M |
| request | scheduling, cpu.weight share | scheduling, eviction ranking, Memory QoS hints |
| limit | cpu.max quota per 100 ms | memory.max hard cap |
| over limit | throttled, never killed | OOM-killed, exit 137, restarted |
| kind | compressible | not compressible |
| idle | usable above request if no limit | usable, but first to be evicted |
memory: 400mis 0.4 bytes.- CPU is absolute, not a share of the node.
- Limit only : the request copies it, even over a LimitRange
defaultRequest. - Smallest useful CPU limit :
10m(1 ms quota).
3.What lands in cgroup v2
| Field | cgroup v2 file | Example |
|---|---|---|
requests.cpu | cpu.weight, from shares = mCPU x 1024 / 1000 (2 to 262144) | 1 CPU = 100 (runc 1.3.2+, crun 1.23+), 39 before |
limits.cpu | cpu.max = "<quota> 100000" | 500m : 50000 100000; none : max 100000 |
limits.memory | memory.max | max when unset |
requests.memory | nothing hard | memory.min, .low, .high only with Memory QoS |
- The kubelet passes values through the CRI; the OCI runtime writes the files. The kubelet still writes Pod and QoS-tier cgroups with the old formula, so their
cpu.weightuses another scale. - An OOM kills every process in the container together (
memory.oom.group). cgroup v1 : the kubelet refuses it by default since 1.35 (failCgroupV1: true), removal not before 1.38.
4.Throttling below the average
- Throttling shows while average usage looks far below the limit. Check
nr_throttledincpu.stat, not averages. - Runtimes read the limit : Go 1.25 (when
go.modsays 1.25+) setsGOMAXPROCSfrom it (rounded up, minimum 2, never above the CPU count) and re-checks it after a resize.
5.Node allocatable
Allocatable = Capacity - kubeReserved - systemReserved - hard eviction threshold. By default both reserves are empty, so only memory. is held back. Docs example : 16 CPU, 32Gi minus 1 CPU / 2Gi kube, 500m / 1Gi system and <500Mi eviction gives 14.5 CPU, 28.5Gi. systemReserved should cover kernel memory, which is not charged to Pods.
cpu.max and memory.max. Over it, CPU is throttled and memory is OOM-killed.1.QoS beyond the rules
Rules in Quick Revision. The class is derived and fixed; an in-place resize that would change it is rejected.
| Class | oom_score_adj | Eviction position |
|---|---|---|
| Guaranteed | -997 | last, by priority |
| Burstable | min(max(3, 1000 - 1000 x memRequest / nodeMemory), 999), so 3 to 999 | first if over its request, else late |
| BestEffort | 1000 | first : usage is always over a zero request |
system-node-criticalPods also get -997. A sidecar's score is capped at that of the app container with the smallest memory request, so it is not killed first.
2.Two ways to die of memory
3.Who is evicted first
- Usage above request ? (not for PID pressure)
- Then lower priority.
- Then usage relative to request.
memory.= capacity minus working set. Page cache inavailable active_filecan fake pressure; the documented fix is memory request = limit.- Disk pressure : dead containers and unused images go first.
MemoryPressure,DiskPressure,PIDPressurebecome taints (scheduling).HugepageAwareEviction(beta, on in 1.37) subtracts huge pages frommemory..available
4.Default hard thresholds (Linux)
memory. | nodefs. | imagefs. | nodefs. | imagefs. |
|---|---|---|---|---|
| <100Mi | <10% | <15% | <5% | <5% |
evictionHard value zeroes the others, unless mergeDefaultEvictionSettings: true . List every one you want. Soft thresholds need a grace period; evictionPressureTransitionPeriod is 5m.5.Commands, and Memory QoS in v1.37
| Command | What it shows |
|---|---|
| kubectl describe node <node> | capacity, allocatable, allocated requests |
| kubectl get pod <p> -o | the QoS class |
| kubectl describe pod <p> | OOMKilled, exit 137, or FailedScheduling |
| kubectl top pod <p> --containers | live usage (needs metrics-server) |
| stat -fc %T /sys/fs/cgroup/ | prints cgroup2fs on cgroup v2 |
| cat cpu.weight cpu.max memory.max cpu.stat | applied values, in the container's cgroup |
- Beta and on, but inert until configured (
memoryThrottlingdefaults to null). With a factor,Factor memory.high = request + factor x (limit - request): 0.9, 256Mi, 1Gi gives about 947Mi. Never for Guaranteed or a container with request = limit. memoryReservationsetsPolicy: Tiered Reservation memory.min(Guaranteed) ormemory.low(Burstable) to the request; a cache-heavy Guaranteed Pod may then OOM. Kernel 5.9+.
A cache or JVM service crash-loops with OOMKilled. Pitfall : its own memory cap (-m, -Xmx, GOMEMLIMIT) equals the container limit, but buffers and off-heap memory live above that cap. Do better : keep the app cap clearly below the limit; raise the limit only once you know why memory grew.
1.Guardrails at admission
- LimitRange defaulting : a missing
defaulttakesmax; a missingdefaultRequesttakesdefault, thenmin. - Request only : gets the default limit. Limit only : request = its own limit.
default.cpu: 500mand a700mrequest with no limit : rejected (must be less than or equal to cpu limit).- Two LimitRanges : the winning default is not deterministic.
maxLimitRequestRatio>= 1. Admission only : running Pods never change (plugin order).
2.ResourceQuota
| Key | What it sums |
|---|---|
requests. / .memory | sum of requests |
limits. / .memory | sum of limits |
requests., hugepages- | scratch, huge pages |
requests. | extended, requests. only |
<class>. | DRA device count |
pods, count/, requests. | objects, PVC storage |
- Scopes :
BestEffort,NotBestEffort,Terminating,NotTerminating,PriorityClass,CrossNamespace,PodAffinity VolumeAttributes.Class - Only non-terminal Pods count.
- Quota on
cpuormemory: Pods must set it (LimitRange). - Over quota : Deployment, no Pods.
- Write access can delete it : RBAC.
3.Effective request : init, sidecars, Pod level
- sidecar proxyindex 0 : 100m
- init migrateindex 1 : 1 CPU + sidecar = 1.1
- app500m; running sum 600m
- effective1.1 CPU, for the Pod's whole life
Init and sidecars : Pods. The exact KEP-753 rule (short form : Advanced Scheduling). RuntimeClass overhead is added after.
Pod-level spec.resources (beta, on since 1.34) : only cpu, memory, hugepages-. Pod values win for scheduling, the Pod cgroup and QoS. Container limits must be <= Pod limits; the rest share the budget. Not used by the CPU, memory or topology managers in 1.37, nor on Windows. An empty Pod-level block no longer changes QoS.
4.The CPU-limits question
| Policy | Good for | Cost |
|---|---|---|
| request only | latency-sensitive services | needs accurate requests; apps lean on spare CPU |
| request = limit | benchmarks, predictable capacity | idle CPU wasted; throttled if sized low |
| request + headroom | a middle ground | still throttles at the cap |
| whole-core limit | greedy batch Jobs | throttling is intended |
Mainstream answer : CPU requests everywhere, sized from data (VPA); CPU limits only for a hard cap (multi-tenant, greedy batch, benchmarks); memory request = limit where it matters. The Kubernetes blog carries a dissent, for predictability.
A gateway proxy's TLS throughput sat far below line rate, throttled at low average CPU. Pitfall : a small CPU limit on a data-path Pod. Do better : no CPU limit on proxies, a request from measured peak, a throttle-ratio alert.
5.Commands
| Command | What it shows |
|---|---|
| kubectl describe quota -n <ns> | used vs hard, per key |
| kubectl describe limitrange -n <ns> | defaults, min, max, ratio |
| sum by (namespace,pod) (rate(container_ | throttle ratio by Pod (Prometheus). Aggregate by Pod, not container name |
Resources and autoscaling
In-Place Pod Resize
1.What is in-place resize ?
- Change the CPU and memory requests and limits of a running Pod's containers without a new Pod : same node, same IP, warm caches.
- KEP-1287, gate
InPlacePodVerticalScaling: alpha 1.27, beta (on) 1.33, GA 1.35, locked on, due for removal in 1.38. - Do not set the gate :
=trueonly logs a warning,=falsestops the component at startup.
- Template change in a Deployment = new Pods (a rollout).
- A /resize patch on the live Pod = the same Pod, new numbers.
- The QoS class never changes : a resize that would change it is rejected.
2.Desired, allocated, actual
| View | Field | Written by |
|---|---|---|
| desired | spec. | you, through /resize |
| allocated | status. (requests only) | the kubelet, when it admits the resize |
| actual | status. | the kubelet, from what the runtime applied |
| progress | status. : PodResizePending, PodResizeInProgress | the kubelet |
status. is deprecated and no longer set since 1.33. Scripts must read the two conditions instead. Pod spec and status basics are in Pods.
3.The resize subresource
- Only
--subresource resizeonpatch,editorapply(kubectl 1.32+) changes resources. A plain update cannot. - RBAC resource :
pods/(RBAC). Grantresize patchon it to whoever resizes. - It can change regular containers, sidecars, running non-sidecar init containers (GA 1.37), Pod-level
spec.(beta 1.36), and with an alpha gate a memory-backedresources emptyDirsizeLimit.
$ kubectl patch pod resize-demo \
--subresource resize --patch \
'{"spec":{"containers":[{"name":"app","resources":
{"requests":{"cpu":"800m"},"limits":{"cpu":"800m"}}}]}}'
4.One resize, end to end
Admission : PodResizeValidator, LimitRanger, ResourceQuota. One resize is atomic; patches sent before the kubelet acts merge.
5.Conditions and events
PodResizePending, reason Deferred : fits the node's allocatable but not its free room. Retried.PodResizePending, reason Infeasible : this node can never do it. Not retried.PodResizeInProgress: allocated, being applied. Reason Error when applying failed.- Both at once : a new resize arrived while the last one is still applying.
- Events :
ResizeStarted,ResizeCompleted,ResizeDeferred,ResizeInfeasible, andResizeErroras a Warning.
conditions: # a resize that waits
- type: PodResizePending
status: "True"
reason: Deferred
message: "... cpu, requested: 6000,
used: 3950, capacity: 8000"
spec, actual in status. You change desired only through /resize; the kubelet catches up in the same Pod and reports progress in PodResizePending and PodResizeInProgress.1.resizePolicy
- Per container and resource : NotRequired (resize live if possible) or RestartContainer (
restartCountgoes up). Only these two; neverPreferNoRestart. - No entry =
NotRequired. An entry with an emptyrestartPolicyfails validation. - Both resources change and either is
RestartContainer: the container restarts. - Pods with
restartPolicy: Nevermust useNotRequired. - Fixed after creation (recreate the Pod). Since 1.36 non-sidecar init containers may not use
RestartContainer.
containers: # Pod resize-demo - name: app image: registry.k8s.io/pause:3.8 resizePolicy: - resourceName: cpu restartPolicy: NotRequired - resourceName: memory restartPolicy: RestartContainer resources: # Guaranteed : change # request and limit together requests: {cpu: 500m, memory: 128Mi} limits: {cpu: 500m, memory: 128Mi}
2.How the kubelet applies it
Node-side admission : other Pods' allocated requests + this Pod's new requests <= allocatable. Then the cgroups are written (cpu.weight, cpu.max, memory.max; see Requests, Limits and QoS).
Growing : pod cgroup first, then containers. Shrinking : containers first, pod cgroup last. Decreases before increases. Memory before CPU. On any error the kubelet stops and retries on a later sync.
3.Shrinking memory
- Allowed with
NotRequiredsince 1.34. - Usage at or above the new limit : the kubelet refuses, sets
PodResizeInProgressreason Error with a WarningResizeError, and retries later. - A race remains : usage can rise between check and write. On cgroup v2 the kernel then reclaims, or OOM-kills.
- Common mistake : expecting a shrink below usage to "stay pending". It is an
Error, notPending.
4.Waiting and accounting
- Retry order : no request increase, then higher PriorityClass, then Guaranteed before Burstable, then the one waiting longest.
- Scheduler and quota count max(desired, allocated, actual) (not desired if
Infeasible). - Race : a new Pod placed from cached data can be rejected
OutOfCPU(open issue). - 1.37 alpha (
...Scheduleron API server, scheduler, kubelet) : preempt lower-priority Pods for aPreemption Deferredresize, same node only.
5.What happens when
| Situation | Outcome |
|---|---|
| fits the node now | PodResizeInProgress, then done |
| fits allocatable, but other Pods use the room | Deferred, retried |
| above the node's allocatable | 403 at admission since 1.36 (PodResizeValidator; it ignores other Pods); was Infeasible |
static or Windows Pod, QoS change, removing a set value, not CPU or memory, resizePolicy change | rejected by API validation (403/422 at the API server); QoS case : "Pod QOS Class may not change as a result of resizing" |
memory change with swap and NotRequired | Infeasible |
| Guaranteed Pod on a static CPU (Memory) manager node, CPU (memory) change | Infeasible unless an alpha gate; the other resource resizes |
| memory limit at or below usage | PodResizeInProgress Error, retried |
Burstable Pods on static manager nodes are not blocked. Ephemeral containers are never resizable.
resizePolicy sets restart per resource, fixed at creation. The kubelet admits against allocatable, writes cgroups in a safe order and refuses memory shrinks below usage.1.Who resizes in place ?
- A
resourceschange in a Deployment, StatefulSet or DaemonSet template still rolls new Pods. Job template resources cannot change. - So a template's
resizePolicymatters only when something patches the live Pod. Workload-level in-place resize is only a design sketch. - A manual patch is out of band from Git : when the Pod is replaced, the template value returns.
- Operators resize in place only if they call
pods/. CloudNativePG recreates its Pods instead.resize
2.Pod-level resize
- Beta and on since 1.36, still beta in 1.37. Patch Pod-level
spec.(the API) throughresources /resize. - No Pod-level
resizePolicy. A container without its own limit inherits the Pod limit, so it resizes too, and restarts if its own policy saysRestartContainer. - Rules : Pod requests >= sum of container requests; each container limit <= Pod limit.
- Needs cgroup v2, Linux, containerd 2.0+ or CRI-O.
3.VPA uses it
updateMode: InPlaceOrRecreate(GA in VPA 1.6.0) resizes all containers throughpods/and falls back to eviction.resize InPlace(alpha, VPA 1.7.0) never evicts.Autois deprecated.- The updater needs
patchonpods/. VPA does not support Pods with Pod-level resources. Fallback rules : VPA.resize
4.Runtimes
| Runtime | CPU live ? | Memory live ? |
|---|---|---|
| Go, Rust, C | yes | yes, more room is usable at once |
| JVM (HotSpot) | mostly : availableProcessors() re-reads cgroups, but GC and pool threads are sized at start | no : max heap is fixed at start |
| Node.js | yes | heap capped by --max-old-space-size, set at start |
| Python | yes | the GA blog says it needs a restart |
Rule of thumb : CPU NotRequired; memory RestartContainer for runtimes with a fixed heap.
- Pitfall : memory
NotRequiredin the Git-managed template of a fixed-heap JVM service, "so future bumps land live". A Git edit still rolls the Pod, and a live memory raise adds only off-heap room, no heap. - Do better : treat
resizePolicyas the contract for/resizecallers (VPA, kubectl), use memoryRestartContainerfor fixed heaps, and write manual resizes back to Git.
5.Commands
| Command | What it does |
|---|---|
| kubectl get pod POD -o jsonpath='{.status. | The CPU request the kubelet allocated |
| kubectl get pod POD -o jsonpath='{.status. | Why a resize waits (reason and message) |
| kubectl get events --field-selector involvedObject. | The five Resize* events |
| kubectl exec POD -- cat /sys/fs/ | Applied cgroup v2 values (max 100000 = no CPU limit) |
| kubectl auth can-i patch pods/resize | RBAC check for the subresource |
Kubelet metrics (alpha) for Prometheus : kubelet_, kubelet_, kubelet_.
/resize caller (kubectl, VPA, an operator) resizes in place, until the next rollout. CPU resizes live almost everywhere; fixed heaps need a restart.Resources and autoscaling
Vertical Pod Autoscaler
1.What is VPA ?
A feedback loop that moves a Pod's requests (and limits in proportion) toward its real usage. It never changes replicas. An add-on from kubernetes/, not core : two CRDs (VerticalPodAutoscaler, ...Checkpoint) and three controllers you install. It reads usage from metrics-server. Beside it, HPA adds Pods and Karpenter adds nodes.
VPA 1.8.0 runs on Kubernetes 1.36 to 1.38. API autoscaling. only (v1beta2 not served since 1.3.0).
2.Three components, one loop
| Component | Does |
|---|---|
| recommender | every 1m reads metrics., writes status and checkpoints. One active, metrics :8942 |
| updater | every 1m evicts or resizes out-of-date Pods. Evicted Pods get new values from the webhook. One active, metrics :8943 |
| admission controller | webhook on Pod CREATE, failurePolicy: Ignore. Patches requests, limits. Several active, metrics :8944 |
3.The VPA object
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata: {name: my-app-vpa}
spec:
targetRef:
{apiVersion: apps/v1, kind: Deployment, name: my-app}
updatePolicy: {updateMode: "Recreate"} # not "Auto"
resourcePolicy:
containerPolicies:
- containerName: "*"
minAllowed: {cpu: 100m, memory: 50Mi}
maxAllowed: {cpu: 1, memory: 500Mi}
controlledResources: ["cpu", "memory"]
controlledValues: RequestsAndLimits
targetRef: any controller with a/scalesubresource, or a well-known kind (Deployment, StatefulSet, DaemonSet, Job...). Its Pods must be owned directly by it, so operator Pods need a CR with/scale.- Per container
mode: Offskips a container (its default,Auto, is not the deprecated one). controlledValues: RequestsOnlyleaves limits alone.updatePolicy.minReplicasoverrides the updater's floor.evictionRequirementsfilter by direction, for example only evict to scale up.
4.Update modes
| Mode | At creation | Running Pods | Status |
|---|---|---|---|
Off | no | no | recommend only |
Initial | yes | no | stable |
Recreate | yes | evicts (PDBs apply) | default when unset |
InPlaceOrRecreate | yes | in place first, eviction as fallback | GA in VPA 1.6.0 |
InPlace | yes | in place only, never evicts | alpha in 1.7.0, gate InPlace |
Auto | yes | evicts | deprecated in 1.5.0, same as Recreate |
5.Install
- Script : install metrics-server first, then
./hack/vpa-up.shfrom a clone ofkubernetes/. It installs intoautoscaler kube-systemand needs OpenSSL 1.1.1+.vpa-down.shremoves it. - Helm : chart 0.13.0, "not ready for production" in its README. It sets
--in-place-skip-, unlike the binary. Recommender only :disruption-budget=true --set updater.enabled=false --set admissionController.enabled=false.
updateMode and by which of them you installed.1.Reading a VPA
kubectl get vpa : MODE, CPU, MEM (first container's target), PROVIDED; -o wide adds MinReplicas, OOMSeconds. Status has four numbers per container :
- targetwhat the webhook applies.
- lowerBound"not guaranteed to be sufficient".
- upperBoundabove it is likely waste.
- uncappedTargetbefore
minAllowed/maxAllowed.
RecommendationProvidedLowConfidenceNoPodsMatchedFetchingHistory
2.The recommender recipe
- Grouped per VPA and container name, not Pod.
- CPU : a decaying histogram, 24h half-life.
- Memory : one peak per 24h interval, 8 intervals.
- Target p90, lower p50, upper p95, then +15%.
- Short history widens bounds : upper x3 at 12h, x2 at 24h, x1.14 at a week; lower x0 with none.
- Per-Pod floor 25m / 250Mi, split across containers.
- Caps :
min/maxAllowed, LimitRange.
An OOMKill adds a synthetic sample of max(used + 100Mi, used x 1.2). The floor unit is MiB, even though the flag says -mb.
| Recommender flag | Default | Effect |
|---|---|---|
--recommender- | 1m | loop |
--cpu-histogram- | 24h | day-old weighs half |
--memory- x -count | 24h x 8 | a peak a day |
--target-cpu-, memory | 0.9 | target |
--recommendation- | 0.15 | +15% on all |
--pod-recommendation-, -memory-mb | 25 / 250 | floor, in MiB |
3.When the updater acts
updatePolicy.minReplicas: 1 if you mean it.- Candidate : a request outside [lowerBound, upperBound] (a missing request counts as outside), or one of the two rules in the box.
- The largest summed change goes first.
- Safety : at least
minReplicas(2) live, at most half evicted at once (tolerance 0.5), PDBs through the Eviction API, an optional rate limit, and no evictions if the admission controller looks dead.
4.In place, and its fallback
InPlaceOrRecreate resizes all containers at once through pods/resize (In-Place Resize). It evicts instead :
- at oncefailed PATCH (a QoS change is rejected),
Infeasible, or reasonError. - later5 min
Deferred, or 1 hour in progress (fixed, not flags).
The fallback still obeys minReplicas and tolerance. The kubelet refuses a memory-limit cut below current usage (reason Error, see In-Place Resize), so VPA evicts instead. InPlace (alpha) never evicts; it retries an infeasible size once the recommendation drops.
5.Limits and LimitRange
RequestsAndLimitskeeps the request:limit ratio. Request 10m with limit 1 CPU (ratio 100) : at request 300m the limit becomes 30 CPU. UseRequestsOnly, or keep them close (Requests and Limits).- A limit of
0is not "unlimited" : with a request set, it is rejected. Omit the limit. - Over a LimitRange
max, the limit drops to max and the request in proportion. - If the VPA policy and a LimitRange conflict, VPA's docs say VPA wins.
1.VPA with HPA
| Pattern | Safe ? | Why |
|---|---|---|
| VPA on CPU, HPA on CPU utilization | no | the loop |
VPA controlledResources: ["memory"], HPA on CPU | yes | upstream pattern |
| HPA on custom or external metrics (RPS, queue depth, KEDA), VPA on both | yes | different signals |
VPA Off, any HPA | yes | nothing applied |
HPA utilization is usage divided by request, and VPA moves the request. More in HPA in Depth.
2.Recommender only
- Install only the recommender, give every workload a VPA with
updateMode: "Off", and change requests by hand in Git : no evictions, no webhook. - Goldilocks : a dashboard on top of it.
- kube-state-metrics removed its VPA metrics in v2.9.0 : export
status.with CustomResourceState.recommendation
max_over_time(container_memory_rss
{namespace="NS",container="C"}[7d])
max_over_time(container_memory_working_set_bytes
{namespace="NS",container="C"}[7d])
PromQL : close together, the target is honest. RSS far below : page cache inflates it.
3.Where the numbers mislead
- Memory follows working set, page cache included : file-heavy workloads look far too big.
- Bursty, short-lived workloads get undersized : p90 + 15% misses rare peaks (Jobs).
- A recommender restart forgets the current interval's peak so far.
- Identical tiny values are the floor, not data.
- Labels matching stray Pods pollute the aggregate. Two VPAs on one Pod : undefined.
- Pod-level
resourcesare not supported. - Bigger than any node or quota : Pods stay Pending. Set
maxAllowed(Karpenter).
4.Operating it
- Read release notes first : the 1.7.0 recommender reset recommendations from checkpoints (fixed in 1.7.1).
- Back up
verticalpodautoscalercheckpointsfirst. - After the roll, prove history loaded : log line
Fetching VPA checkpoints ... count=N.PROVIDED=Trueproves nothing (old status stays). - Do not resize from numbers within 24h of a recommender restart.
The webhook's failurePolicy: Ignore silently skips mutation while it is down.
- Memory limits copied from VPA on a file-heavy service came out many times its resident memory. Bursty proxies sized from VPA came out too small : VPA saw only steady state.
- Pitfall : treating the VPA target as the size.
- Do better : use VPA as one input. Check memory against peak RSS, size bursty and data-path Pods from peaks. Keep a CPU limit off the recommender : its own throttling skews every number.
5.Commands
| Command | What it shows |
|---|---|
| kubectl get vpa -A | MODE, CPU, MEM, PROVIDED for every VPA |
| kubectl describe vpa web -n shop | recommendation, conditions, eviction events |
| kubectl get vpa web -n shop -o | just the targets |
| kubectl get pod <p> -o | what the webhook applied (vpaUpdates) |
| kubectl get verticalpodautoscalercheckpoints -A -o json > checkpoints.json | back up before an upgrade |
| kubectl -n kube-system logs deploy/ | did history load |
updateMode: "Off", and check memory targets against resident memory before you trust them.Resources and autoscaling
Horizontal Pod Autoscaler in Depth
1.Beyond the basics
- The formula :
desired = ceil(current x value / target); a ratio of 0.9 to 1.1 changes nothing (tolerance 0.1). Deployments and Scaling works an example. Everything else here stops that formula from flapping. autoscaling/v2is GA since 1.23 (v2beta2not served since 1.26).autoscaling/v1is still served, CPU only; v2-only fields survive as annotations.- It scales anything with a
scalesubresource : Deployment, StatefulSet, ReplicaSet, CRDs. Never a DaemonSet. - A controller loop in kube-controller-manager, every 15s. In 1.37 a new HPA, or one whose spec changed (
HPAGeneration, beta, on), is reconciled at once.
2.One sync
- Read the target's
/scale: replicas and selector. - Out of
min..maxtoday ? Clamp, without reading metrics. currentin the formula isscale.spec.replicas, the desired count, not the Pods running now.- Every metric failed, or some failed and the rest say "fewer" : an error, no change.
- Some failed and the rest say "more" : scale up, and still report the error.
3.Which Pods count
- Pods being deleted are ignored, Failed Pods discarded (phases : Pods).
- Pending Pods (every metric type) and Pods with no sample are set aside.
- CPU only : a not-yet-ready Pod is set aside. In its first 5 minutes if NotReady or its last sample predates its readiness change; after that only if it was never Ready.
- The ratio comes from the rest. Inside tolerance : stop.
- Recompute conservatively. If that flips direction or lands inside tolerance : no change (status still shows the plain average).
On a scale down a missing Pod counts as 100% of its request (or the target if higher), on a scale up as 0%. Set-aside Pods : 0% on a scale up. Pods metrics : the target, going down.
Object, External : a Value target multiplies by Running and Ready Pods; AverageValue gives ceil(total / target), checked against status.replicas.
4.kube-controller-manager flags
Flag ("..." = --horizontal-) | Default |
|---|---|
...-sync- | 15s |
...-tolerance | 0.1 |
...-downscale- | 5m |
...-cpu- | 5m |
...-initial- | 30s |
--concurrent- | 5 |
Cluster-wide; a per-HPA tolerance overrides it. -upscale-delay and -downscale-delay are gone since 1.31 : set one and the process exits (unknown flag).
5.Utilization needs requests
Utilizationis usage as a percent of the request (see Requests, Limits and QoS).- If any container in a Pod (often a sidecar) has no request for that resource, the HPA takes no action on that metric.
- Fix : request on every container, or
ContainerResourcefor one named container (next sheet). - With Pod-level requests set (
PodLevelResources, beta, on) and no container named, the Pod-level request is the denominator.
1.Five metric types
| Type | Reads | Targets | From |
|---|---|---|---|
Resource | cpu or memory per Pod, summed over containers | Utilization, AverageValue | metrics. |
ContainerResource | one named container (GA 1.30) | Utilization, AverageValue | metrics. |
Pods | a per-Pod custom metric, averaged | AverageValue only | custom. |
Object | one object in the namespace (describedObject) | Value, AverageValue | custom. |
External | not tied to an object; matching series are summed | Value, AverageValue | external. |
No metric set : CPU Utilization 80%. Several metrics : each one proposes a count and the largest wins.
2.Utilization, AverageValue, Value
| Target | Compares | Replicas |
|---|---|---|
Utilization | mean usage / request, in percent, over the counted Pods | ceil(current x mean% / target%) |
AverageValue | mean per Pod (Object, External : total / status.) | ceil(current x mean / target), or ceil(total / target) |
Value | one raw series value, as is | ceil(readyPods x value / target) |
AverageValue on CPU or memory needs no request (raw usage, 500m). 1500m means 1.5.
3.Where the numbers come from
metrics.k8s.io/v1is GA in 1.37, yet the 1.37 HPA readsv1beta1of both the resource and external APIs : keep serving it.custom.: the version the server prefers (discovery).metrics.k8s.io - Installing KEDA takes
external.from prometheus-adapter. See metrics-server, Prometheus.metrics.k8s.io - The docs prefer custom over external : the external API can expose anything.
4.The behavior field
behavior:
scaleDown:
stabilizationWindowSeconds: 300 # kcm flag
policies:
- {type: Percent, value: 100, periodSeconds: 15}
scaleUp:
stabilizationWindowSeconds: 0
policies:
- {type: Percent, value: 100, periodSeconds: 15}
- {type: Pods, value: 4, periodSeconds: 15}
selectPolicy: Max
- Each sync records a recommendation. Scale down takes the highest one in the window (0 to 3600s) : a rolling max.
- A policy caps change over its
periodSeconds(1 to 1800).selectPolicy:Max= most change,Min= least,Disabled= off. - Per-direction
tolerance, GA in 1.37 : 0.05 down and 0.01 up on a 100Mi target scale down below 95Mi, up above 101Mi. - No
behaviorat all (everyautoscaling/v1HPA) : the legacy path, scale up capped atmax(2 x current, 4)per sync (ScaleUpLimit).
5.A policy at work
Pods 4 / 60s and Percent 10 / 60s (Max) : above 40, 10% allows more, so it removes 8, then 8 (7.2 rounded up); below 40, 4 at a time."Scale down feels slow" : that is the 300s window, by design. Change it per HPA or with the kcm (kube-controller-manager) flag.
behavior : no wait up, a 5-minute rolling max down, policies cap each step.1.Reading describe hpa
| Condition | True reasons | False reasons |
|---|---|---|
AbleToScale | ReadyForNewScale, ScaleUpStabilized, ScaleDownStabilized, SucceededRescale ... | FailedGetScale, FailedUpdateScale |
ScalingActive | ValidMetricFound | ScalingDisabled, FailedGet...Metric ... |
ScalingLimited | TooManyReplicas, TooFewReplicas, ScaleUpLimit, ScaleDownLimit | DesiredWithinRange |
ScaledToZero | ScaledToZero | NotScaledToZero |
Events : SuccessfulRescale, FailedComputeMetricsReplicas, AmbiguousSelector (two HPAs, same Pods : no scaling).
2.Rollouts and GitOps
- The HPA owns
replicas; the Deployment splits them across ReplicaSets, surge Pods included. - Drop
spec.replicasfrom the manifest, or every apply resets it. Naively it drops to 1 once : usekubectl apply edit-last-applied(client-side) or transfer field ownership (server-side, API Machinery, Flux). - Readiness that passes only when warm keeps startup CPU from scaling up (Rollouts and Probes). CPU target with headroom :
(1 - buffer) / (1 + growth); 10% and 30% give about 70%.
3.Scale to zero in 1.37
- Gate
HPAScaleToZero: beta, on in 1.37, in kube-apiserver and kube-controller-manager. Upgrade both first;minReplicas >= 1before a downgrade. minReplicas: 0needs anObjectorExternalmetric; CPU or memory alone is rejected.ScaledToZerocame in 1.36. RaisingminReplicaswakes an HPA-held zero. From zero : no tolerance or ready-Pod count; on the way to zero the 300s window applies.
4.Beside VPA, KEDA and Karpenter
| Tool | Changes | Signal | See |
|---|---|---|---|
| HPA | replica count | resource, custom, external metrics | this chapter, Deployments and Scaling |
| VPA | container requests | usage history | Vertical Pod Autoscaler |
| KEDA | creates and drives an HPA, adds 0 to 1 | event sources | KEDA |
| Karpenter | nodes | unschedulable Pods | Karpenter |
- Never VPA and HPA on the same resource metric (VPA). KEDA owns
keda-hpa-<name>: add no HPA of your own to that workload. min = max : a no-op. - CPU limit = request : utilization tops out near 100%, so a sync grows replicas by at most about
100 / target(1.25x at 80%); throttling hides demand (Requests and Limits).
KEDA-scaled services that never go to zero, with a tuned cooldownPeriod. Pitfall : it only governs scaling to zero; N to 1 follows the HPA's 300s window. Do better : set behavior in advanced.; check the metric with kubectl get --raw.
5.Commands
| Command | Shows |
|---|---|
| kubectl get hpa -A | TARGETS as current / target |
| kubectl describe hpa web | metrics, conditions, events |
| kubectl get events --field-selector involvedObject.kind= | rescale history and failures |
| kubectl get --raw /apis/ | the numbers the HPA reads |
replicas, and give scale to zero an Object or External metric. Never let two autoscalers drive the same workload on the same signal.Networking internals
Pod Networking and CNI
1.What Kubernetes asks for
- Kubernetes states the rules (one IP per Pod, Pod to Pod with no NAT). A plugin builds them.
- Observed on k3s v1.37.1 : Pod IPs crossed the tunnel intact (
10.42.1.2 > 10.42.0.5).
- Pod IPsthe network plugin (CNI)
- Service IPskube-apiserver
- Node IPskubelet or cloud controller
2.Who calls CNI
- The kubelet only calls CRI. containerd makes the netns (none for hostNetwork) and runs CNI.
- The kubelet sorts the IPs into
status.podIPs, then setsPodReadyToStartContainers(GA v1.37). - Errors :
kubectl get events -A --field-selector reason=FailedCreatePodSandBox
3.The CNI contract
- A plugin is a binary :
CNI_*env in, JSON config on stdin, a result (ips,routes) or an error code on stdout. - Spec 1.1.0 added
GCandSTATUS. containerd sends STATUS (1.1.0 configs only), never GC. - DEL runs the chain in reverse, must be repeatable, and follows every ADD, even a failed one.
| Variable | ADD | DEL | GC | STATUS |
|---|---|---|---|---|
CNI_COMMAND | req | req | req | sent |
CNI_CONTAINERID | sandbox | req | - | - |
CNI_NETNS | req | opt | - | - |
CNI_IFNAME | eth0 | req | - | - |
CNI_ARGS | K8S_POD_* | opt | - | - |
CNI_PATH | opt | opt | req | opt |
4.A conflist, chained
- containerd loads only the lexically first file (
max_conf_num = 1) : a runtime rule, not the spec. - k3s uses
/.var/ lib/ rancher/ k3s/ agent/ etc/ cni/ net. d - Later plugins get
prevResult; the main one runs its own IPAM plugin. bandwidth: Pod egress is shaped on anifbnamedbwp<hash>.
{ "name": "cbr0", "cniVersion": "1.0.0", "plugins": [
{ "type": "flannel", "delegate": { "hairpinMode": true,
"forceAddress": true, "isDefaultGateway": true } },
{ "type": "portmap", "capabilities": { "portMappings": true } },
{ "type": "bandwidth", "capabilities": { "bandwidth": true } } ] }
5.A node with no network
- Empty conf dir :
cni plugin not initialized, NodeReady=False,not-readytaints (evict after 300s). - Only hostNetwork Pods start, so CNI agents are hostNetwork DaemonSets.
- A conflist
mv'd in is ignored (create event). Checkcrictl info | jq '..lastCNILoadStatus' NetworkUnavailableis a separate signal with its own taint.
| Writer | NetworkUnavailable reason |
|---|---|
| Route controller | RouteCreated, NoRouteCreated |
| Upstream flannel | FlannelIsUp |
| calico-node | CalicoIsUp, CalicoIsDown |
| Cilium operator | CiliumIsUp |
| kubelet, k3s flannel | never |
1.Inside the Pod, on the host
- Pod :
eth0@if5,10.42.1.2/24,default via 10.42.1.1(thecni0address). - Host :
veth71d39cee@if2 ... master cni0.if5and index5:are one veth pair. cni0appears with the first non-hostNetwork Pod. Pod MTU 1450 fromFLANNEL_MTU.
Observed on k3s v1.37.1+k3s1, flannel VXLAN.
2.Bridge or routed
- flannel : Pods share an L2 on
cni0, so a Pod withNET_RAWcan ARP-spoof. k3s says use restricted. - Calico : a
cali*host side with proxy ARP, one /32 route per local Pod. - Calico v3.33 has Felix, not BIRD, program IPIP routes by default. Its docs page still says BGP.
| flannel + bridge | Calico routed | |
|---|---|---|
| Host side | port on cni0 | cali*, /32 route |
| Pod gateway | cni0 (.1) | 169.254.1.1 |
| Neighbours | local Pods, one L2 | MAC ee:ee:ee:ee:ee:ee |
| ARP spoofing | yes, with NET_RAW | no shared L2 |
3.One packet across nodes (flannel VXLAN, observed)
- Per peer : one route, one neighbour, one FDB entry (
ip route,ip neigh,bridge fdb), from Node annotations. tcpdump -nni <uplink> udp port 8472:VXLAN ... vni 1aroundIP 10.42.1.2 > 10.42.0.5.- 50 bytes each :
ping -M do -s 1422passes, 1423 fails. A missed MTU : pings work, big transfers hang.
4.Overlay, routed or VPC-native
- Overlay : only UDP between nodes. Routed : one L2 (
host-gw) or BGP. VPC-native : real VPC IPs. - Cilium defaults to
routingMode: tunnel(VXLAN). - 8472 (kernel) vs 4789 (IANA) : a firewall for one breaks the other. Never expose 8472.
| On a 1500 link | Port | Cost | Pod MTU |
|---|---|---|---|
| flannel vxlan | UDP 8472 | 50 | 1450 |
| flannel host-gw | none | 0 | 1500 |
| wireguard-native | UDP 51820 | 80 | 1420 |
| Calico VXLAN | UDP 4789 | 50 | 1450 |
| Cilium VXLAN, Geneve | 8472, 6081 | 50 | 1450 |
5.k3s defaults and multi-NIC traps
- Pods
10.42.0.0/16, Services10.43.0.0/16, DNS10.43.0.10. - Backends :
vxlan,host-gw,wireguard-native,none. Others fail. - flannel binds the default-route NIC, not the node-ip one : nodes Ready, 100% Pod loss.
flannel.1dies with its uplink; k3s v1.36.0 to v1.37.1 (IPv4-only) then panic and restart.
node-ip: 172.31.254.11
flannel-iface: fabric0 # on every node
ip -d link show flannel.1:local <ip> dev <nic>flannel.must equal the InternalIP.alpha. coreos. com/ public-ip - Alert on a missing
flannel.1, k3s-agent restarts.
flannel-iface.1.Node Pod ranges
- KCM
--allocate-node-cidrs(off by default) carves /24 and /64 blocks. spec.podCIDRsis set once : no change, not even a second family.- A /16 of /24s holds 256 nodes; node 257 gets
CIDRNotAvailable, no range. - IPv6 : /64 per node needs a /48 or longer cluster CIDR.
kubectl get nodes -o custom-columns=N:.metadata. name,CIDR:. spec. podCIDRs
2.When the CNI owns IPAM
- In-tree ClusterCIDR was removed in v1.29.
node-ipam-controllerv0.3.0 brings it back as a CRD. - Calico and Cilium ignore
podCIDRs; block sizes are permanent. - Cilium's default
10.0.0.0/8can hit node networks.
| IPAM model | Picked by | Per node |
|---|---|---|
| RangeAllocator | KCM | /24, 253 IPs |
| ClusterCIDR | node-ipam-ctrl | perNodeHostBits |
| calico-ipam | Calico | /26 blocks |
| cluster-pool | cilium-operator | mask 24 |
3.Service ranges are objects
- ServiceCIDR and IPAddress : GA v1.33,
v1beta1gone in v1.37. Seekubectl get servicecidr,ipaddress. - Random picks use the upper band : hand-picked IPs (DNS
.10) go low. - Full :
failed to allocate a serviceIP ... range is full. - v1.37 : no IPv4 size limit. IPv6 shorter than /64 shows Ready, allocates nothing.
apiVersion: networking.k8s.io/v1 kind: ServiceCIDR metadata: name: newcidr1 spec: cidrs: ["10.96.0.0/24"]
The failed Service then got 10.96.0.173. Guard it : Admission.
4.Dual-stack end to end
- Both families on every component : apiserver, KCM, kubelet
--node-ip(first = primary), CNI. status.podIPsfollows the node's primary family, not the CNI (10.42.1.2, thenfd00:42:0:1::2). k3s : dual-stack only at creation.ipFamilyPolicy:SingleStack(default),PreferDualStack,RequireDualStack. Primary never changes.get svchides the second IP :kubectl describe svc <s> | grep -E '^IP'.
5.Symptoms, and second networks
- portmap : skip
KUBE-MARK-MASQ(absent in nftables mode); its nftables backend before v1.9.0 has CVE-2025-67499. - Multus sorts first (
00-multus.conf) and addsnet1. DRA reportsnetworkData(GA v1.37).
| Symptom | Check | Cause |
|---|---|---|
| no IP addresses available | files in / | leaks (no CNI GC) or too many Pods |
| cni0 already has an IP | did the node subnet change ? | forceAddress: false |
| subnet doesn't contain podCIDR | flannel Network vs KCM | kubeadm range not 10.244.0.0/16 |
| hostPort refused, no events | conflist portMappings ? | no portmap in the chain |
| incompatible CNI versions | CNI_ on the binary | plugins older than the conflist |
Networking internals
The Service Dataplane
1.A controller, not a proxy
A Service is kernel NAT rules, not a process. This picks up where the watch loops in Services stop (v1.37).
- Informers : Services (never headless ones), EndpointSlices, ServiceCIDRs and its own Node.
- Runner : syncs at least 1s apart, retries within 30s, full sync at least hourly.
- Partial sync : changed Services only. Over 1000 endpoints, iptables skips the hourly full sync.
2.Ports, health and metrics
:10256/healthz: 200 unless an update waited over 2 xsyncPeriod(60s) or the Node is being deleted./livezignores the Node.127.0.0.1:10249(loopback) :/metrics,/configz,/proxyMode. Trycurl -s 127..0. 0. 1:10249/ proxyMode - k3s binds healthz to 127.0.0.1, so LB probes on
<nodeIP>:10256fail.
| kubeproxy_... | Tells you |
|---|---|
| sync_full_ / sync_partial_ | Sync cost |
| network_programming_ | Slice change to rules |
| sync_proxy_rules_ | Staleness |
| ..._iptables_restore_ | Failed syncs |
3.The config that matters
- Unset
mode= iptables plus an error log line. TheKubeProxyDefaultModeevent never arrives : grep the logs forNo kube-proxy mode specified. - Sync settings come from the section matching
mode. Flags lose to--config, and an unknown field only logs. Lint with--write-config-to.
apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: nftables # never leave it unset clusterCIDR: 10.244.0.0/16 nftables: # not iptables: syncPeriod: 30s # retry, not resync
4.iptables mode, chain by chain
KUBE-SERVICES: one rule per Service port, walked in order. O(Services) per new connection.KUBE-SVC: rule i jumps with--probability 1/(N-i), the last always. Each endpoint gets 1/N.KUBE-SEPDNATs toPodIP:port. Marked packets (0x4000) are masqueraded on the way out.
| Hook | Jumps to (verdict) |
|---|---|
| nat PREROUTING, OUTPUT | KUBE-SERVICES (DNAT) |
| nat POSTROUTING | KUBE-POSTROUTING (SNAT) |
| filter FORWARD, OUTPUT | KUBE-SERVICES (REJECT) |
| filter INPUT, FORWARD | KUBE-EXTERNAL-SERVICES (DROP) |
Real chain names from iptables-save on a kind node (v1.37.1, observed 2026-10-07).
5.When rules vanish
- Only the mangle
KUBE-PROXY-CANARYis polled. If it goes, a full resync follows (observed : 18s). - Flushing only nat or filter goes unnoticed (observed : broken after 80s). Restart kube-proxy.
The mark prints as --set-xmark 0x4000/0x4000, probabilities as 0.50000000000. Grep names : iptables-save -t nat | grep -E 'KUBE-(
KUBE-SERVICES, takes a random jump and is DNATed.1.nftables mode
- Owns
table ip kube-proxyandtable ip6 kube-proxy(noinettable). Base chains :nat-prerouting,nat-output,nat-postrouting, filter chains. servicesmarks off-cluster ClusterIP traffic, then does one lookup :nft list map ip kube-proxy service-ips.
- A Service chain picks the endpoint in one
numgen random mod Nrule (since v1.36). One endpoint : a plaindnat to. Endpoint chains remain only for ClientIP affinity. - The
no-endpoint-*maps reject (no endpoints) or drop (none local). Never edit this table or the mark.
chain services {
ip daddr @cluster-ips ip saddr != 10.244.0.0/16 meta mark set meta mark | 0x00004000 ...
ip daddr . meta l4proto . th dport vmap @service-ips
ip daddr @nodeport-ips meta l4proto . th dport vmap @service-nodeports }
chain service-SUSTUHLJ-default/web/tcp/http {
meta l4proto tcp dnat ip to numgen random mod 2 map { 0 : 10.244.0.5 . 8080, 1 : 10.244.1.2 . 8080 } }
2.Why it scales, and the switch
- iptables : O(Services) per new connection and update, global lock. nftables : one lookup, updates O(changes).
- Changes : NodePorts on the primary IP only, none on
localhost(v1.37 alpha gate). Open the NodePort range yourself. Kernel 5.13+. - Switch : check the
kubeproxy_counters, setiptables_* mode: nftables. Old rules go away; mixed nodes are fine.
| Release | iptables | nftables | ipvs |
|---|---|---|---|
| v1.37 | default | GA | deprecated, gated |
| v1.39 | kubeadm rejects 5.4 and 5.10 kernels | ||
| v1.40 | not default | default | off |
| v1.43 | ok | ok | removed |
From v1.39 on : plans (KEP-5343 is provisional).
3.ipvs, while it lasts
- Service IPs sit as /32s on the dummy
kube-ipvs0. One IPVS virtual server per IP and port, schedulerrr, NAT. - Masquerade, NodePorts and filtering still use iptables, matched against 17 ipsets.
- Leaving ipvs for iptables keeps
kube-ipvs0: stop kube-proxy, then runkube-proxy --cleanup.
4.conntrack, the hidden half
- Later packets skip the nat table, so only new connections re-balance. gRPC and HTTP/2 stay put. Pins :
conntrack -L -d <clusterIP>. - UDP : stale entries are deleted after a sync (v1.37 : also with zero endpoints). TCP : never.
- Max = 32768 x CPUs, capped at 1,048,576 (v1.36), at least 131072. Check
conntrack -S. - Read-only sysctl (LXC) : v1.36 and v1.37.1 can fail to start. Set
maxPerCore: 0. - TCP established 24h, close-wait 1h. k3s leaves both to the kernel.
5.SNAT and the client IP
- SNAT :
masqueradeAll, off-cluster or host sources to a ClusterIP, hairpin, and all external traffic under eTP=Cluster. - MASQUERADE uses the egress address. "Local Pod" comes from
detectLocalMode(defaultClusterCIDR; noclusterCIDR= no-op).
| Path | Backend sees (observed) |
|---|---|
| Pod to ClusterIP | the client Pod IP |
| Pod to its own Service | SNAT, 10.244.0.1 |
| host to ClusterIP | node IP if remote, 10.244.0.1 if local |
| external, eTP=Cluster | a node address, even if local |
Test : wget -qO- http://<svc>/ (agnhost). eTP=Local : Services.
1.What kube-proxy reads
servingmirrors Pod Ready,terminatingcomes with deletion,ready= serving and not terminating.- Local =
nodeNameequals kube-proxy's node name, never by IP. A wronghostnameOverridemakes all endpoints remote. - 100 endpoints per slice by default (Controllers).
| Field or label | Effect |
|---|---|
kubernetes. | Required link |
addresses[0] | The only IP used |
conditions | ready, serving, terminating |
nodeName | Locality |
zone, hints | Topology |
2.Which endpoints a connection can reach
- Per policy : Ready endpoints, else serving and terminating ones. This holds for Cluster policy too (docs say Local only).
- Observed : a Pod with a 120s
preStopsleep, scaled to 0, kept taking new connections. - kube-proxy uses hints only if every Ready endpoint has one. Local policies win over hints.
PreferSameZonehas no overload guard : spread replicas (Scheduling). Values : Services.
3.Health checks for Local policies
- Past the eTP panel in Services : one listener per Service on
healthCheckNodePort, 200 with local Ready endpoints, else 503, plus headerX-Load-Balancing-Endpoint-Weight. - Probe :
curl -si http://<nodeIP>:<port>/. Terminating local Pods fail it, so the LB drains the node.healthz - iTP=Local with no local endpoint DROPs (timeout), not the REJECT the docs imply.
4.eBPF : Services without netfilter
- Verified programs run at hooks (XDP, tc, cgroup
connect4) and keep state in maps. - Cilium
kubeProxyReplacement: true: socket LB atconnect(), NodePorts per packet (snat,dsr,hybrid).
- Calico eBPF : connect-time LB for TCP only. NodePorts keep the client IP, even with eTP=Cluster.
- Traps : socket LB breaks Istio sidecars, KubeVirt, Kata, gVisor. tcpdump misses XDP.
| iptables | nftables | ipvs | Cilium KPR | Calico eBPF | |
|---|---|---|---|---|---|
| v1.37 | default | GA | deprecated | opt-in | opt-in |
| Dispatch | rule walk | one map | IPVS table | BPF map | BPF map |
| State | conntrack | conntrack | + IPVS | BPF maps | BPF maps |
| Kernel | - | 5.13 | - | 5.10 | 5.10 |
5.Removing kube-proxy
- kubeadm :
--skip-phases=addon/. Live cluster : delete the DaemonSet and ConfigMap, runkube-proxy kube-proxy --cleanup, setk8sServiceHost. - Cilium breaks open Service connections in the switch. Calico keeps TCP flows, drops VXLAN ones, wants ipvs gone first. k3s :
--disable-kube-proxy, or--kube-proxy-arg=proxy-mode=nftables. - Check :
cilium-dbg status --verbose | grep -A12 'KubeProxyReplacement Details'. CNI side : Pod Networking.
connect().Networking internals
Cluster DNS in Depth
1.Two halves
The kubelet writes each Pod's resolver file once, at sandbox creation (GetPodDNS), from clusterDNS, clusterDomain and resolvConf. CoreDNS (Deployment coredns, label k8s-app=kube-dns) answers from API watches behind the Service still named kube-dns : the 10th Service IP on kubeadm (10.96.0.10), 10.43.0.10 on k3s.
2.What the kubelet writes
- ClusterFirst options are exactly
ndots:5. Nodeedns0,timeout,attemptsare dropped. - hostNetwork : silently Default (kind), use ClusterFirstWithHostNet. No
clusterDNS: Default +MissingClusterDNS. dnsConfigis appended, options merged by name (Pod wins).None:dnsConfigalone.
| dnsPolicy | server | search | opts |
|---|---|---|---|
| ClusterFirst | clusterDNS | 3 cluster + node | ndots:5 |
| + hostNetwork | node file | node file | node |
| ...WithHostNet | clusterDNS | 3 cluster + node | ndots:5 |
| Default | resolvConf | same file | file |
Over 3 servers, 32 searches or 2048 chars : the API rejects, the kubelet trims the merged file (DNSConfigForming). See it : kubectl exec <pod> -- cat /etc/resolv.conf. Basics : Namespaces.
3.The ndots:5 walk
- Under 5 dots : all suffixes, then as typed. A 4-dot FQDN too.
- musl always sends A and AAAA.
- glibc drops AAAA only with no IPv6 but
::1. Link-localfe80::counts : 8 queries for 3 suffixes on kind. - Plan for 2 x (suffixes + 1) queries.
4.glibc vs musl
| Behaviour (seen in the lab) | glibc | musl (Alpine) |
|---|---|---|
| name with dots >= ndots | as typed, then search | as typed only |
| NODATA or SERVFAIL on a suffix | keeps walking | stops and fails |
. in the search list | as typed at that spot | skipped |
| search line over about 255 chars | used | whole line ignored |
| TCP fallback on truncation | yes | musl 1.2.4+ (Alpine 3.18+) |
The docs' dnsutils Pod runs musl 1.1.24 (no TCP fallback). dig and nslookup use their own resolver, and dig skips the search list without +search. Test in the app's image : kubectl exec <pod> -- getent ahosts <name>.
5.Cutting the cost
- Trailing dot : one name, any libc.
- ndots: "2" : names with 2+ dots go upstream first (seen on kind). musl never searches them.
- autopath : 1 round trip. Needs
pods verified, Pod IP. - Searches may hold
_and a lone.(GA v1.34).
dnsConfig:
options:
- { name: ndots, value: "2" }
- name: single-request-reopen
- { name: timeout, value: "2" }
# options timeout:2 ndots:2 single-request-reopen
1.Records beyond the basics
| Query (kind) | Answer |
|---|---|
TXT dns-version.<zone> | "1.1.0", the spec version |
| AAAA, IPv4-only Service | NOERROR, no answer |
dig -x 10.96.0.10 | kube-dns. |
| headless, none ready | NXDOMAIN |
| ExternalName | CNAME + the target's A records |
Basic records : Services, StatefulSets.
hostname+subdomain= headless Service, same ns.- Ready Pods only.
publishNotReadyAddressesadds unready and terminating ones. setHostnameAsFQDN: 64 chars max.hostnameOverride(GA v1.37) : hostname only, not DNS.
2.pods modes, and a captured name
disabled(default) :a-b-c-d.<ns>.NXDOMAIN.pod insecure(kubeadm, k3s) : no Pod check,1-2-3-4.gave 1.2.3.4. Nodefault. pod *.podcerts.
verified: Pod must exist. Watches all Pods (~4x memory, old data).- Capture : Service
githubin namespacecomanswersgithub.com(derived). Guard namespaces.
3.A query through CoreDNS
- Order is compiled (
plugin.cfg), not the Corefile's. ttl(plugin 5, kubeadm 30) is also the negative TTL.- k3s caches cluster names : new headless names NXDOMAIN up to 30s.
- Over
max_concurrent: REFUSED. SERVFAIL retried only withfailover.
4.The default Corefile
.:53 { # kubeadm v1.37.1
health { # k3s: plain health
lameduck 5s
}
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30 # k3s: no ttl (5)
}
forward . /etc/resolv.conf { max_concurrent 1000 }
cache 30 { # k3s: plain cache 30
disable success cluster.local
disable denial cluster.local
}
} # + errors ready prometheus :9153 loop reload loadbalance
- health:8080. lameduck : stays 200, exit delayed 5s
- ready:8181. 503 until plugins sync,
Shutting downin lameduck - loopHINFO self-probe for 30s
- reloadSHA512 check every 30s +/- 15s
The debug page's Corefile fails : 1.14.7 rejects upstream. k3s adds hosts NodeHosts and import *.override. kubeadm : 2 replicas, CoreDNS v1.14.6. k3s v1.37.1+k3s1 : 1 replica, 1.14.7.
5.Changing it safely
- kubeadm : edit ConfigMap
coredns, allow 2 minutes (kind : 59s and 84s). - A bad config keeps the old one, but restarts crash. Check :
coredns -conf ./Corefile. - Now :
rollout restart deploy/coredns(kube-system). - Stub domain : own block, or
forward consul.localfirst.
corednsis rewritten every start.*.serveradds blocks.- A 2nd
hosts,cache,errors,prometheusfails. Lameduckhealthis fine.
kind: ConfigMap # coredns-custom, kube-system data: example-com.override: | # inside .:53 forward example.com 10.0.0.1
1.Enough CoreDNS
- cluster-proportional-autoscaler (not in kubeadm) :
max(with the docs' values. 40 nodes x 8 cores : 3.ceil( cores/ 256), ceil(nodes/16)) - Re-reads its ConfigMap every 10s. Tag
:v1.11.0, not:1.11.0.
- Rollouts : 1 replica without lameduck lost 1 of 450 queries per rollout (kind). With
lameduck 5s: none. - k3s ships 1 replica, no lameduck, no PodDisruptionBudget.
2.NodeLocal DNSCache
- hostNetwork DaemonSet
node-local-dns. Its dummynodelocaldnsholds 169.254.20.10 and the kube-dns IP. - 12 rules per IP : 8 raw NOTRACK (6 DNS, 2 health port), 4 filter ACCEPT.
- External names skip CoreDNS : no stub domains, rewrites or k3s overrides.
- Create
kube-dns-upstreamfirst. An OOM kill leaves the rules : no DNS there. - Moved to
kubernetes-sigs/. v1.37.1 pins 1.26.8.node-local-dns
3.NodeLocal by dataplane, and DNS egress
| Mode | How Pods reach it | Kubelet change | If the cache dies |
|---|---|---|---|
| kube-proxy iptables | NOTRACK on the bound kube-dns IP | none | clean exit falls back |
| ipvs (deprecated) | Pods use 169.254.20.10 | clusterDNS: [169.254.20.10] | no fallback |
| nftables | undocumented; iptables install worked (1 kind node) | none | fell back on clean exit |
| Cilium | CiliumLocalRedirectPolicy on kube-dns | none | kube-dns backends |
169.254.20.10/32) stopped matching on kindnet. ipBlock: 10.96.0.10/32 fixed it. Test your CNI. See Network Policy.4.Two failures up close
- 5s race : parallel A and AAAA both pass DNAT unconfirmed, one is dropped at confirm.
- Fixed by kernels v4.19 to v5.9. IPVS is hit too. See
conntrack -Sinsert_failed. Basics : Service Dataplane. - Loop : node
127.0.0.53makes CoreDNS forward to itself. kubeadm fixesresolvConfonly if unset (leaked intokubelet-configbefore v1.37.1).
5.Other failures, and the playbook
| Symptom | Cause | Fix |
|---|---|---|
| External SERVFAIL, or REFUSED under load | upstreams down, or max_concurrent | fix upstreams; scale |
| New headless name NXDOMAIN for ~30s | cluster domain cached | disable success/denial |
| Cluster names SERVFAIL | no EndpointSlice RBAC, or first 5s | describe clusterrole system:coredns |
| One node has no DNS after a NodeLocal OOM | rules left behind | raise its memory limit |
Playbook : Pod resolv.conf, events, getent ahosts, CoreDNS Pods, EndpointSlice and logs, system:coredns, log on (2 min), metrics, conntrack -S.
Metrics : coredns_, coredns_ (old coredns_ gone since 1.11), hit ratio = cache_hits_total / cache_.
Networking internals
Network Policy in Depth
1.A request, not a firewall
- No status : the API server defaults, validates, stores. Alpha
statusleft in v1.28. - Not in conformance : the 49 enforcement e2e tests are feature-tagged.
- No enforcer (plain flannel) : every policy is stored and silently ignored.
- Agents : kube-router (k3s), Calico Felix, cilium-agent, antrea-agent, kube-network-policies.
2.Defaulting traps
- Egress-only, no
policyTypes: all ingress is cut too. kubectl describe netpol: "isolated for ingress".- No
podSelector(optional, v1.34+) : every Pod.
v1.37. Basics : Namespaces and Ingress, Architecture.
| You wrote | Stored (seen on v1.37.0) |
|---|---|
no policyTypes, no rules | [Ingress] |
| egress rules only | [Ingress, Egress] |
no podSelector | podSelector: {} |
a port, no protocol | protocol: TCP |
3.After DNAT : Pod ports
- Calico, Cilium, Antrea, kube-network-policies check after Service DNAT. Docs are silent.
- Name the Pod's
targetPortor port name, and the backend Pods. - Named ports resolve per destination Pod.
endPortneeds a numericport. - Only TCP, UDP, SCTP.
ICMPis rejected.
# Service api : port 80 to containerPort "http" (8080) ports: [{name: http, port: 80, targetPort: http}] # NetworkPolicy api-from-frontend ingress: - from: [{podSelector: {matchLabels: {app: frontend}}}] ports: [{protocol: TCP, port: http}] # or 8080. 'port: 80' never matches
4.Where NAT and IP rules blur
ipBlockis for external CIDRs. Pod IPs in it : e2e expects a match, as kind does. Cilium ignores them by default.- Strict CIDRs since v1.36 (Beta) :
10.0.0.5/24, leading zeros, IPv4-mapped IPv6 fail on create. - Stored values ratchet with a warning, so a fresh copy fails. Test with
kubectl apply --dry-run=server --validate=strict.
| Traffic | The policy sees |
|---|---|
| NodePort, LB with eTP=Cluster | a node IP (SNAT). Use eTP=Local or allow node IPs |
| probes, local host processes | the local node : ingress allowed (kind : root's sockets only) |
| a hostNetwork Pod | the node IP : undefined |
| NodeLocal DNSCache | hostNetwork : only an ipBlock matches |
5.The edges the API leaves open
| Edge | The API says | Implementations |
|---|---|---|
| ICMP | undefined | kube-router accepts echo, unreachable, time-exceeded. Calico can name ICMP. Cilium blocks unrelated ICMP under L4 rules |
| Pod to own node | godoc : ingress from it. Docs : "to and from" | kube-network-policies enforces Pod-to-host. Felix default DROP, calico.yaml ACCEPT |
| Pod to itself via its Service | cannot block itself | the upstream e2e test expects it blocked |
- Undefined too : open connections after a change, new Pods before programming (windows). Not expressible : explicit deny, logging, Service or FQDN peers, L7, a cluster-wide default.
policyTypes defaulting, and treat ICMP and node traffic as undefined.1.ClusterNetworkPolicy (v1alpha2)
- network-policy-api v0.2.0 (2026-04-21) : one cluster-scoped CRD,
cnp. - Replaces ANP, BANP.
Allowis nowAccept,portsnowprotocols. v1beta1 PR #411 open. - tier Admin or Baseline. priority 0 to 1000, lower wins, ties ambiguous.
tcp: {}is rejected. Any port :range: {start: 1, end: 65535}.
| Peer | Ingress | Egress | Chan. |
|---|---|---|---|
namespaces | yes | yes | std |
pods | yes | yes | std |
networks | no | yes, hits Pods too | std |
nodes, domainNames | no | yes | exp |
- Subject : namespaces or pods, not hostNetwork. 25 rules max.
- Install from the
v0.2.0tag :mainchanged the schema. kubectl get cnp: Tier, Priority. (channels)
2.Evaluation order
Accept,Denyare final.Passskips the rest of Admin.Acceptcovers replies and related ICMP. An egressAcceptkeeps the far end's ingress rules.- DNS guarantee : an Admin
Accepttok8s-app: kube-dnsPods, port 53. Upstream user-story-2 usesapp: kube-dns, a wrong label.
3.Who implements what (checked 2026-10-07)
| Implementation | CNP v1alpha2 | Notes |
|---|---|---|
| kube-network-policies v1.1.2 | yes, v1.0.0+ | reference. nftables + NFQUEUE, fails open. kind v0.33.0 embeds an older copy : no CNP |
| Calico v3.33.0 | yes, v3.32+ | tiers kube-admin, kube-baseline. ANP, BANP dropped. A bad peer fails the rule |
| Cilium 1.20.2 | opt-in | k8sClusterNetworkPolicy.. Conformance needs [pods, nodes] CIDR match |
| Antrea v2.7.0 | alpha gate, off | gate ClusterNetworkPolicy. v1alpha1 deprecated |
| OVN-K8s, Kube-OVN | OVN-K8s v1alpha1 | Kube-OVN : pre-v0.2.0 build, v0.2.0 on master |
The upstream list (21 April 2026) omits Cilium 1.20 and Antrea 2.7. No conformance reports from them or Calico.
4.k3s : kube-router, one packet
- k3s v1.37.1+k3s1 : fork v2.6.3-k3s1, iptables + ipset, no CNP.
--disable-network-policyleaves its rules behind.- Unmarked : NFLOG group 100, then REJECT. Ping works.
- New Pods fail open : no
--netpol-default-deny. - Chains :
iptables -t filter -S | grep KUBE-.
5.Calico and Cilium
- Tiers by
order. NetworkPolicies sit indefaultasknp..default.<name> - Allow, Deny, Pass, Log. Follow Log with Allow, or the tier's end drop wins.
cali-pi-/cali-po-chains, or a tc hook in eBPF mode.
- Labels become identities. Per endpoint : a BPF map of (identity, port, proto), 16k.
ingressDeny/egressDenybeat every allow.toFQDNsneeds arules.dnsallow.- Dump :
cilium-dbg bpf policy get <ep-id>.
Cilium 1.20 may balance NodePort at the client (Service Dataplane). Antrea has two CNP kinds.
1.Stored is not enforced
- Only a probe seen to fail proves a deny.
- Counts lie :
KUBE-POD-FWcan read 0 on a node withoutiptables;nftshowed 70. - An untested deny is a guess : probe it from a Pod that should fail.
The official task expects wget: download timed out; k3s refuses at once.
| Dataplane | Client sees | agnhost |
|---|---|---|
| kube-router, k3s | refused at once | REFUSED |
| Calico, kube-network-policies | timeout | TIMEOUT |
| Cilium L3/L4 | timeout (opt-in ICMP on egress) | TIMEOUT |
| Cilium L7 | HTTP 403, DNS REFUSED | connects |
2.A per-node probe for your cluster
- Server Pods on each node. Default-deny ingress, only label
allowedpasses. - A client DaemonSet probes from every node : all must fail.
- Control 1 : one allowed client must pass, or the network is just broken.
- Control 2 : agent off on one node, FAIL there only.
- No ping. Self-health on
127.0.0.1: own-Service calls may be blocked.
# a sketch, not run. Namespace np-probe : Service server, client DaemonSet (agnhost) for p in $(kubectl -n np-probe get pods -l app=client -o name); do n=$(kubectl -n np-probe get "$p" -o jsonpath='{.spec.nodeName}') if kubectl -n np-probe exec "$p" -- /agnhost connect server.np-probe.svc.cluster.local:80 \ --timeout=3s --protocol=tcp 2>err.txt then echo "FAIL $n: denied connection worked"; else echo "ok $n: $(cat err.txt)"; fi done # ok lines read TIMEOUT or REFUSED
Upstream e2e diffs an expected x/y/z truth table against agnhost results. No expected X, no proof.
3.Watching verdicts
| Dataplane | Where | Read it with |
|---|---|---|
| k3s kube-router | NFLOG 100. Empty podSelector not logged (k3s #8008) | tcpdump -ni nflog:100 |
| Cilium | Hubble (relay off). 133 no allow, 181 deny rule | hubble observe --verdict DROPPED |
| Calico | Log rules : calico-packet. Whisker preview | journalctl -k | grep calico-packet |
Antrea : np.log. kube-network-policies : -v=2 (not on kind : no flags). policy-assistant simulates from YAML (v1, v1alpha1 only).
4.Start-up and change windows
| Implementation | New Pod before programming | Open connection after a new deny |
|---|---|---|
| kube-router, k3s | fails open (no --netpol-default-deny) | survives |
| Calico | opt-in hold, linux (default 0) | survives until reconnect |
| Cilium | reserved:init, allowed unless selected | not documented |
| kube-network-policies | NRI learns Pod IPs early | cut (--strict-mode, default) |
| kind (its built-in copy) | open until its node lists it (0.7 s) | survives (no strict mode) |
The docs suggest an init container that waits for its dependencies, and advise against changing policies under live connections.
5.Troubleshooting checklist
- Anything enforcing ? flannel alone is not.
- Pod selected ?
kubectl describe netpol. - Both ends allow ? DNS (Cluster DNS) ?
- Pod port named ? Source NATed ?
- Higher tier ?
kubectl get cnp, Calico tiers. - Read the verdict, then probe every node.
Storage internals
CSI Drivers and the Volume Lifecycle
1.What a CSI driver is
- A gRPC server on a UNIX socket (
CSI_ENDPOINT, ends in.sock). - Identity (always), Controller (provision, attach, expand, snapshot), Node (stage, publish, stats). Also GroupController and alpha SnapshotMetadata.
- Capabilities switch steps off, down to NodePublish and NodeUnpublish only. No capability says "block".
| Capability | Without it |
|---|---|
| CREATE_DELETE_VOLUME | provisioner refuses |
| PUBLISH_UNPUBLISH_VOLUME | "trivial handler" |
| STAGE_UNSTAGE_VOLUME | kubelet skips NodeStage |
| VOLUME_MOUNT_GROUP | kubelet applies fsGroup |
| EXPAND, MODIFY, CLONE ... | no resize, VAC, copies |
Plus plugin CONTROLLER_SERVICE to provision. v1.37 vendors a commit just before spec v1.13.0.
2.Two halves, two callers
- Controller plugin : Deployment or StatefulSet, leader election. More replicas add availability, not throughput. Sidecars share the socket in an
emptyDir. - Node plugin : privileged DaemonSet,
Bidirectionalpropagation on/andvar/ lib/ kubelet/ pods .../plugins(staging). Privileged Pod Security level.
3.The sidecars
| Sidecar | Calls | Newest |
|---|---|---|
| external-provisioner | Create / Delete | v6.3.0 |
| external-attacher | Publish / Unpublish | v4.13.0 |
| external-resizer | Expand / Modify | v2.3.0 |
| csi-snapshotter | CreateSnapshot | v8.6.0 |
| external-health-monitor | VolumeHealth | v0.19.0 |
| node-driver-registrar | GetPluginInfo only | v2.18.0 |
| livenessprobe | Probe | v2.20.0 |
Each controller sidecar is a reconcile loop with leader election. Checked 2026-10-07 : provisioner --timeout is 10s in code (README : 15s), set it for slow backends. Resizer image tops out at v2.2.1. The health monitor exits unless the driver advertises health.
4.Registration on a node
- Registrar :
GetPluginInfoonly, then a socket inplugins_registry/. Kubelet (inotify) :GetInfo,NodeGetInfo(2m), then thecsi.annotation.volume. kubernetes. io/ nodeid --kubelet-registration-path= host path of the driver socket. Wrong :not found in the list of registered....- Same watcher as DRA, no
maxSurgeupgrades (all send"1.0.0").
5.Three API objects
- CSIDriver : 11 fields in v1.37, two immutable (path misspelled
spec.). No object is not neutral : attach stays on, no Pod info, tokens or inline.attachedRequired - CSINode : by the kubelet, never the driver.
nodeID,topologyKeys(keys only),allocatable.count. - VolumeAttachment :
csi-<sha256(.handle+driver+node)> specimmutable,statusonly by the attacher.
| CSIDriver field | Default |
|---|---|
| attachRequired | true, immutable |
| volumeLifecycleModes | [Persistent], immutable |
| podInfoOnMount | false |
| tokenRequests | none |
| requiresRepublish | false |
| serviceAccountTokenInSecrets | unset |
The other five : Volume Topology, Volume Operations.
1.Provision : PVC to PV
- PV controller :
ExternalProvisioning. WaitForFirstConsumer waits for selected-node. - external-provisioner : CreateVolume
pvc-<UID>, minus reserved keys; writes the PV, finalizer. - No secret : provisioning fails, deletion does not.
| Reserved key | Becomes | Read by |
|---|---|---|
| fstype | PV fsType | kubelet |
| provisioner-secret-* | Create, Delete | provisioner |
| *-publish-, node-stage-* | PV *SecretRef | attacher, kubelet |
| *-expand-, *-modify-* | refs, annotation | resizer, kubelet |
Prefix csi.. Secrets : -name / -namespace, templates like ${pvc.namespace}.
2.Attach : the VA handshake
- The attach-detach controller (ADC) creates VA
csi-<sha256>, waits 2m forstatus.attached. - external-attacher : finalizer, node ID from CSINode, ControllerPublishVolume,
attachmentMetadata. - Immutable. v1.36+ : flipping to false cleans old VAs.
| attachRequired | true (default) | false |
|---|---|---|
| VolumeAttachment | per volume per node | never |
| driver can publish | ControllerPublish | skipped (wrong) |
| driver cannot | trivial handler, no RPC | nothing |
| attacher missing | timed out after 2m | fine |
| publish_context | VA attachmentMetadata | nil |
3.The whole lifecycle
4.Stage and publish on the node
- MountDevice :
globalmount+vol_data.json, NodeStage. No staging : directory only. - SetUp : Pod directory +
vol_data.json, driver makesmount, then fsGroup. Canceled,DeadlineExceeded,Unavailable,ResourceExhausted,Aborted: uncertain, kept. Others : removed.
| Path (/var/lib/kubelet/...) | Written by |
|---|---|
plugins_ | registrar |
plugins/<driver>/ | driver |
plugins/ | kubelet |
pods/<podUID>/ | kubelet, driver |
Reconcile every 100ms. Only the first access mode is sent (RWO as 7 if advertised). Before v1.24 : .../.
5.Teardown and kubelet restart
- Unpublish, Unstage (last Pod on the node),
volumesInUsecleared (0 to 10s), VA deleted, ControllerUnpublish (observed : 34.327, 34.428, 35.118). Why : Volume Operations. - DeleteVolume is retried : idempotent. A VA left :
VolumeFailedDelete ... still attached. Reconstruction (GA v1.30) : a restarted kubelet rebuilds fromvol_data.jsonand re-issues Stage and Publish.
6.Commands
| Command | Shows |
|---|---|
VA=csi-$(printf '%s%s%s' "$HANDLE" "$DRIVER" "$NODE" | sha256sum | cut -d' ' -f1) | VA name (a hash) |
printf '%s' "$HANDLE" | sha256sum | Staging dir (a hash) |
kubectl get volumeattachment <va> -o jsonpath='{. | Error, finalizers |
findmnt -o TARGET,SOURCE,FSTYPE,PROPAGATION | grep kubernetes.io | Real mounts (hostpath : no globalmount) |
journalctl -u kubelet | grep -E 'MountVolume.( | Node steps |
1.Volumes that skip steps
Raw block : volumeMode: Block, devicePath, no fsType; paths under .../, held by a loop device.
| CSI persistent | CSI inline | Generic ephemeral | Image volume | In-tree, migrated | |
|---|---|---|---|---|---|
| API field | PV spec.csi | volumes[].csi | volumes[].ephemeral | volumes[].image | old PV fields |
| Parameters | admin | Pod author | admin | Pod author | admin |
| CSI calls | all | Publish only | all | none | all, translated |
| VA | if attachRequired | never | if attachRequired | never | if needed |
| Capacity-aware | yes | no | yes | n/a | yes |
2.Inline, generic and image volumes
- Inline : CSIDriver lists Ephemeral (Volumes).
volume_id=csi-<sha256(.podUID+name)> podInfoOnMount: five keys (pod name, namespace, uid,serviceAccount.,name ephemeral) to NodePublish only. - Pod authors set
volumeAttributes; restricted Pod Security allowscsi. Guard with admission. - Generic : PVC
<pod>-<volume>, no PVC create right needed. Names collide :pod is not owner.
3.Credentials into drivers
4.Migration and what comes next
- Ten in-tree plugins were removed (1.26 to 1.36). Seven types need their CSI driver (Volumes). CephFS, RBD, GlusterFS have no translator : move by hand first. FlexVolume : deprecated, still there.
- Horizon : all-in-one sidecars (KEP-4958, alpha target v1.38); COSI v1alpha2 only on
main(tag v0.2.2).
5.Debugging and commands
Attach and mount symptoms : Volume Operations. -v=5 driver logs are sanitized (not hostpath's gRPCCall).
| Symptom | Look at | Cause and fix |
|---|---|---|
PVC Pending, only ExternalProvisioning | provisioner, leader | provisioner down, or another driver named |
does not support snapshot restore | dataSource | no CREATE_DELETE_SNAPSHOT (clones: CLONE_VOLUME), no RPC |
timed out waiting for external-attacher | VAs, attacher pods | deploy one, or recreate with attachRequired false |
not found in the list of registered | registrar logs | no node plugin, wrong path |
VA CSINode ... does not contain driver | kubectl get csinode | not registered on that node |
volume mode "Ephemeral" not supported | volumeLifecycleModes | inline volume, Persistent-only driver |
| Command | Shows |
|---|---|
kubectl logs <controller-pod> -c csi-attacher | Trivial sync started |
kubectl logs <node-pod> -c node-driver-registrar | GetInfo, NotifyRegistrationStatus |
ls -l / | Both sockets |
find /var/lib/kubelet -name vol_data.json -exec sh -c 'echo {}; cat {}' \; | Kubelet bookkeeping |
kubectl get --raw / | Kubelet CSI metrics |
kubectl delete csidriver <d> && kubectl apply -f csidriver.yaml | Only way to change immutables |
Storage internals
Volume Topology and Scheduling
1.Where a volume can live
- Topology = the PV spec.nodeAffinity. Terms are ORed, expressions in a term ANDed.
- CSINode topologyKeys are key names. The values are Node labels.
- Immediate is the default. The mode is immutable.
| Mode | Immediate | WFFC |
|---|---|---|
| Volume made | PVC created | Pod scheduled |
| Zone from | provisioner | scheduler's node |
| Capacity | never | if storageCapacity |
| Pod, unbound | Pending | normal |
| Risk | full zone | stale selected-node |
2.VolumeBinding in the framework
- A missing class or a pre-bound claim counts as immediate.
bindTimeoutSeconds: 0fails at once, not "no wait". The waiting node shows as NOMINATED NODE.- UnschedulableAndUnresolvable : evicting cannot help, so no preemption (Advanced Scheduling).
3.The handshake, end to end
PVC events : WaitForFirstConsumer, WaitForPodScheduled, ExternalProvisioning. Basics : Volumes and Storage. A nodeName pin skips the scheduler : a bound PV fails MountVolume.NodeAffinity check failed, a WFFC claim never binds.
4.What CreateVolume receives
- Plain WFFC : requisite is the aggregated topology, every Node with the key labels, driver or not.
- allowedTopologies keys must be driver keys, or ... is not in requisite.
- Node-local storage : --strict-topology.
| Node in zone b | requisite | preferred |
|---|---|---|
| WFFC, plain | a, b, c | b, c, a |
| WFFC, strict | b | b |
| WFFC, allowed a, b | a, b | b, a |
| Immediate | a, b, c | hash(set) + ordinal |
| Immediate, topology off | none | none |
5.Provisioning fails on the node
- Only RESOURCE_EXHAUSTED (or an uncached CSINode NotFound) frees the Pod.
- v6.0.0 and v6.1.0 could retry in the wrong zone : use v6.0.1, v6.1.1, v6.2.0+.
| CreateVolume | Provisioner | Pod |
|---|---|---|
| RESOURCE_EXHAUSTED | drops the annotation | rescheduled |
| INVALID_ARGUMENT | 5m slow retry | pinned |
| anything else | same node, backoff | pinned, 600s |
6.Commands
| Command | Shows |
|---|---|
kubectl get pvc <pvc> -o jsonpath='{.metadata.annotations. | Node picked (empty if static) |
kubectl get pod <pod> -o jsonpath='{.status. | Current verdict (events lag) |
kubectl get events --field-selector involvedObject. | PVC events, ProvisioningFailed |
kubectl describe pv <pv> | Node Affinity, Term 0 |
1.Storage capacity tracking
- Opt in : CSIDriver storageCapacity: true and provisioner
--enable-capacity. Objects land in the provisioner's namespace. - Only new volumes of WFFC classes are checked. No match : did not have enough free storage.
A hint only, no bookkeeping, unlike Dynamic Resource Allocation.
2.Why the numbers are only hints
- Each claim checked alone, new volumes not subtracted, data up to 1m old.
- Flag true before any object exists blocks every node : ship false, then flip.
- Matched by class name only : write access to csistoragecapacities steers scheduling (RBAC).
| Field | Rule | Used by |
|---|---|---|
| nodeTopology | nil = none, {} = all | Filter |
| storageClassName | name only | Filter |
| capacity | available_capacity | Filter, Score |
| maximumVolumeSize | optional | Filter first |
The first two are immutable. Flag turned off : PVC Pods wait for the 5 min flush.
3.Storage scoring in v1.37
- StorageCapacityScoring : beta and on.
- Dynamic : most free space wins. Static : the node whose smallest fitting PV is largest wins.
- Weight 1. Gate off with a
shapeset : kube-scheduler will not start.
| Claim, space | % | Def. | Inv. |
|---|---|---|---|
| 10Gi, 100Gi | 10 | 90 | 10 |
| 10Gi, 50Gi | 20 | 80 | 20 |
| 50Gi, fit 200Gi | 25 | 75 | - |
| 50Gi, fit 100Gi | 50 | 50 | - |
| 50Gi, PVs 60 + 500Gi | 83 | 17 | - |
4.Attach slots per node
- NodeVolumeLimits checks unique volumes against CSINode allocatable.count (in-tree EBSLimits and co : removed v1.32). Over : exceed max volume count, preemption may run.
- nodeAllocatableUpdatePeriodSeconds (>= 10, GA v1.36) and external-attacher v4.10.0+ : a code 8 attach fails the Pod for a new one.
5.Nodes without the driver
- VolumeLimitScaling (beta, on) : the flag rejects driverless nodes : <driver> CSI driver is not installed on the node.
- Only with CA
--enable-csi-node-aware-scheduling=true(1.35.0+, default false) or no autoscaler. Karpenter : out of scope.
kind: CSIDriver # name: ebs.csi.aws.com spec: attachRequired: true nodeAllocatableUpdatePeriodSeconds: 60 preventPodSchedulingIfMissing: true # CA flag first, or no autoscaler
6.Commands
| Command | Shows |
|---|---|
kubectl describe csinode <node> | Node ID, count, topology keys |
kubectl get csidriver <d> -o jsonpath='{.spec.preventPodSchedulingIfMissing} {.spec. | The two fields get hides |
kubectl get csistoragecapacities -A -o custom-columns='SC:.storageClassName, | What the scheduler matches |
kubectl get volumeattachments -o custom-columns='NODE:.spec.nodeName, | Code 8 = RESOURCE_EXHAUSTED |
1.Who says no, and can preemption help ?
- Each node reports only its first failing filter, so one reason can hide another. Alerts on had volume node affinity conflict (renamed v1.33) are silent.
| Plugin | Reason, node(s) ... | Status | Preempt ? |
|---|---|---|---|
| VolumeBinding | didn't match PersistentVolume's node affinity, didn't find available persistent volumes to bind, did not have enough free storage. PreFilter : pod has unbound immediate PersistentVolumeClaims | ...Unresolvable | never |
| VolumeZone | had no available volume zone (legacy PV zone labels only) | ...Unresolvable | never |
| NodeVolumeLimits | exceed max volume count, <driver> CSI driver is not installed | Unschedulable | possible |
| VolumeRestrictions | RWOP claim already in-use by another pod | Unschedulable | possible |
...Unresolvable = UnschedulableAndUnresolvable, also given by the last two for a missing PVC.
2.Reading one FailedScheduling message
Format : Advanced Scheduling. A PreFilter failure gives one reason. A trailing . not found in v1.37 is DRA noise.
3.Node-local storage
- local-static-provisioner v2.9.0 : one PV per disk, can drop a startup taint once disks are found.
- Node-local CSI (TopoLVM, host-path) : strict topology, provisioner central or per node.
- Node deleted : PV stays, Pod Pending. Same name reused : wrong disk.
| local-static | Node-local CSI | |
|---|---|---|
| Volumes | pre-made disks | carved per claim |
| Key | kubernetes.io/hostname | driver key |
| Class | no-provisioner + WFFC | driver + WFFC |
| Scheduler sees | PV sizes | CSIStorageCapacity |
| v1.37 score | largest best fit | most free space |
| Node loss | node-cleanup (opt-in) | force-delete PV |
4.A zone dies under a StatefulSet
- A bound replica fails PV affinity everywhere, spare PVs or not. Dropping its PVC still hits zone spread unless nodeTaintsPolicy: Honor (GA v1.33) skips tainted nodes. Old Pod gone first : Volume Operations.
- Or restore (Snapshots, Clones and Backup), or go regional, widen PV affinity (alpha MutablePVNodeAffinity).
5.Autoscalers and volume zones
- cluster-autoscaler : group per zone. Per-node CSIStorageCapacity blocks scale-up : add one for
template-node=true. - Karpenter : adds a bound PV's zone, picks one allowedTopologies zone at random, no region topology (Karpenter).
- Attach limits : CA
--enable-csi-node-aware-scheduling. Karpenter is out of KEP-5030's scope.
6.Commands
| Command | Shows |
|---|---|
kubectl get events --field-selector involvedObject. | Reason history |
kubectl get pv -o custom-columns='PV:.metadata.name,KEYS:.spec.nodeAffinity. | PV topology keys |
kubectl delete pv <pv> --wait=false --grace-period=0 --force, then clear its finalizers | Drop a dead node's PV |
kubectl -n kube-system logs <scheduler> | grep 'Plugin scored node for pod' | Scores (--v=10) |
Storage internals
Volume Operations and Failure Modes
1.Who writes what
- Basics : Volumes and Storage. Gate :
PersistentVolumeClaimResizeadmission (on). Its only dynamically provisioned pvc can be resized misleads : static PVs on an expandable class grow. - CSI : external-resizer does the controller step and status, the kubelet the node step.
2.Expansion, step by step
- Resizer :
allocatedResources, conditionResizing, then ControllerExpandVolume (10s timeout). Node step needed : conditionFileSystemResizePending. - Offline-only driver : FAILED_PRECONDITION while published. The resizer waits for zero Pods.
- An unused PVC never finishes : Waiting for user to (re-)start a pod.
3.Status values
Exactly five are accepted. The API reference still names ControllerResizeFailed and NodeResizeFailed : both are rejected.
| Value | By | Means |
|---|---|---|
| ControllerResizeInProgress | resizer | started |
| ControllerResizeInfeasible | resizer | driver refused |
| NodeResizePending | resizer | grow the FS |
| NodeResizeInProgress | kubelet | node call running |
| NodeResizeInfeasible | kubelet | node refused |
Infeasible : InvalidArgument, OutOfRange, NotFound. Timeouts : still running.
4.Undo a typo (GA v1.34)
- Lower to anything strictly above
status.capacity. Quota (max of spec and allocated) drops with it. - No help while
ControllerResizeInProgressholds, or once only the node step failed. - Editing a PV's capacity cancels the resize. Raising the PVC to match then fakes a grown backend.
$ kubectl patch pvc data ... 1000Gi 1000Gi ControllerResizeInfeasible 10Gi $ kubectl patch pvc data ... 100Gi # ok $ ... 10Gi # Forbidden: can not be less
5.VolumeAttributesClass
- Edit IOPS of a Bound PVC. GA v1.34,
v1beta1fails on v1.37. - Immutable classes, no default,
vac-protectionon each. - Cancel = old class back.
get pvc: desired,get pv: applied. - A VAC quota checks create and spec. Status can pass
hard.
| Expansion | VAC modify | |
|---|---|---|
| Spec field | requests.storage | VAC name |
| Allowed on | expandable class | MODIFY_VOLUME |
| Progress | allocatedResource* | modifyVolumeStatus |
| Controller | ControllerExpand | ControllerModify |
| Node step | NodeExpandVolume | none |
| Infeasible | 3 codes | INVALID_ARGUMENT |
| Undo | lower, > capacity | previous class |
6.Commands
| Command | Shows |
|---|---|
kubectl get pvc data -o jsonpath='{.spec.resources.requests.storage}{" alloc="} | State in one line. Read it first |
kubectl events --for pvc/data | Resize and VolumeModify events |
kubectl get pvc data -o jsonpath='{.status.currentVolumeAttributesClassName} | Applied class, change in flight |
kubectl -n <ns> get resourcequota <q> -o jsonpath='{.status.used} | VAC quota over hard |
1.Two hidden walks
- A mount can walk the whole volume twice : kubelet fsGroup chown, runtime SELinux relabel. Both grow with file count.
VOLUME_driver : getsMOUNT_ GROUP volume_, no walk, both policy fields ignored.mount_ group
2.What the fsGroup walk does
Lchownto the GID, OR in0660(0440ro), setgid on dirs. Never removes bits.- OnRootMismatch skips if the root has the GID,
0770(0550ro) and setgid. - 30s :
VolumePermissionChangeInProgress, then every 60s (CSI only). - RWX skipped by default : NFS needs
File.
| Lever | Where | Effect |
|---|---|---|
| OnRootMismatch | Pod | skip if root matches |
| Always, or unset | Pod | walk every mount |
| fsGroupPolicy: None | CSIDriver | never apply |
| fsGroupPolicy: File | CSIDriver | always, even RWX |
| VOLUME_ | driver | driver does it |
| readOnly: true | Pod | skipped |
3.SELinux : a mount option
- PV,
seLinuxOptions., driverlevel seLinuxMount: true: mount-o context. Else (secret, configMap ...) a full relabel. - RWOP and change-policy gates locked v1.36.
SELinuxMountGA v1.37, unlocked :=falsein v1.37 only.
| MountOption | Recursive | |
|---|---|---|
| Who labels | kubelet, -o context | runtime walk |
| Cost | constant | grows with files |
| One node | same label only | any labels |
| Needs | label, opt-in | nothing |
4.Label conflicts from v1.37
- Two labels (or privileged + not) on one volume and node : the 2nd Pod hangs. Fix :
Recursiveor one label. - Audit first :
selinux-warning-controller(off, Controllers). Its metric names Pods : admin-only. Kubelet :volume_.manager_ selinux_*
5.Access modes, enforced where
| Mode | Scheduler | A/D ctrl | Kubelet |
|---|---|---|---|
| RWO | no check | one node | others may write |
| RWOP | preempts | one node | refuses a 2nd Pod |
| RWX, ROX | none | many nodes | none |
6.Commands
| Command | Shows |
|---|---|
kubectl get csidriver <d> -o jsonpath='{. | fsGroupPolicy, seLinuxMount |
kubectl patch csidriver <d> --type=merge -p '{"spec":{"fsGroupPolicy":"None","seLinuxMount":false}}' | Mutable. attachRequired is not |
kubectl get events -A --field-selector reason=SELinuxLabelConflict | Potential conflicts (Normal) |
kubectl get --raw /metrics | grep kubernetes_ | Gate states (SELinuxMount 1) |
kubectl get pod <p> -o jsonpath='{range .status.containerStatuses[*].volumeMounts[*]} | Recursive read-only took effect ? |
1.Attach and detach, the normal path
- A second node is refused unless the PV lists RWX or ROX : Waiting for detach (was Multi-Attach error before v1.37.0, v1.36.4, v1.35.9). Reason is still
FailedAttachVolume. In a rollout it is usually a race. - The 6m clock (hard-coded) starts once the Pod is gone or terminated. Call order : CSI Drivers.
2.A node dies under RWO
- Terminating Pods look running : no clock. Deleting the Node fails too.
- Node off :
out-of-serviceNoExecutetaint (StatefulSets). Instant detach, no health check : never on a rebooting node. Untaint after. - Or force-delete the Pod : detach after 6m unless
--disable-force-detach-on-timeout.
| Path | Trigger | Skips | Risk |
|---|---|---|---|
| Normal | unmounted | nothing | none |
| Timeout | 6m, NotReady | in-use check | corruption |
| Taint | taint key | + no wait | same |
| VA finalizer | by hand | Unpublish | still attached |
3.New signals
- Unused : Bound PVCs, Pending Pods count. Nothing acts.
- Health : redesigned, off, writes dropped.
- Stats need
GET_VOLUME_STATS(not KSM).
| Signal | By | v1.37 |
|---|---|---|
| Unused condition | kcm | beta, on |
| 3 health fields | sidecar, kubelet | alpha, off |
| kubelet_volume_stats_* | kubelet, 1m | alpha |
| ..._forced_detaches | kcm | no _total |
4.Symptom, cause, fix
FailedAttachVolume = attach side. FailedMount (FailedMapVolume for Block) = node side. A plain timeout posts no summary event since v1.28 : read the kubelet log. FileSystemResizeFailed is read-only mounts only.
| Event reason | Message starts | Cause | Fix |
|---|---|---|---|
| FailedAttachVolume | Waiting for detach | RWO in use elsewhere | wait, Recreate, or taint a dead node |
| FailedAttachVolume | AttachVolume.Attach failed | ControllerPublish error | VA attachError, attacher logs |
| FailedMount | MountVolume.MountDevice / SetUp failed | NodeStage / NodePublish | node plugin logs |
| FailedMount | ... failed while expanding volume | node expansion | node logs, lower if infeasible |
| FailedMount | ... conflicting SELinux labels | two labels, one node | Recursive, or one label |
| ExternalExpanding | nothing after it | no resizer for the driver | deploy csi-resizer |
| VolumeResizeFailed | only supports offline / OutOfRange | offline driver / infeasible | stop Pods / lower request |
| VolumeModifyFailed | InvalidArgument | modify Infeasible | set previous class |
Missing attacher or driver : CSI Drivers. VolumeAttachmentLimitExceeded : Volume Topology. Restore as the way back : Snapshots.
5.Commands
| Command | Shows |
|---|---|
kubectl get volumeattachment -o custom-columns=PV:.spec.source.persistentVolumeName, | Stuck detaches : fix the driver, keep the finalizer |
kubectl get node <n> -o jsonpath=' | Kubelet view, controller view |
kubectl taint nodes <n> node. | Node is off. Add - to remove |
kubectl get events -A --field-selector reason=FailedAttachVolume | Any wording |
kubectl get pvc <p> -o jsonpath='{. | Unused since |
kubectl get --raw "/ | Fill per PVC (nodes/proxy) |
Storage internals
Snapshots, Clones and Backup
1.Not core : CRDs plus two controllers
snapshot.CRDs (GA v1.20), CSI only : VolumeSnapshot (storage. k8s. io/ v1 vs, namespaced), VolumeSnapshotContent (vsc), VolumeSnapshotClass (vsclass).- The distro installs them and one snapshot-controller; each driver runs a csi-snapshotter (v8.6.0, CSI Drivers). CEL rules in the CRDs replaced the webhook (gone in v8.2.0); VolumeSnapshotClass has none, so edits pass.
2.One dynamic snapshot
- No class : the driver's one class with
is-default-class: "true". Zero or two fail (unlike StorageClass). - Driver returns at the cut; readyToUse may lag hours.
| Field | VolumeSnapshot | Content |
|---|---|---|
| link | boundVolumeSnapshotContentName | volumeSnapshotRef |
| handle | none | snapshotHandle |
| creationTime | date-time | int64 ns |
| restoreSize | quantity | int64 bytes |
3.Pre-provisioned, deletionPolicy
- Import : a content with
snapshotHandle,deletionPolicy,volumeSnapshotRef, ideallysourceVolumeMode. - Trust it if both point at each other. Delete the VS, not the content (finalizers).
| When | Delete | Retain |
|---|---|---|
| VS deleted | storage snapshot goes | both stay |
| namespace gone | storage snapshot goes | content orphaned |
| imported | set by hand | set by hand |
| Velero | once backup expires | while backup exists |
4.Restore : always a new PVC
- dataSourceRef names the VolumeSnapshot. No revert in place.
- Mode guard (GA v1.30) : Block <-> Filesystem needs
allow-volume-mode-change: "true"on the content, so content write access is powerful (RBAC). - Errors :
ProvisioningFailedon the PVC.
| Check, in order | Error says |
|---|---|
| CREATE_ | capability is not reported |
| snapshot not deleting | is being deleted |
| bound, content points back | bound to a different snapshot |
| content driver = provisioner | not handled by CSI driver of ... |
| readyToUse and a handle | is not Ready |
| size >= restoreSize | is less than the size |
| volume mode guard | does not have permission |
5.Clone a PVC
- Source PVC in dataSource : Bound,
storageClassNameon both, size >= source, same driver (any class, v3.2.0+), same volumeMode, CLONE_VOLUME. - Same namespace (cross-namespace is alpha). Errors say
not bound or invalid: see the provisioner log.
6.Commands
| Command | Shows |
|---|---|
kubectl get vs -n <ns> | RESTORESIZE 1Gi |
kubectl get vsc | in bytes |
kubectl get vsclass -o custom-columns='NAME:. | Two defaults ? |
kubectl patch vsc <c> --type=merge -p '{"spec":{"deletionPolicy": "Retain"}}' | Keep the data |
1.dataSource vs dataSourceRef
- A custom kind in dataSource is dropped on create, no warning : an empty volume. Use dataSourceRef.
AnyVolumeDataSource: GA v1.33, gate removed in v1.37. Passing it stops the component.
| dataSource | dataSourceRef | |
|---|---|---|
| kinds | PVC, snapshot | any non-core kind, PVC |
| other kind | dropped silently | rejected |
| sync | copied to the Ref | copied back if no namespace |
| both set | must match; both immutable | |
| namespace | must stay empty | alpha gate only |
2.Volume populators
- Unknown kind : the provisioner assumes a populator, stops.
- lib-volume-populator v3.4.0 : waits for
selected-node(WFFC), fillsprime-<uid>, moves the PV over. - A
VolumePopulator(populator.) registers each kind. Populators rebind PVs : control-plane code (Operators).storage. k8s. io/ v1beta1
3.Across namespaces (alpha, stalled)
CrossNamespaceVolumeDataSource: alpha, off since v1.26, KEP-3294 closed "not planned". Gate on kube-apiserver, controller-manager and csi-provisioner, plus the ReferenceGrant CRD and RBAC.- The grant sits in the source namespace; an empty
to.nameopens every object of that kind. Gate off : a namespaced ref is dropped silently.
4.Group snapshots : one cut
- GA v1.36. Snapshotter v8.6.0 serves
v1, storesv1beta2, gate Beta and off. Driver needs GroupController. - Label the PVCs. Members are owned by the group : none deleted alone, one driver.
- Use
v1: conversion is None, so v1beta1 readers lose members and its writes skip CEL (CRDs).
| Single | Group | |
|---|---|---|
| stored | v1 | v1beta2 (serves v1) |
| turn on | CRDs, controller | + CSIVolume |
| picks | one PVC name | label selector, resolved once |
| consistency | one volume | crash-consistent across all |
| restore | PVC from it | one PVC per member |
| CSI call | CreateSnapshot | CreateVolume |
5.Changed block tracking
- KEP-3314, beta v1.36, out of tree : the driver advertises
SNAPSHOT_and runs external-snapshot-metadata v1.1.0. Its SnapshotMetadataService gives address, audience, CA.METADATA_ SERVICE - GetMetadataDelta (base = an older snapshotHandle) returns block ranges, not data.
FAILED_: full backup. Velero 1.18 skips it.PRECONDITION
6.Commands
| Command | Shows |
|---|---|
kubectl get volumepopulators | Registered populator kinds |
kubectl get crd volumegroupsnapshots. | v1 served, v1beta2 stored |
kubectl get vs -n <ns> -o jsonpath='{range . | Members of a group |
kubectl get sms | CBT drivers (cbt.storage.k8s.io/v1beta1) |
Group snapshots ignored ? In v8.6.0 support is a flag on both snapshot components : check their args before the driver.
1.Why a snapshot is not a backup
- It sits on the same storage as the volume and goes with its namespace under Delete. Record handles.
| Failure | CSI snapshot | Velero, no data move | Velero data mover, fs-backup |
|---|---|---|---|
| namespace deleted | gone too under Delete | kept (Retain), YAML in bucket | data in the bucket |
| storage pool lost | lost if kept beside the volume | lost, only the YAML survives | survives if the bucket is elsewhere |
| whole cluster lost | re-import by snapshotHandle | re-imported, same driver name | any cluster with CSI and the bucket |
| ransomware, bad write | only if nobody deleted it | same | separate bucket; guard the key |
2.Consistency and hooks
- One snapshotcrash-consistent
- Groupcrash-consistent, all volumes
- + fsfreeze hooksapp-consistent (frozen FS)
- fs-backupnot point in time
KEP-3476 : quiesce weekly, group snapshot nightly.
3.Velero v1.18 : three ways
- v1.18.4 (now
velero-io/velero) needs--features=EnableCSI, though CSI is built in since v1.14. Hooks : no shell; first container, Fail, 30s. - Class pick : PVC annotation, Backup
_<driver>annotation, class labelvelero., the default, the only class.io/ csi-volumesnapshot-class
| CSI snapshot | CSI snapshot data movement | fs-backup (kopia) | |
|---|---|---|---|
| turn on | --features= | + --use-node-agent, --snapshot-move-data | --use-node-agent + pod annotation |
| data goes to | stays on the storage | kopia repo in the Backup | kopia repo |
| works for | CSI volumes | CSI volumes | mounted volumes, not hostPath |
| runs as | Velero server | data-mover Pods, root, BestEffort | node-agent, kubelet hostPath |
| one volume | snapshot or fs-backup, never both. Restic restores stop working in Velero 1.19 | ||
4.One backup with data movement
csiSnapshotTimeout(10m) waits for the handle (the cut);itemOperationTimeout(4h) cancels uploads.- All repos share one static key : set
repository-passwordfirst. Groups :velero.label (1.18.x talks v1beta2).io/ volume-group
6.Commands
| Command | Shows |
|---|---|
velero install --features=EnableCSI --use-node-agent --plugins=<object-store-plugin> ... | Velero with CSI and the node-agent |
velero backup create NAME --snapshot-move-data | Snapshot, then copy the data off |
kubectl -n velero get datauploads -o wide -l velero. | Watch these, not the backup phase |
velero restore create --from-backup NAME --resource-modifier-configmap <cm> | Restore with patches (new class). Postgres : CNPG |
Cluster management
Cluster API
1.What is Cluster API ?
A SIG Cluster Lifecycle sub-project. Whole clusters become custom resources : create, scale, upgrade and delete them declaratively, on cloud or on-prem.
- One CRD API instead of a different tool per installer and cloud.
- Controllers reconcile it, like any operator.
- Management : runs the CAPI and provider controllers, and holds the Cluster and Machine objects.
- Workload : the cluster those objects describe. Your apps run there, CAPI does not.
v1.14.2 (2026-09-08), API v1beta2. Workload clusters v1.31 to v1.37.
2.Bootstrap and pivot
A throwaway kind cluster builds the real management cluster, then hands over.
- Create a temporary kind cluster.
clusterctl init: kind can now manage clusters.- Apply the target cluster's YAML. Wait for it.
- Get its kubeconfig,
clusterctl initthere. clusterctl move: pause, copy, resume.- Delete kind. The target manages itself.
move is not backup and restore.
3.Core objects
- Cluster : pod and service CIDRs, serviceDomain. Refs to its InfraCluster and its control plane.
- KCP (KubeadmControlPlane) : control plane Machines, etcd membership, certificates.
- Machine : one host, one Node (
status.nodeRef). - Templates : each new Machine gets its own clones of an InfraMachineTemplate and a KubeadmConfigTemplate. Treat them as immutable : copy, then repoint the ref.
4.Same pattern as workloads
| CAPI object | Like | Job |
|---|---|---|
| Machine | Pod | One host, one Node. Never edited : its owner replaces it |
| MachineSet | ReplicaSet | Keeps N Machines. Not used directly |
| MachineDeployment | Deployment | Rolls a template change across an old and a new MachineSet |
| KubeadmControlPlane | StatefulSet | Control plane Machines and etcd. Upgrade with spec.version |
| MachinePool | ASG, VMSS | Hands workers to a provider-native group. Beta, on by default |
| MachineHealthCheck | auto-repair | Deletes unhealthy Machines owned by a MachineSet or KCP |
5.Providers
- Infrastructure : VMs, networks, load balancers.
- Bootstrap : CABPK, kubeadm and cloud-init.
- Control plane : KCP. Also IPAM and add-on (Helm) providers.
clusterctl initadds core, kubeadm bootstrap and KCP by default. The CAPI Operator uses CRDs instead.
| Infra provider | Flag | Notes |
|---|---|---|
| CAPA v2.13.1 | aws | EKS too. IAM via clusterawsadm |
| CAPZ v1.27.0 | azure | AKS too. Azure Service Operator |
| CAPV v1.17.0 | vsphere | OVA images v1.30 to v1.37 |
| CAPM3 v1.14.1 | metal3 | Bare metal, bring Ironic |
| CAPD | docker | Development and tests only |
Flag = clusterctl init --infrastructure <name>.
1.ClusterClass and managed topology
A ClusterClass bundles templates, variables and patches. A Cluster then sets only spec.topology. The ClusterTopology gate is alpha and off in v1.14 : export CLUSTER_ before clusterctl init.
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata: { name: capi-quickstart }
spec:
topology:
classRef: { name: my-class-v1 }
version: v1.37.0 # bump to upgrade
controlPlane: { replicas: 3 } # odd for etcd
workers:
machineDeployments:
- class: default-worker
name: md-0
replicas: 3
2.Upgrades replace Machines
A new Machine joins, then the old one is drained and deleted. Control plane first, then workers.
- Topology : change only
spec.. Since v1.12 it can chain minors from ClusterClasstopology. version spec..kubernetesVersions - Rollout : RollingUpdate (maxSurge 1, maxUnavailable 0) or OnDelete (MD only).
spec.forces one.rollout. after - In-place (alpha, v1.12) : an update extension changes Machines without replacing them.
3.Commands
| Command | Does |
|---|---|
clusterctl init --infrastructure docker | Make this a management cluster. Export gates first |
clusterctl generate cluster capi-quickstart --flavor development --kubernetes-version v1.37.0 > capi-quickstart.yaml | Render a template from the provider repo. --list-variables shows what it needs |
clusterctl describe cluster capi-quickstart | Tree of the Cluster and its Machines, surfacing problems |
clusterctl get kubeconfig capi-quickstart > capi.kubeconfig | Admin kubeconfig, from the <name>-kubeconfig Secret |
clusterctl move --to-kubeconfig=mgmt.kubeconfig --dry-run | Pivot. The target needs the same or newer providers |
kubectl patch cluster capi-quickstart --type json -p '[{"op":"replace", "path":"/spec/topology/version", "value":"v1.37.1"}]' | Upgrade a topology cluster, control plane first |
kubectl delete cluster capi-quickstart | Teardown. Cascades through the infrastructure in order |
- Never
kubectl delete -f cluster.yaml: infrastructure can be left behind. clusterctl upgradeupgrades providers and CRDs, never a cluster's Kubernetes.- No CNI, no Ready nodes. CAPI installs none.
- nodeDrainTimeoutSeconds 0 = no limit. A PDB can stall a rollout.
- Add-ons : ClusterResourceSet (GA) applies a CNI by label. CAAPH installs Helm charts.
- GitOps : Flux applies the Cluster YAML. A Kustomization with kubeConfig.secretRef
<cluster>-kubeconfigdeploys into the workload cluster. - v1beta2 Cluster and Machine refs use apiGroup, not apiVersion. v1beta1 stops being served in v1.16.
GitOps
GitOps with Flux: Deep Dive
1.Repo layout
Running Flux for many apps and clusters, at v2.9.6. The basics are in Flux CD v2.
- apps/ : a base plus one overlay per env. infrastructure/ : controllers, then configs.
- clusters/<env>/ : only Flux objects, such as Kustomizations that point at that env's overlays.
- Env differences : Kustomize overlays, or
.spec.patcheson a Flux Kustomization.
- fleet/
- apps/ : base/, staging/, production/
- infrastructure/ : controllers/, configs/
- clusters/staging/, clusters/production/
- flux-system/ : written by bootstrap
- artifacts.yaml, infrastructure.yaml, apps.yaml
2.Promotion
| Layout | How a change moves on |
|---|---|
| Monorepo | Overlay bump. Big PRs can hide prod |
| Repo per env | Cross-repo PRs. Hardest for infra |
| Repo per team | Platform onboards each team repo |
| Repo per app | Config repo points at it by semver |
- Staging gets new app versions from image automation.
- Production gets them by pull request.
- Infra goes to staging first, then on after e2e tests.
- Helm : a passing staging upgrade fires an Alert to a
githubdispatchProvider. CI readsmetadata.revisionand opens the prod PR.
3.Layers with dependsOn
- Controllers and CRDs, then their custom resources, then apps. All with
prune: true. - infra-configs has no wait : apps starts once configs are applied, healthy or not.
- ArtifactGenerator slices the monorepo. Its revision is a content hash, so a commit outside apps/ leaves apps idle.
- Needs source-watcher. ExternalArtifact gate off : Stalled AccessDenied.
4.What Ready means
| Dependency sets | Its Ready means |
|---|---|
| nothing | applied, no more |
healthChecks | listed objects healthy |
wait: true | every object healthy |
| only CEL exprs | applied (CEL unused) |
Only the dependency's wait counts, not the dependent's.
- Reads the API server, not the cache : generation observed and Ready=True.
- Same source : the dependency must have applied that revision too.
- No watch : a blocked dependent re-checks every 30s (
--requeue-dependency). readyExprreplaces the built-in check. Repeat the Ready test in it.- Health : kstatus or a CEL expr, polled every 5s until timeout. Failed ends it early.
5.Interval is not the delay
A run starts at once on a new source revision, a spec change, flux reconcile or a Receiver. Also on an edit to a ConfigMap or Secret it uses in substituteFrom, decryption or kubeConfig, if labelled reconcile.. Labelling anything else does nothing.
- intervalfull cycle that fixes drift. Shorter = more API load, not faster delivery
- retryIntervaldefaults to interval. Flat, no backoff
- timeoutinterval minus 30s (1h : 59m30s)
1.Apply in stages, then prune
2.Why kubectl edits come back
- Before each apply Flux takes over the fields of every field manager whose name starts with kubectl. Then Git wins : extra fields go, Git's values return.
- Spared : /status and /scale fields,
ssa: Mergeobjects, other managers. rollout restartrolls twice. Add--field-manager=flux-client-side-apply.
Table keys live under kustomize..
| Object key | Effect |
|---|---|
| ssa: Override | Default. Git wins |
| ssa: Merge | Keeps others' fields, not atomic lists |
| ssa: IfNotPresent | Create once, never patch |
| ssa: Ignore | Skipped, health too |
| prune: Disabled | Kept when it leaves Git |
| reconcile: disabled | No apply, no prune |
| force: Enabled | Recreate if immutable |
3.Ignore rules (2.9)
.spec.ignore: JSON pointer paths plus an optional target. No target means every object.- kubectl edit and the HPA (via /scale) are Update-type, so their values are adopted.
- HPA : commit a starting replicas and ignore
/spec/replicas. - An ignored path keeps hand edits forever. Scope it.
4.SOPS and variables
- Encrypt only
dataandstringData. - secretRef keys :
*.agekey,*.asc,sops.*. It replaces the global age key. - No secretRef : global age key or KMS identity.
- No decryption :
ENC[...]is applied as is. See Secrets in Git with SOPS or External Secrets.
Test : kustomize build . | flux envsubst --strict. flux debug ks <n> --show-vars prints secrets.
decryption:
provider: sops
secretRef: {name: sops-age}
postBuild:
substitute: {env: prod} # wins
substituteFrom: # later wins
- {kind: ConfigMap, name: vars}
2.9 strict : a ${var} with no value and no default fails the build, when substitution runs. ${var:=x} is fine.
5.Tenant lockdown
--no-cross-namespace-refs=true: a sourceRef to another namespace stalls (AccessDenied). dependsOn is not checked.--no-remote-bases=true, and--default-service-account=defaulton kustomize and helm (source-watcher from 2.10).- flux-system keeps cluster-admin through
serviceAccountName: kustomize-controller. See RBAC.
2.HelmRelease defaults that bite
| Field | Default | Change it to |
|---|---|---|
| *.strategy | RemediateOnFailure, 0 retries | RetryOnFailure : 5m, no rollback |
| upgrade.crds | Create : never updates CRDs | CreateReplace |
| driftDetection.mode | disabled | enabled |
| postRenderStrategy | combined (2.9) : sees hooks | nohooks |
| upgrade.serverSideApply | auto : keeps the old method | enabled |
| chartRef (OCI) | version+digest : re-push upgrades | DisableChartDigestTracking |
3.Receivers and latency
- No webhooks : scan interval + Git poll interval + push and rollout. 5m + 1m is about 6 min.
- Receivers on both edges : seconds plus rollout.
- Never on the path : Kustomization, HelmRelease and automation intervals.
- An automation sees policies in its namespace only, and never pokes its GitRepository.
- Patch
requestedAton the sources they list. Path in..status. webhookPath - 2.9
generic-oidc: no Secret. Pin the token claims. - Rate-limit the public Ingress.
- Commit status : a
githuborgitlabProvider + an Alert. - Same event dropped for 5m. Big fleets :
DisableCommitStatusEvent. - Chat :
eventSeverity: error.
4."My change is not deployed"
| Ready reason | Likely cause | Look with |
|---|---|---|
| ArtifactFailed | Source not Ready : auth, ref, verify | flux get sources all -A |
| BuildFailed (strict mode) | A ${var} with no value | flux debug ks <n> --show-vars |
| DependencyNotReady | Dependency not Ready, or behind | flux get ks <dependency> |
| HealthCheckFailed | Rollout slower than timeout, or broken | kubectl describe the object |
| Stalled AccessDenied | Cross-namespace ref, or gate off | flux debug ks <n> --show-status |
| HelmRelease RetriesExceeded | 0 retries by default : one failure stalls | fix, then flux reconcile hr <n> --reset |
Start with flux get all -A --status-selector ready=false. Owner : flux trace -n <ns> deployment <n>. Preview : flux diff ks <n> --path ./dir.
5.Scale
- One work queue per kind, sized by
--concurrent. - Shards : label a source and its users
sharding..fluxcd. io/ key - Upgrades : FluxInstance
"2.9.x"floats patches.
| cluster. | Workers | Limits | Dep. requeue |
|---|---|---|---|
| small | 5 | 1 CPU, 512Mi | 10s |
| medium | 10 | 2 CPU, 1Gi | 5s |
| large | 20 | 3 CPU, 3Gi | 5s |
Rows set kustomize and helm. Plain defaults : kustomize 4, helm 4, source 2.
GitOps
Secrets in Git with SOPS
1.Envelope encryption
- SOPS encrypts values, not files. The YAML shape stays readable.
- One random 256-bit data key per file seals each selected value : AES256-GCM, its own IV, its key path as authenticated data.
- The data key is wrapped once per master key (age, PGP, cloud KMS, Vault or OpenBao, HuaweiCloud KMS). Any one can decrypt.
- Adding a recipient re-wraps the data key. Values stay as they are.
SOPS v3.13.3, age v1.3.2, Flux v2.9.6. Flux basics : Flux CD v2. Short version : GitOps with Flux: Deep Dive.
2.What stays readable
- Keys,
apiVersion,kindandmetadatastay plain, so a diff shows which field changed. Encrypt onlydataandstringData. - Comments : the default
unencrypted_encrypts all. Withsuffix encrypted_regex, only those inside the selected keys. - The MAC covers every value, plain or not. A hand edit of
namespacegives "MAC mismatch" (exit 51).mac_(3.9.0+) covers encrypted values only.only_ encrypted: true
apiVersion: v1
kind: Secret
metadata: {name: webhook-token, namespace: default}
stringData:
token: ENC[AES256_GCM,data:...,iv:...,tag:...,type:str]
sops:
age:
- recipient: age1...
enc: # wrapped data key
encrypted_regex: ^(data|stringData)$
lastmodified: "..."
mac: ENC[...]
version: 3.13.3 # sorted keys, 4-space indent
3..sops.yaml creation rules
- The first matching
path_regexwins. It is read relative to the config file. - The config is found from the current directory upwards, not from the file's folder.
- Overlay trap : a rule with no selector encrypts
apiVersion,kindandmetadatatoo. An overlay's own.sops.yamlcounts only if youcdinto it. sops editandsops setreuse the selector stored in the file.
creation_rules: # Helm values: whole file - path_regex: .*values\.yaml$ age: age1... # Secret manifests: values only - path_regex: clusters/prod/.*\.ya?ml$ encrypted_regex: ^(data|stringData)$ age: age1...
Rule order matters : under the Secret rule a values file stays plain.
4.Everyday commands
| Task | Command |
|---|---|
Make an age key (post-quantum : -pq) | age-keygen -o age.agekey |
| Encrypt a Secret, values only | sops encrypt --age age1... --encrypted-regex '^( |
| Encrypt using .sops.yaml | sops encrypt --in-place s.yaml (run inside the repo) |
| Edit, keeping the MAC valid | sops edit s.yaml |
| Read one value | sops decrypt --extract '["data"]["password"]' s.yaml |
| Set one value (JSON) | sops set s.yaml '["stringData"]["k"]' '"v"' |
| Is it encrypted ? | sops filestatus s.yaml |
| MAC over encrypted values only | sops --mac-only-encrypted encrypt -i s.yaml (global flag, before the subcommand) |
Readable git diff | *.yaml diff=sopsdiffer in .gitattributes, then git config diff. |
.sops.yaml above where you run it.1.Where Flux decrypts
- Only kustomize-controller decrypts. helm-controller never does : it reads Secrets already in the cluster.
- Runs only when
spec.is set on the Kustomization that applies the objects. On the rootdecryption. provider: sops flux-systemit does nothing for its children. - No MAC check in the controller : patches and generators would break it. The CLI does check.
2.The decryption block
| Field or key | Effect |
|---|---|
| provider | Required. Only sops |
| secretRef.name | Key Secret, same namespace |
| serviceAccountName | Secret-less cloud or Vault auth |
| decrypt: Disabled | Skips the post-build pass only |
| substitute: disabled | Keeps ${...} in values |
| watch: Enabled | Key Secret edit starts a run |
Last three : annotations or labels under kustomize. and reconcile..
kind: Kustomization # kustomize.toolkit.fluxcd.io/v1 spec: path: ./apps/production sourceRef: {kind: GitRepository, name: flux-system} decryption: provider: sops secretRef: name: sops-age # entry must end in .agekey
CLI : flux create kustomization ... --decryption-provider=sops --decryption-secret=sops-age.
3.Where the keys come from
| Secret entry | Holds |
|---|---|
| *.agekey | age identities (PQ from Flux 2.9) |
| *.asc | armored PGP key, no passphrase |
| sops.vault-token | Vault or OpenBao token |
| sops.aws-kms | static AWS keys (YAML) |
| sops.azure-kv | service principal or identity |
| sops.gcp-kms | service-account JSON |
Any other name, such as age.key, is ignored without a warning.
- secretRef : a Secret with the entries in the table.
- Global age key :
--sops-age-secret(2.7+). Still needsprovider: sopsper Kustomization. AsecretRefreplaces it, no merge. - Cloud identity : the controller's own, or
serviceAccountNamewith theObjectLevelWorkloadIdentitygate (2.6+, else Stalled). See RBAC. - Vault, in order : static token,
--sops-vault-configmap(2.9+),VAULT_TOKEN. No fallback.
4.Generators and Helm values
- Generator files are decrypted before the build. Format comes from the key name (
values.yaml=). - Helm : generate a Secret from the encrypted values, point
valuesFromat it. The hash suffix upgrades the release on change. - Values also land in Helm release Secrets (
helm get values).
secretGenerator:
- name: podinfo-values
files:
- values.yaml=my-values.enc.yaml
configurations: [kustomizeconfig.yaml]
# kustomizeconfig.yaml: nameReference
# Secret -> spec/valuesFrom/name (HelmRelease)
5.No safety net
Decryption off, ciphertext under stringData : the Secret goes live with the raw ENC[...] text and the Kustomization says Ready=True. ConfigMaps and generated Secrets too. kubectl apply -k never decrypts.
- Normalize drops the top-level
sopsfield, so the guard ("is SOPS encrypted, configuring decryption is required") only fires for ciphertext underdata:. - Land
decryptionin an earlier commit than the first encrypted file.
provider: sops is set, with keys found by name or cloud identity. Nothing refuses ciphertext.1.Picking a master key
| Backend | In .sops.yaml | Flux gets it from | Watch out |
|---|---|---|---|
| age | age: age1... | *.agekey entry or global key | Simplest. PQ keys need Flux 2.9, sops 3.12.1+ |
| PGP | pgp: <fingerprint> | *.asc entry | Not deprecated, but docs prefer age |
| Cloud KMS | kms, gcp_kms ... | creds entry or workload identity | No key file. KMS must answer every run |
| Vault, OpenBao | hc_ | token, or Kubernetes auth | Role name fixed to {namespace}_{sa} |
2.Keyless AWS KMS
- One KMS key, no age, no static AWS keys. Kustomizations set only
provider: sops, so the controller uses its own identity. - A sidecar,
aws-signing-helper serve, plays the EC2 metadata service.AWS_points the SDK at it.EC2_ METADATA_ SERVICE_ ENDPOINT - It signs with a cert-manager client cert and gets short-lived creds for a
kms:Decrypt-only role. - Fresh cluster : SOPS Kustomizations retry until the CA exists.
3.Rotation
updatekeys: recipients from.sops.yaml, same data key.rotate: new data key, every value re-encrypted. Do it after removing a recipient.- Leaked key (docs order) : drop it, updatekeys, rotate, commit, then change the real passwords. History still opens with the old key.
$ sops updatekeys -y secrets/*.yaml $ sops rotate -i secrets/db.yaml # new key Secret in the cluster, then $ flux reconcile ks apps
A key Secret edit starts no run unless labelled reconcile..
4.When it breaks
| Symptom | Cause | Fix |
|---|---|---|
| "failed to load age identities ... no master key was able to decrypt" | No *.agekey entry loaded: misnamed, or a secretRef hides the global key | Rename to identity.agekey |
| "MAC mismatch" (CLI, exit 51) | Hand edit, or decrypting merged build output | sops edit, or decrypt the patch file |
| "illegal base64 data at input byte 3" | Committed **SOPS** preview output, or plain text under data: | Use stringData (Secrets). Never commit previews |
| "cipher: message authentication failed" | Plain patch onto an encrypted object | Encrypt the patch too |
| Secret value changed after decrypt | ${...} was substituted | substitute: disabled or $${var} |
Ready=True, ENC[...] live | No decryption on that Kustomization | Add provider: sops |
flux build and flux diff never decrypt : Secret values show as **SOPS**, ConfigMaps as ENC[...].
5.SOPS or something else ?
| SOPS in Flux | Sealed Secrets | External Secrets | |
|---|---|---|---|
| In Git | ciphertext in normal files | a SealedSecret | a reference, no values |
| Decrypts | kustomize-controller | its own controller | pulls from a store |
| Rotation | per file, by hand | by hand (key renewal is not rotation) | in the store |
| Needs live | KMS or Vault at reconcile | only its controller | the store, at sync |
More on the third option in External Secrets. Argo CD needs a plugin such as ksops for SOPS, and its docs caution against secret-generating plugins.