Extending the API
Controllers and Reconciliation
1.What is a controller ?
A loop that keeps one kind of object true. You write the desired state in spec. The controller moves the world toward it and reports in status.
- Read the desired state (spec) from the API server.
- Observe the current state.
- Act to close the gap, then report back in status.
- Most built-in controllers act only through the API server. The Job controller never runs a container. It asks for Pods, and kubelets run them.
2.Level, not edge
- Reconcile from what the object looks like now, never from the event that woke you.
- spec.replicas goes 2 5 3 ? The system does not have to pass through 5.
- Your controller may have been down while things changed. Edge triggers and watches only make the loop faster.
for { desired := get(); current := observe(); makeChanges(desired, current) }
3.The control loop
- One pass handles one key,
namespace/name. It is idempotent : create if missing, update if different, do nothing if converged. Safe when it runs twice, runs late or reads a stale cache. - Open world : users, the HPA and the garbage collector change things too.
- A child's watch event queues its owner's key, via ownerReferences (Owners and GC).
- Outside systems (cluster autoscaler, cloud routes) : desired state from the API, status back to it.
Each pass compares whole states. A missed event does no harm.
4.Spec in, status out
# kubectl get deploy web -o yaml (trimmed) metadata: generation: 7 # +1 on a spec change resourceVersion: "184213" # opaque spec: replicas: 3 # desired, written by you status: # observed, via /status observedGeneration: 7 # == generation availableReplicas: 3 conditions: - type: Available status: "True" reason: MinimumReplicasAvailable lastTransitionTime: "2026-10-06T14:02:11Z"
5.Status rules
- generation : set by the API server. Deployments bump it on spec or annotation changes, CRDs with a status subresource on any change except metadata and status.
- observedGeneration : the generation the controller acted on. Lower means the status is stale.
- conditions : keyed by type. status True, False or Unknown, a CamelCase reason (required), lastTransitionTime = last flip. Missing means Unknown.
- Write status only through /status. PUT and POST on the main resource ignore it.
- 409 Conflict : someone wrote first. Re-read and retry (return the error to requeue).
6.Commands
| Command | Shows |
|---|---|
kubectl get deploy -o custom-columns=NAME:.metadata.name, | OBSERVED < GEN : not caught up yet |
kubectl get deploy web -o jsonpath='{range . | Each condition with its reason |
kubectl wait --for=condition=Available deploy/web --timeout=120s | Blocks until the condition is True |
kubectl rollout status deploy/web | observedGeneration first, then replicas |
kubectl get pods -w --output-watch-events | The event stream an informer gets |
kubectl events --for deployment/web | Events recorded while reconciling |
1.Inside a client-go controller
client-go gives you the informer. You write the handlers, the queue wiring and reconcile.
- Informer = Reflector + RealFIFO (DeltaFIFO before 1.33) + Indexer, one per type per process via a SharedInformerFactory.
- Reflector : LIST, then WATCH from its resourceVersion. On 410 Gone it relists (API Machinery).
- Lister : reads the cache, never the API server.
factory.Start(), thencache., only then the workers.WaitForCacheSync() - Handlers only Add keys, the owner's key for a child (
metav1.). Deletes may arrive as aGetControllerOf() DeletedFinalStateUnknown.
2.The workqueue
- Fair (FIFO) and stingy : a key added many times before a worker gets it runs once.
- Two workers never hold the same key at once.
- Re-added while processing ? It runs again after Done().
- N workers = N goroutines calling Get().
| Call | When |
|---|---|
| Done(key) | Always, in a defer |
| Forget(key) | Success : reset its backoff |
| AddRateLimited(key) | Error : retry with backoff |
| AddAfter(key, d) | Recheck after d |
Default limiter : max of per-key 5ms x 2n (cap 1000s) and a 10 qps, burst 100 token bucket.
3.kube-controller-manager
kube-controller-manager (kcm) is one binary with 49 built-in loops in v1.37, each with its own workers (Architecture). --controllers defaults to * (every on-by-default loop), and -name turns one off. Three are off by default : bootstrap-signer, selinux-warning and token-cleaner.
| --controllers name | Reads, then manages | Tuning flag (default) |
|---|---|---|
| deployment-controller | Deployment ReplicaSets | --concurrent-deployment-syncs (5) |
| replicaset-controller | ReplicaSet Pods | --concurrent-replicaset-syncs (5) |
| daemonset-controller | DaemonSet one Pod per eligible node | --concurrent-daemonset-syncs (2) |
| job-controller | Job Pods, then marks the Job finished | --concurrent-job-syncs (5) |
| endpointslice-controller | Service EndpointSlices | --concurrent-service-endpoint-syncs (5) |
| namespace-controller | Terminating ns deletes all inside | --concurrent-namespace-syncs (10) |
| garbage-collector-controller | ownerReferences dependents of gone owners | --concurrent-gc-syncs (20) |
| node-lifecycle-controller | Node Leases node health | --node-monitor-grace-period (50s) |
4.Leader election on a Lease
- Replicas race to update one
coordination.Lease. resourceVersion lets exactly one write win.k8s. io/ v1 - The leader keeps updating renewTime. Standbys wait leaseDurationSeconds after they last saw it change (own clock).
- kcm defaults :
--leader-elect=true, lease 15s, renew deadline 10s, retry 2s. - Lost the Lease ? kcm exits on purpose and the kubelet restarts it.
holderIdentity is <hostname>_<uuid>. Not fencing : keep reconcile idempotent.