Extending the API

Controllers and Reconciliation

1.What is a controller ?

A loop that keeps one kind of object true. You write the desired state in spec. The controller moves the world toward it and reports in status.

  • Read the desired state (spec) from the API server.
  • Observe the current state.
  • Act to close the gap, then report back in status.
  • Most built-in controllers act only through the API server. The Job controller never runs a container. It asks for Pods, and kubelets run them.

2.Level, not edge

  • Reconcile from what the object looks like now, never from the event that woke you.
  • spec.replicas goes 2  5  3 ? The system does not have to pass through 5.
  • Your controller may have been down while things changed. Edge triggers and watches only make the loop faster.
for { desired := get(); current := observe(); makeChanges(desired, current) }

3.The control loop

  • One pass handles one key, namespace/name. It is idempotent : create if missing, update if different, do nothing if converged. Safe when it runs twice, runs late or reads a stale cache.
  • Open world : users, the HPA and the garbage collector change things too.
  • A child's watch event queues its owner's key, via ownerReferences (Owners and GC).
  • Outside systems (cluster autoscaler, cloud routes) : desired state from the API, status back to it.
You / GitOps apply spec (desired) API server : generation +1 read the current value, not the change status observedGeneration + conditions written via /status reconcile(key) one key, one pass observed cache + world create / update / delete only if different children or an external system watch event

Each pass compares whole states. A missed event does no harm.

4.Spec in, status out

# kubectl get deploy web -o yaml (trimmed)
metadata:
  generation: 7        # +1 on a spec change
  resourceVersion: "184213"   # opaque
spec:
  replicas: 3          # desired, written by you
status:                # observed, via /status
  observedGeneration: 7   # == generation
  availableReplicas: 3
  conditions:
  - type: Available
    status: "True"
    reason: MinimumReplicasAvailable
    lastTransitionTime: "2026-10-06T14:02:11Z"

5.Status rules

  • generation : set by the API server. Deployments bump it on spec or annotation changes, CRDs with a status subresource on any change except metadata and status.
  • observedGeneration : the generation the controller acted on. Lower means the status is stale.
  • conditions : keyed by type. status True, False or Unknown, a CamelCase reason (required), lastTransitionTime = last flip. Missing means Unknown.
  • Write status only through /status. PUT and POST on the main resource ignore it.
  • 409 Conflict : someone wrote first. Re-read and retry (return the error to requeue).

6.Commands

CommandShows
kubectl get deploy -o custom-columns=NAME:.metadata.name,GEN:.metadata.generation,OBSERVED:.status.observedGenerationOBSERVED < GEN : not caught up yet
kubectl get deploy web -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'Each condition with its reason
kubectl wait --for=condition=Available deploy/web --timeout=120sBlocks until the condition is True
kubectl rollout status deploy/webobservedGeneration first, then replicas
kubectl get pods -w --output-watch-eventsThe event stream an informer gets
kubectl events --for deployment/webEvents recorded while reconciling
In short : A controller is a level-based loop : it reads spec, compares it with what exists, changes only the difference, and reports observedGeneration and conditions in status.

1.Inside a client-go controller

kube-apiserver shared informer LIST (or stream), then WATCH Reflector deltas RealFIFO store Indexer local cache notify Event handlers Add(key) Workqueue namespace/name keys Get() Workers x N one key reconcile(key) your code Lister Get() read only, no API call requeue on error writes, incl. /status

client-go gives you the informer. You write the handlers, the queue wiring and reconcile.

  • Informer = Reflector + RealFIFO (DeltaFIFO before 1.33) + Indexer, one per type per process via a SharedInformerFactory.
  • Reflector : LIST, then WATCH from its resourceVersion. On 410 Gone it relists (API Machinery).
  • Lister : reads the cache, never the API server.
  • factory.Start(), then cache.WaitForCacheSync(), only then the workers.
  • Handlers only Add keys, the owner's key for a child (metav1.GetControllerOf()). Deletes may arrive as a DeletedFinalStateUnknown.

2.The workqueue

  • Fair (FIFO) and stingy : a key added many times before a worker gets it runs once.
  • Two workers never hold the same key at once.
  • Re-added while processing ? It runs again after Done().
  • N workers = N goroutines calling Get().
CallWhen
Done(key)Always, in a defer
Forget(key)Success : reset its backoff
AddRateLimited(key)Error : retry with backoff
AddAfter(key, d)Recheck after d

Default limiter : max of per-key 5ms x 2n (cap 1000s) and a 10 qps, burst 100 token bucket.

3.kube-controller-manager

kube-controller-manager (kcm) is one binary with 49 built-in loops in v1.37, each with its own workers (Architecture). --controllers defaults to * (every on-by-default loop), and -name turns one off. Three are off by default : bootstrap-signer, selinux-warning and token-cleaner.

--controllers nameReads, then managesTuning flag (default)
deployment-controllerDeployment  ReplicaSets--concurrent-deployment-syncs (5)
replicaset-controllerReplicaSet  Pods--concurrent-replicaset-syncs (5)
daemonset-controllerDaemonSet  one Pod per eligible node--concurrent-daemonset-syncs (2)
job-controllerJob  Pods, then marks the Job finished--concurrent-job-syncs (5)
endpointslice-controllerService  EndpointSlices--concurrent-service-endpoint-syncs (5)
namespace-controllerTerminating ns  deletes all inside--concurrent-namespace-syncs (10)
garbage-collector-controllerownerReferences  dependents of gone owners--concurrent-gc-syncs (20)
node-lifecycle-controllerNode Leases  node health--node-monitor-grace-period (50s)

4.Leader election on a Lease

  • Replicas race to update one coordination.k8s.io/v1 Lease. resourceVersion lets exactly one write win.
  • The leader keeps updating renewTime. Standbys wait leaseDurationSeconds after they last saw it change (own clock).
  • kcm defaults : --leader-elect=true, lease 15s, renew deadline 10s, retry 2s.
  • Lost the Lease ? kcm exits on purpose and the kubelet restarts it.
kcm-a (leader) OnStartedLeading controllers leader only kcm-b (standby) kcm-c (standby) Lease kube-system/ kube-controller-manager holderIdentity renewTime renew every 2s read 15s with no renew : standbys race, one write wins, rest get 409

holderIdentity is <hostname>_<uuid>. Not fencing : keep reconcile idempotent.

In short : Informers share one cache per type, handlers only queue keys, workers reconcile one key at a time with backoff, and one kube-controller-manager replica leads through a Lease.