API machinery

API Machinery and Server-Side Apply

1.Resources, kinds and URLs

  • Resource vs kind : the URL uses the resource (deployments), the body the kind (Deployment).
  • Core group : /api/v1, apiVersion v1. Named : /apis/GROUP/VERSION, apiVersion apps/v1.
  • Identity = group + resource + namespace + name. v1beta1 and v1 are two views of one stored object, so names clash.
/apis /apps /v1 /namespaces/prod groups group version namespace /deployments /web /status resource name subresource bodyapiVersion: apps/v1, kind: Deployment
Discovery/api and /apis list what is served. Aggregated discovery (stable since 1.30) returns every group in 2 requests. Schemas at /openapi/v3.

2.A request, step by step

  • Every request : authentication (401), audit and impersonation, APF (429), authorization (403, RBAC and Service Accounts).
  • Writes : mutating admission, object schema validation (422), validating admission (Admission Control), then etcd. Stale resourceVersion : 409.
  • Unknown fields are dropped with a Warning. fieldValidation=Strict makes them a 400.
  • Reads skip admission and come from the watch cache. While it is cold, watches and unpaginated or filtered lists get 429 with Retry-After.
kubectl / client-go Authentication 401 Audit + impersonation APF (FlowSchema, level) 429 Authorization 403 writes Mutating admission Schema validation 422 Validating admission etcd 409 reads : get, list, watch skip admission Watch cache serves reads change stream

3.resourceVersion and conflicts

  • resourceVersion (RV) : a string, new on every write. Pass it back unmodified. Compare as integers only within one group + resource (required since 1.35).
  • PUT sends metadata.resourceVersion. Stale ? 409 Conflict. GET again, reapply your change, retry.
  • Patches skip this check unless they carry the RV or a JSON Patch test op.
FieldChanges on
resourceVersionEvery write, status and labels too
generationDesired state (spec) changes
observedGenerationSet in status by the controller. Equal to generation : status is current

How controllers use observedGeneration : Controllers and Reconciliation.

In short : Every request passes authentication, APF and authorization, and writes also pass admission and validation. resourceVersion changes on every write, so a stale PUT gets 409, while generation moves only when the desired state does.

1.List, then watch

  • A LIST returns the items and the collection's RV. WATCH from there, not from an item's RV.
  • etcd keeps about 5 min of history. Older RV : 410 Gone. Clear the cache, re-list.
  • Bookmarks carry only a fresh RV for reconnects. No interval promise.
  • Pages : limit=N returns metadata.continue. All pages share one RV. Tokens expire in about 5 min.
  • Streaming list (WatchList, beta, on) : one WATCH with sendInitialEvents=true sends the state as ADDED events, then a BOOKMARK.
Client Reflector / informer kube-apiserver served from watch cache 1. LIST ?limit=500 items + RV 10245 + continue token 2. LIST &continue=... until empty 3. WATCH ?resourceVersion=10245 ADDED / MODIFIED / DELETED BOOKMARK : fresh RV only 4. dropped ? re-WATCH from last RV 410 Gone : RV compacted (~5 min) clear the cache, back to 1

2.Ways to write an object

MethodHTTP / Content-TypeConflictsUse for
updatePUT, the whole object409 if metadata.resourceVersion is staleRead-modify-write, you write the retry loop
JSON PatchPATCH application/json-patch+jsonOnly via a test op or the RVExact path ops, compare-and-set
JSON Merge PatchPATCH application/merge-patch+jsonNone, last write winsSmall partial edits
Strategic Merge PatchPATCH application/strategic-merge-patch+jsonNoneBuilt-ins only, never CRDs. Superseded by SSA
Server-Side ApplyPATCH application/apply-patch+yaml ?fieldManager=409 on another manager's field unless force=trueDeclarative config, controllers, create-or-update

3.Server-Side Apply (SSA)

  • Stable since 1.22. The body is your partial intent : it creates the object if missing, patches it otherwise.
  • managedFields lists each manager, its operation (Apply or Update) and the fields it owns.
  • Apply over another manager's field : 409 unless force=true. Update never conflicts, it takes ownership. Same value : shared.
  • A field you drop is removed only if no one else owns it. Controllers should always force.
  • Lists merge by x-kubernetes-list-type (atomic, set, map). CRD lists are atomic unless marked (Custom Resources (CRDs)).
1. you apply 3 2. controller sets 5 3. apply 3 again replicas: 3 kubectl owns it operation : Apply replicas: 5 controller owns it Update, no conflict 409 Conflict field owned by the controller A : force --force-conflicts you own it alone B : drop it remove replicas give up ownership C : match apply the live 5 shared ownership
Gotchas
  • A renamed manager stays on as a second owner.
  • Drop replicas before the HPA owns it : it resets to 1.
  • Clear managedFields with [{}], not [].
In short : Watch from the list's RV and re-list on 410. A PUT checks resourceVersion, most patches do not, and SSA tracks an owner per field instead : apply over another manager's field and you get 409.

1.API Priority and Fairness

  • Stable since 1.29. Each apiserver has 600 seats : 400 inflight + 200 mutating.
  • Priority levels split them by nominalConcurrencyShares, and lend and borrow.
  • FlowSchemas match by ascending matchingPrecedence. First match wins.
  • A flow = FlowSchema + distinguisher (ByUser, ByNamespace). Shuffle sharding picks its queue.
  • A LIST costs seats by object count. exec and logs bypass APF.
Request user, verb, ns FlowSchema first match wins Priority level Limited or Exempt Exempt : no queue Limited Exempt level system:masters Queues shuffle sharding Flow user or namespace seat frees up full or waited Execute holds its seats 429 Too Many Requests
Gotchas
  • Edits to suggested objects revert each minute unless annotated apf.kubernetes.io/autoupdate-spec: "false".
  • Replicas under one ServiceAccount are one flow.

2.Deprecation and version skew

API level or componentRule
Alpha APIOff by default. Can be removed in any release without notice
Beta APINew betas off by default since 1.24. Still served 9 months or 3 releases after deprecation, whichever is longer
GA APICan be deprecated, never removed within v1
kube-apiserver (HA)Instances within 1 minor of each other
controller-manager, schedulerNot newer than kube-apiserver, up to 1 older
kubelet, kube-proxyNot newer than kube-apiserver, up to 3 older
kubectlWithin 1 minor of kube-apiserver, either way

3.Commands

CommandShows
kubectl api-resources -o wideDiscovery as a table : APIVERSION, NAMESPACED, KIND, VERBS
kubectl get pods -w --output-watch-eventsList, then watch, printing each event type
kubectl apply --server-side --field-manager=ci-cd -f deploy.yamlSSA as manager ci-cd (default kubectl). A conflict fails with 409
kubectl apply --server-side --force-conflicts -f deploy.yamlTake the conflicting fields, become sole owner
kubectl get deploy web -o yaml --show-managed-fieldsmanagedFields : who owns which field, Apply or Update
kubectl get flowschemas,prioritylevelconfigurationsAPF config
In short : APF queues each flow fairly, so one noisy client cannot starve the rest. Alpha APIs can vanish any time, a deprecated beta is served for at least 3 releases, GA stays for all of v1, and the kubelet may trail kube-apiserver by 3 minors but never lead it.