API machinery

Owners, Finalizers and Garbage Collection

1.Owners and dependents

The garbage collector (GC) deletes a dependent once all its owners are gone. The link lives on the dependent. Deployment, ReplicaSet, StatefulSet, DaemonSet, Job and CronJob write it for you :

ownerReferences:            # on the Pod
- apiVersion: apps/v1
  kind: ReplicaSet
  name: nginx-deployment-6b474476c4
  uid: <owner uid>
  controller: true          # at most one
  blockOwnerDeletion: true  # default false
Deployment nginx-deployment ReplicaSet nginx-deployment-6b474476c4 Pod ...476c4-a Pod ...476c4-b ownerReferences[0] controller: true ownerReferences controller: true garbage collector kube-controller-manager watches every listable resource, builds the graph deletes it once ALL owners are gone the owner keeps no list of dependents

Arrows point from dependent to owner. Labels pick what a controller manages, owners pick what gets cleaned up.

2.Same namespace, or cluster-scoped

  • A reference has no namespace field, so the owner must sit in the dependent's namespace, or be cluster-scoped.
  • Owner in another namespace : the ref counts as absent, so the GC may delete your dependent.
  • Cluster-scoped dependent with a namespaced owner : never collected (v1.20+). Both raise a Warning event, OwnerRefInvalidNamespace.

3.Cascading deletion : three modes

Deleting an owner picks the order things disappear in. kubectl --cascade maps to DeleteOptions.propagationPolicy.

Mode, kubectl flagOwner getsWhat happens
Background (default)
--cascade=background
no finalizerOwner goes at once. The GC deletes the dependents afterwards. Everyday deletes.
Foreground
--cascade=foreground
foregroundDeletionOwner stays (deletionTimestamp set) until the GC has deleted each dependent with blockOwnerDeletion=true. For teardown that must not lose the owner first.
Orphan
--cascade=orphan
orphanThe GC strips the owner uid from each dependent, then the owner goes. Dependents keep running, so you can swap a controller without killing its Pods.

A raw API DELETE of a batch/v1 Job defaults to orphan. orphanDependents is deprecated. blockOwnerDeletion only matters for foreground, and the plugin that polices it (OwnerReferencesPermissionEnforcement) is off by default in v1.37.

4.Other collectors

Who, knobWhat it collects
Job TTL controller
ttlSecondsAfterFinished (unset = keep)
A finished Job and its Pods, N seconds after it finishes (0 = at once). Stable v1.23
PodGC
--terminated-pod-gc-threshold (12500)
Succeeded / Failed Pods above the count
kubelet GC
  • Images, 5 min : disk over imageGCHighThresholdPercent (85), least recently used first, down to imageGCLowThresholdPercent (80). imageMinimumGCAge 2m, imageMaximumGCAge 0 (off).
  • Containers, 1 min : MinAge 0, MaxPerPodContainer 1, MaxContainers -1 (flags deprecated).

5.Commands

CommandWhat it does
kubectl get pod <pod> -o jsonpath='{.metadata.ownerReferences}'Lists the owners. "controller":true marks the controller
kubectl delete deployment nginx-deployment --cascade=foregroundOwner stays (foregroundDeletion) until its ReplicaSet and Pods are gone
kubectl patch job pi --type=merge -p '{"spec":{"ttlSecondsAfterFinished":600}}'Makes a finished Job eligible for TTL cleanup
In short : The owner is written on the dependent, the garbage collector deletes a dependent once all its owners are gone, and the cascade mode (background, foreground, orphan) sets the order.

1.Finalizers : deletion gates

A finalizer is a key in metadata.finalizers. While any key is left, the object stays.

  • DELETE sets deletionTimestamp and returns 200 OK (202 only for the deprecated orphanDependents=false). Terminating : keys can only be removed, no undelete.
  • Custom keys need a domain, example.com/cleanup (CRs only warn). Bare : kubernetes, orphan, foregroundDeletion.
  • --force --grace-period=0 skips graceful termination, not finalizers.
kubectl kube-apiserver etcd 1. DELETE 3. 200 OK 7. removed 2. set deletionTimestamp Object, Terminating finalizers: [example.com/cleanup] Owning controller External resource 4. watch event 6. remove only its own key, then Update 5. cleanup

2.In your controller

  1. Add your key before making anything external.
  2. deletionTimestamp set ? Clean up (idempotent).
  3. Remove only your own key, then Update.
  4. Failed ? Return the error and retry. The key stays.
  5. RBAC : plain update on the kind. <kind>/finalizers only gates setting blockOwnerDeletion on children.

More : Controllers and Reconciliation, Operators.

Why it hangs
  • Namespace : an aggregated APIService is down (NamespaceDeletionDiscoveryFailure), or CRs outlived their operator (NamespaceFinalizersRemaining).
  • PVC : a Pod still mounts it (pvc-protection).
  • CR : its operator was uninstalled before it.

3.Built-in finalizers you will meet

KeyOn (added by)Gone when
kubernetes.io/pvc-protection,
kubernetes.io/pv-protection
PVC, PV (StorageObjectInUseProtection)No Pod uses the PVC, the PV is unbound
kubernetes.io/pv-controller (in-tree),
external-provisioner.volume.kubernetes.io/finalizer (CSI)
PV with reclaim Delete (v1.33)Backend volume deleted
foregroundDeletion, orphanAny owner (API server)GC is done with the dependents
kubernetes (in spec.finalizers)Namespace, on createAll content gone, Pods first
batch.kubernetes.io/job-trackingJob Pods (v1.26)Job controller counted the Pod

4.Stuck in Terminating ?

A finalizer is a promise that someone will clean up. Find the someone before you delete the promise.

What holds it ?metadata.finalizers
Read the keyprefix names the controller
Check itrunning ? RBAC ? logs ?
Fix itit drops its key

5.Commands

CommandWhat it does
kubectl get pvc -n shop -o custom-columns=NAME:.metadata.name,FINALIZERS:.metadata.finalizersWhat is Terminating, and which key holds it
kubectl get ns shop -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'Why a namespace is stuck. On a DiscoveryFailure, fix the APIService that kubectl get apiservices shows as False
kubectl describe pvc data-postgres-0 -n shopUsed By lists the Pods that keep pvc-protection in place
kubectl patch widgets.example.com w1 -n shop --type=json -p='[{"op":"remove","path":"/metadata/finalizers"}]'Last resort, drops ALL keys. Only if the controller is gone for good and you did its cleanup
In short : A finalizer holds an object until its controller cleans up and drops its own key. When one is stuck, fix the controller before you touch the key.