Extending the API

Operators

1.What is an operator ?

An operator is CRDs + a custom controller. It turns a human operator's runbook (install, upgrade, backup, failover) into a reconcile loop that runs forever.

  • k8s.io : "software extensions to Kubernetes that make use of custom resources to manage applications and their components." The idea came from a CoreOS blog post.
  • Every operator is a controller, not every controller is an operator. The Deployment controller is generic. CloudNativePG knows Postgres : promotion, WAL archiving, backups.

2.What it automates

CRDs + controller + runbook = Operator new kinds watch, diff, act ops know-how runs forever
Jobs k8s.io lists
  • Deploy an app on demand
  • Take and restore backups
  • Upgrades with schema or config changes
  • Publish a Service for apps that cannot use the Kubernetes API
  • Simulate failure to test resilience
  • Leader election for apps that have none

3.Anatomy (k8s.io SampleDB)

Deploy it like any app : CRDs plus a controller Deployment, outside the control plane.

  • Users touch only the CR : kubectl edit SampleDB/example-database.
  • You write .spec (a change bumps metadata.generation). The operator writes .status.conditions with observedGeneration.
  • Children carry an ownerReference, so a StatefulSet change re-runs reconcile.
  • Delete the SampleDB : a finalizer takes a snapshot, then the StatefulSet and volumes go.
You / Flux apply .spec SampleDB CR .spec : you .status : operator kube-apiserver every write goes through here watch SampleDB + owned kinds writes .status Operator Pod controller-manager status change re-runs it (Owns) create / update + ownerRef StatefulSet PVCs Init Job Backup Pod Upgrade Job first run on schedule old version

Every write goes through kube-apiserver. Arrows to the children are shorthand.

4.Capability levels

L5 Auto Pilotauto-scale, self-heal, auto-tune L4 Deep Insightsmetrics, alerts, events L3 Full Lifecyclebackup, restore, failover L2 Seamless Upgradesoperand + operator upgrades L1 Basic Installinstall via CR, ready in status L1 to L2 L1 to L5 L1 to L5 Helm Ansible Go
In these notesLevel*
CloudNativePG5 Auto Pilot
External Secrets4 Deep Insights
OTel Operator4 Deep Insights
cert-manager3 Full Lifecycle
Knative Operator2 Seamless Upgrades
Flux CD2 Seamless Upgrades

* OperatorHub level, self-declared by the publisher in the CSV (metadata.annotations.capabilities). Read the docs before you trust it.

5.Write one, or not ?

You needReach forWhy
Install / upgrade a stateless appHelm chart + Flux HelmReleaseLevels 1-2, no Go code to maintain
App reads a config fileConfigMapk8s.io : a file a Pod reads is not a CR
Logic on built-in kindsPlain controller, --resource=falseNo new API needed
Validate or default fieldsCRD schema + CEL, or admission policyNo webhook (CRDs, Admission)
Parent-to-children mappingMetacontrollerOne JSON webhook, it does the watching
Non-declarative APIA stand-alone APIk8s.io : CRDs and aggregation fit declarative only
It already existsInstall itSomeone else carries the pager
Backup, failover, ordered upgradesWrite an operatorLevel 3+ is why operators exist
In short : An operator is CRDs plus a controller that runs a runbook as a reconcile loop : you write .spec, it owns the children and reports .status. Write one only for level 3+ work.

1.One reconcile pass

Request : ns/name Get CR return nilNotFound : gone being deleted ?deletionTimestamp set yes Cleanupfinal backup no Add finalizer Remove finalizer CreateOrUpdate children+ SetControllerReference list empty API serverdeletes the CR Set status conditions periodic work return RequeueAfter return errrequeue with backoff

Red dashed : a step failed, Result ignored, retry with backoff. Each pass reads the current state, never "what changed". Finalizer and ownerReference rules : Owners, Finalizers and GC.

2.The calls (controller-runtime v0.25)

func (r *SampleDBReconciler) Reconcile(ctx context.Context,
    req ctrl.Request) (res ctrl.Result, err error) {
  var db dbv1.SampleDB
  if err = r.Get(ctx, req.NamespacedName, &db); err != nil {
    return res, client.IgnoreNotFound(err) } // gone : nil
  if db.DeletionTimestamp != nil { // final backup first
    controllerutil.RemoveFinalizer(&db, fin)
    return res, r.Update(ctx, &db) }
  if controllerutil.AddFinalizer(&db, fin) {
    return res, r.Update(ctx, &db) }
  sts := child(&db) // *appsv1.StatefulSet, name + namespace
  _, err = controllerutil.CreateOrUpdate(ctx, r.Client, sts,
    func() error { // set sts.Spec, then own it
      return ctrl.SetControllerReference(&db, sts, r.Scheme) })
  if err != nil { return res, err }
  meta.SetStatusCondition(&db.Status.Conditions, ready(&db))
  res.RequeueAfter = time.Hour // periodic work
  return res, r.Status().Update(ctx, &db)
}

Builder : For(&dbv1.SampleDB{}).Owns(&appsv1.StatefulSet{}), a child change requeues. Requeue: true is deprecated, reconcile.TerminalError(err) stops retries.

3.Pick a toolkit

Tool (language)LatestYou writeUsed here / notes
controller-runtime (Go)v0.25.2Reconcile(ctx, req) (Result, error)Flux CD, CNPG, ESO
Kubebuilder (Go)v4.16.0Types, markers, ReconcileCNPG, ESO, OTel
Operator SDK (Go, Ansible, Helm)v1.42.3Go via Kubebuilder, a playbook or a chartOTel plugins. Helm type : level 2 max
Java Operator SDK (Java)v5.6.2A Reconciler class (fabric8 client)Quarkus extension
Kopf (Python 3.11+)1.44.6@kopf.on.create, timer, daemonFeature-complete, stable
kube-rs (Rust)4.2.0async fn reconcile(obj, ctx)In kube::runtime
Metacontroller (any)v4.17.2A JSON sync webhookAPI still v1alpha1
OLM v1 operator-controllerv1.12.0A ClusterExtension : package + channelOperatorHub.io catalog, cluster-admin

4.Build, test, ship (Kubebuilder)

CommandDoes
kubebuilder init --domain <domain> --repo <go-module>Scaffold the project and a Kind e2e suite
kubebuilder create api --group db --version v1alpha1 --kind SampleDBCRD Go type, controller, envtest suite
make manifests generateCRD YAML + ClusterRole from markers, DeepCopy
make install runCRDs in, controller runs on your laptop
make testenvtest : etcd + kube-apiserver from setup-envtest (KUBEBUILDER_ASSETS)
make test-e2eKind cluster + Ginkgo, deleted on pass
make docker-build docker-push IMG=<image> && make deploy IMG=<image>Push image, deploy CRDs, RBAC, Deployment

5.Gotchas & good to know

Gotchas
  • Finalizer deadlock : remove the operator before its CRs and they hang in Terminating.
  • ownerReferences cannot cross namespaces (OwnerRefInvalidNamespace events).
  • envtest has no controller-manager : no GC, no Pods. Assert ownerReferences, .status.
Good to know
  • Keep --leader-elect on before replicas > 1.
  • Upgrade the operator first. It only knows operands up to its own version.
  • Operator down : the app keeps serving, only automation pauses.
  • OLM v1 installs, then upgrades along the channel.
In short : Reconcile is an idempotent pass : get, finalize, create-or-update owned children, write status. Scaffold with Kubebuilder, test on envtest, ship with kustomize, OLM v1 or Helm.