Extending the API

Custom Resources (CRDs)

1.What is a CRD ?

A CustomResourceDefinition teaches kube-apiserver a new kind, with its schema, validation, defaults and versions. The API server serves and stores the objects itself.

  • The CRD is cluster-scoped (apiextensions.k8s.io/v1). Its metadata.name must be <plural>.<group>.
  • spec.scope (Namespaced or Cluster) is for the custom objects, not the CRD.
  • A CRD only stores data. A controller acts on it (Operators).
kubectl 1. apply crd.yaml 2. get ct kube-apiserver CRD handler CRD status NamesAccepted, Established Discovery kind, plural, shortNames OpenAPI v3 schema read by kubectl explain REST path /apis/stable.example.com/v1/ namespaces/*/crontabs storage version etcd publishes serves

Use the kind once the CRD is Established.

CRD or aggregated API ?CRD : kube-apiserver serves it and stores it in etcd, with no server to run and validation from the schema and CEL. Aggregated API : an APIService routes /apis/<group>/<version> to your own server (like metrics.k8s.io from metrics-server), which picks its own storage, verbs and subresources. Start with a CRD.

2.A CRD in YAML

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com # plural.group
spec:
  group: stable.example.com
  scope: Namespaced       # of the CronTabs
  names:
    kind: CronTab
    plural: crontabs      # URL segment
    shortNames: ["ct"]
    categories: ["all"]   # kubectl get all
# then, an object of the new kind
apiVersion: stable.example.com/v1
kind: CronTab
metadata: {name: nightly}
spec: {cronSpec: "0 2 * * *", image: "backup:1.4"}
  versions:
  - name: v1
    served: true
    storage: true         # exactly one
    subresources:
      status: {}
      scale:
        specReplicasPath: .spec.replicas
        statusReplicasPath: .status.replicas
    additionalPrinterColumns:
    - name: Image
      type: string
      jsonPath: .spec.image
      priority: 1         # only in -o wide
    schema:
      openAPIV3Schema: {...}  # structural
  conversion:
    strategy: None        # the default

3.Field by field

FieldRule / defaultPurpose
names.singular, names.listKinddefault lowercased kind, <Kind>ListCLI alias, kind of list responses
names.shortNames, names.categoriesoptional, lowercasekubectl get ct, kubectl get all
versions[].served, storagerequired, exactly one storage: trueserve over REST, version written to etcd
versions[].schema.openAPIV3Schemastructuralvalidation, pruning, defaulting
versions[].subresourcesstatus: {} and/or scale/status for controllers, /scale for kubectl scale and HPA (labelSelectorPath)
versions[].additionalPrinterColumnsnone = NAME, AGEkubectl get columns
versions[].selectableFieldsmax 8--field-selector on custom fields
versions[].deprecateddefault falsewarning header on requests
spec.conversion.strategyNone (default) or Webhookhow objects convert between versions
status.storedVersionsAPI server adds, migration trimsevery version ever stored
In short : A CRD adds a kind (names, scope, versions, schema) that kube-apiserver serves.

1.Structural schema and pruning

  • Structural (a must in v1) : a type on every node, no type or default inside allOf, anyOf, oneOf, not.
  • Pruning drops unknown fields (API warns, kubectl rejects). x-kubernetes-preserve-unknown-fields opts a subtree out.
  • default: on request, on read and after a webhook patch. nullable: true keeps a null.
  • x-kubernetes-list-type (atomic by default, set, map) and x-kubernetes-list-map-keys tell server-side apply how to merge a list.
Requestserved version Decode, pruneunknown fields Defaultsrequest version Mutatingadmission Defaults againif webhook patched Validationschema + CEL ratcheting on update Validatingadmission Convertto storage version etcd

Dashed : admission. On read : storage-version defaults, then conversion.

2.CEL validation rules

image:
  type: string
  maxLength: 256        # bounds the CEL cost
  x-kubernetes-validations:
  - rule: "self == oldSelf"   # transition
    message: "image is immutable"
replicas: {type: integer, default: 1}

A rule also takes messageExpression, reason, fieldPath, optionalOldSelf.

  • x-kubernetes-validations holds CEL rules (Stable 1.29). self is the node the rule sits on.
  • Uses oldSelf ? It is a transition rule, run only on updates where old and new both exist.
  • optionalOldSelf: true (Stable 1.33) runs it on create too.
  • Ratcheting (Stable 1.33, on) passes an update if the invalid part did not change.

3.Versions and conversion

  • Exactly one version has storage: true. Each has its own schema, subresources and columns.
  • None (default) only rewrites apiVersion. A renamed or split field needs a Webhook.
  • status.storedVersions lists every version ever stored. Drop a version only once it has left the list.
  • v1.37 : StorageVersionMigration is GA and on by default. It rewrites every object at the storage version.
Client Aasks for v1beta1 Client Basks for v1 kube-apiserver ConversionReview Conversion webhook :443, https writes at v1 etcd objects at v1beta1 and v1 storedVersions [v1beta1, v1], then [v1] StorageVersionMigration GA in v1.37 rewrites at v1 once Succeeded

Changing storage: true never rewrites old objects. Real example : Gateway API CRDs (channels, v1beta1 to v1, storedVersions).

4.Commands

CommandDoes
kubectl wait --for=condition=Established crd/<name>Block until the new endpoint is served
kubectl api-resources --api-group=stable.example.comKinds, shortNames and scope the CRD added
kubectl explain crontabs.spec --api-version=stable.example.com/v1Field docs from the published OpenAPI v3 schema
kubectl apply --dry-run=server -f my-crontab.yamlFull server check (schema, CEL, admission), nothing saved
kubectl get ct -o widePrinter columns, plus those with priority > 0
kubectl get crd <name> -o jsonpath='{.status.storedVersions}'Versions that may still have objects in etcd
Gotchas
  • Deleting a CRD deletes every object of that kind.
  • With status on, a normal update ignores .status. Use --subresource=status.
  • A failing conversion webhook can block reads, updates, even deletes.
Good to know
  • Most RBAC roles miss new kinds. Add Roles.
  • *.k8s.io and *.kubernetes.io groups need the api-approved.kubernetes.io annotation.
  • CEL over budget rejects the CRD. Bound it with maxItems, maxProperties, maxLength.
In short : A structural schema lets the API server prune, default and check every write with CEL. Versions share one storage version, and a StorageVersionMigration moves old objects to it.