Identity and trust

Private PKI and Trust Distribution

1.Why a private CA ?

  • For names and clients no public CA will vouch for : internal names, mTLS client certs, cloud trust anchors (AWS IAM Roles Anywhere).
  • Let's Encrypt dropped client certs : classic lost client auth on 2026-02-11, the tlsclient profile ended on 2026-07-08.
  • CA/B Forum ballot SC-081 shrinks public TLS certs : 200 days (2026-03-15), 100 days (2027-03-15), 47 days (2029-03-15). Neither rule binds a private CA.
  • An OIDC issuer on a private CA means the API server must trust that root too.
Which issuer ?
  • A real client hits a public hostname : a public CA (Let's Encrypt).
  • A workload proves who it is with a cert (mTLS, Roles Anywhere) : the private CA.
  • Anything else : no cert at all, TLS ends at the gateway.

2.Hierarchy

root CA offline if you can, 2 to 10 years signs intermediate, one per purpose 6 months to 2 years pathlen:0, name constraints signs leaf leaf leaf leaves : 30 to 90 days single tier : the root signs leaves itself, so the root key stays online

Lifetimes are OpenBao's PKI guidance. Keep the root key outside the online signer. For contrast, one real cluster runs a 10-year root, no intermediate and 1-year leaves renewed 30 days early.

3.The cert-manager bootstrap

ClusterIssuer selfSigned signs Certificate isCA: true writes Secret in cert-manager tls.crt, tls.key, ca.crt secretName ClusterIssuer ca signs leaf Certificates
  • A ClusterIssuer's CA Secret lives in the cluster resource namespace (default cert-manager).
  • Give the root a real subject : an empty issuer DN breaks RFC 5280 (BadConfig).
  • The self-signed issuer should sign only the root.

4.What the CA issuer does not do

cert-manager itself warns : CA issuers "are generally for cert-manager demos or for advanced users with experience and tooling for running a PKI".

It does notSo you
check name constraints or path length when it issuesrely on clients that enforce RFC 5280 at the handshake
make CRLs or OCSP responses (it only embeds their URLs)plan revocation elsewhere, or keep leaves short
rotate a CA Secret you made by handtrack its expiry yourself
re-issue leaves when the CA Secret changesrun cmctl renew
stop leaves outliving the CAset the leaf duration yourself

5.Offline root, online intermediate

  1. Make the root key and cert on an offline machine (CA: true, up to 10 years).
  2. Sign the intermediate there : CA: true, pathlen:0, permitted DNS names, marked critical.
  3. Load a Secret : tls.crt = intermediate then root, tls.key = the intermediate key only (kubectl -n cert-manager create secret tls).
  4. Point a ca ClusterIssuer at it. Leaves get ca.crt = the root.
  5. Publish only the root to trust stores. Track the intermediate's expiry yourself.
In short : A private CA covers internal names and client certs. cert-manager bootstraps one from a self-signed issuer, but its CA issuer checks no constraints, makes no CRLs and re-issues nothing when the CA changes.

1.Copy the root, never the key

  • Copy only the public root into a dedicated ConfigMap or Secret. Not tls.crt (it can hold intermediates), and never tls.key.
  • A dedicated source lets you choose when trust changes roll out. Then fan it out to Pods, nodes and people.
root CA Secret tls.crt, tls.key, ca.crt reflector whole copy, has tls.key copy, no key root cert only trust-manager to ConfigMaps Pods mount ClusterTrustBundle kubelet writes a file in the Pod plain HTTP SHA-256 check laptops, nodes

2.Ways to distribute trust

OptionMovesLands asKey stays out ?
reflectorwhole Secret or ConfigMapsame-named copy per namespaceno, copies tls.key
trust-manager Bundlechosen certs + public rootsConfigMap per chosen namespaceyes
ClusterTrustBundleCA certs onlykubelet-written file in the Podyes
kube-root-ca.crtthe cluster's own CA onlyConfigMap in every namespaceyes, not for your CA
node storethe root PEM/etc/ssl/certs on each hostyes

3.trust-manager

  • v0.25.0 ships one CRD, Bundle (trust.cert-manager.io/v1alpha1, cluster-scoped). Its target ConfigMap takes the Bundle's name.
  • Sources : configMap, secret, inLine, useDefaultCAs (pin it : Debian trixie since v0.23.0).
  • Unset namespaceSelector = all namespaces today, documented to change.
  • PKCS#12 target for Java (JKS is deprecated). Secret targets : off by default, grant read on all Secrets.
spec:  # a Bundle : current + previous root
  sources:
  - useDefaultCAs: true
  - configMap: {name: root-current, key: ca.crt}
  - configMap: {name: root-previous, key: ca.crt}
  target:
    configMap: {key: ca-bundle.pem}
    additionalFormats: {pkcs12: {key: bundle.p12}}
    namespaceSelector: {matchLabels: {trust: corp}}

4.ClusterTrustBundle

  • GA in v1.37 (certificates.k8s.io/v1) : cluster-scoped, CA certs only, readable by every ServiceAccount. v1beta1 : deprecated, never served by default.
  • Signer-unlinked : no colon, plain RBAC. Signer-linked : the name starts with the signer name (/ written as :) then a colon, e.g. example.com:mysigner:foo; writing needs attest on the signer (admission ctbattest).
  • Mount a projected source by name, or signerName plus labelSelector : {} matches all, unset none; the Pod will not start unless optional: true (docs wrong).
  • The kubelet keeps the file updated; trust-manager cannot write one yet. Pod Certificates signers are expected to publish here.
kind: ClusterTrustBundle  # v1, no colon in name
metadata: {name: corp-root}
spec: {trustBundle: "-----BEGIN CERTIFICATE-----..."}
- clusterTrustBundle: {name: corp-root, path: ca.crt}

5.Runtimes read trust differently

RuntimeAdd a private rootWatch out
Debian, Ubuntua .crt in /usr/local/share/ca-certificates/, then update-ca-certificatesother file extensions are silently ignored
GoSSL_CERT_FILE or SSL_CERT_DIRthe pool is cached per process
Node.jsNODE_EXTRA_CA_CERTSread only when the process starts
Javakeytool -importcert -cacerts, or a PKCS#12 truststorerestart the JVM

Python requests : REQUESTS_CA_BUNDLE (certifi, not the OS store). k3s image pulls : registries.yaml tls.ca_file, restart k3s per node.

Seen in practice

A root served over plain HTTP is fine only if people compare a published fingerprint; openssl x509 -noout -fingerprint prints SHA-1 without -sha256. Pitfall : reflector mirroring the CA Secret copies the root key too. Do better : publish the SHA-256 fingerprint out of band; fan out only ca.crt.

In short : Copy only the public root into a dedicated source, then fan it out with trust-manager or a ClusterTrustBundle, and set each runtime's trust knob. reflector copies the private key too.

1.Rotating a root with overlap

  • Same subject, different key : old leaves do not validate ("ECDSA verification failure").
  • So the new root reaches every store first, and the old one stays until its last leaf expires.
  • In OpenBao's words, the hard part is rotating root certificates. It can take months.
old root trusted new root trusted 1. new root made, new key 2. added to every trust store 3. wait for propagation 4. issuer switched, leaves re-issued 5. old root removed after last leaf

2.The rotationPolicy trap

Since cert-manager v1.18.0 a renewal makes a new private key by default (rotationPolicy: Always, locked since v1.20.0). On a CA that is a new root key : trust-manager reading that Secret swaps the root at once.

Always (default)Never
key at renewalnew key and key id, same subjectreuses tls.key; subject, key id unchanged
new leaves vs old root certfailverify, until the old notAfter
trust storesneed the new root before leaves moveneed the renewed cert before the old expires
key compromiseeach renewal limits exposurekey lives as long as the hierarchy
Never buys time, not freedom. OpenSSL and Go reject the chain once the old root cert passes notAfter. Fixes : two Bundle sources (current, previous), or Never plus the renewed cert in every store within renewBefore.

3.OpenBao PKI as the issuer

  • Root outside, intermediate in OpenBao (internal : no key export; leaf_not_after_behavior err or truncate). Classic endpoints : only permitted_dns_domains (CEL role templates : full constraints).
  • cert-manager's vault issuer calls the sign path, pki_int/sign/<role>. Roles limit names (allowed_domains).
  • serviceAccountRef : 10-minute tokens for a dedicated ServiceAccount, via a Role to create serviceaccounts/token.
kind: ClusterIssuer  # cert-manager.io/v1
metadata: {name: bao-int}
spec:
  vault:
    server: https://bao.example.com:8200
    path: pki_int/sign/internal
    caBundleSecretRef: {name: bao-ca, key: ca.crt}
    auth: {kubernetes: {role: cert-manager,
      serviceAccountRef: {name: vault-issuer}}}

Same Vault API, so expected to work, but neither project documents or tests OpenBao with this issuer.

4.Issuer choices

CA issuer : no extra service, key in a Secret, no CRL. Vault issuer + OpenBao : key stays inside, CRLs, one more service. SPIRE : short-lived, SPIFFE IDs, not DNS names. PodCertificateRequest : native, you deploy the signer.

5.Commands and one real cluster

CommandWhat it does
kubectl -n cert-manager get secret <ca> -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -subject -dates -ext basicConstraints,keyUsageshow CA:TRUE, pathlen, dates
kubectl -n cert-manager get certificate <ca> -o jsonpath='{.status.renewalTime}'when it will renew, and re-key
cmctl renew -n <ns> <certificate>re-issue now (leaves after a CA change)
openssl verify -CAfile root.crt -untrusted int.crt leaf.crtcheck a chain
Seen in practice
  • Pitfall : one online root that is also an outside trust anchor, for example a cloud IAM Roles Anywhere anchor behind SOPS decryption and OpenBao auto-unseal. When cert-manager renews it with a new key, the outside copy goes stale silently and everything behind it fails at once.
  • Do better : an offline root with an online intermediate, alerts on CA expiry and on anchor drift, and a rehearsed rotation with two anchors overlapping.
  • Deleting a CA Secret re-keys it at once. Guard it with RBAC and a backup.
In short : Rotate a root with overlap : add the new one everywhere, re-issue, drop the old. cert-manager re-keys a CA on renewal, and even Never needs the renewed cert in every store in time.