Identity and trust
Private PKI and Trust Distribution
1.Why a private CA ?
- For names and clients no public CA will vouch for : internal names, mTLS client certs, cloud trust anchors (AWS IAM Roles Anywhere).
- Let's Encrypt dropped client certs :
classiclost client auth on 2026-02-11, thetlsclientprofile ended on 2026-07-08. - CA/B Forum ballot SC-081 shrinks public TLS certs : 200 days (2026-03-15), 100 days (2027-03-15), 47 days (2029-03-15). Neither rule binds a private CA.
- An OIDC issuer on a private CA means the API server must trust that root too.
- A real client hits a public hostname : a public CA (Let's Encrypt).
- A workload proves who it is with a cert (mTLS, Roles Anywhere) : the private CA.
- Anything else : no cert at all, TLS ends at the gateway.
2.Hierarchy
Lifetimes are OpenBao's PKI guidance. Keep the root key outside the online signer. For contrast, one real cluster runs a 10-year root, no intermediate and 1-year leaves renewed 30 days early.
3.The cert-manager bootstrap
- A ClusterIssuer's CA Secret lives in the cluster resource namespace (default
cert-manager). - Give the root a real
subject: an empty issuer DN breaks RFC 5280 (BadConfig). - The self-signed issuer should sign only the root.
4.What the CA issuer does not do
cert-manager itself warns : CA issuers "are generally for cert-manager demos or for advanced users with experience and tooling for running a PKI".
| It does not | So you |
|---|---|
| check name constraints or path length when it issues | rely on clients that enforce RFC 5280 at the handshake |
| make CRLs or OCSP responses (it only embeds their URLs) | plan revocation elsewhere, or keep leaves short |
| rotate a CA Secret you made by hand | track its expiry yourself |
| re-issue leaves when the CA Secret changes | run cmctl renew |
| stop leaves outliving the CA | set the leaf duration yourself |
5.Offline root, online intermediate
- Make the root key and cert on an offline machine (
CA: true, up to 10 years). - Sign the intermediate there :
CA: true,pathlen:0, permitted DNS names, marked critical. - Load a Secret :
tls.crt= intermediate then root,tls.key= the intermediate key only (kubectl -n cert-manager create secret tls). - Point a
caClusterIssuer at it. Leaves getca.crt= the root. - Publish only the root to trust stores. Track the intermediate's expiry yourself.
1.Copy the root, never the key
- Copy only the public root into a dedicated ConfigMap or Secret. Not
tls.crt(it can hold intermediates), and nevertls.key. - A dedicated source lets you choose when trust changes roll out. Then fan it out to Pods, nodes and people.
2.Ways to distribute trust
| Option | Moves | Lands as | Key stays out ? |
|---|---|---|---|
| reflector | whole Secret or ConfigMap | same-named copy per namespace | no, copies tls.key |
trust-manager Bundle | chosen certs + public roots | ConfigMap per chosen namespace | yes |
| ClusterTrust | CA certs only | kubelet-written file in the Pod | yes |
kube-root-ca.crt | the cluster's own CA only | ConfigMap in every namespace | yes, not for your CA |
| node store | the root PEM | /etc/ssl/certs on each host | yes |
3.trust-manager
- v0.25.0 ships one CRD,
Bundle(trust.cert-manager.io/v1alpha1, cluster-scoped). Its target ConfigMap takes the Bundle's name. - Sources :
configMap,secret,inLine,useDefaultCAs(pin it : Debian trixie since v0.23.0). - Unset
namespaceSelector= all namespaces today, documented to change. - PKCS#12 target for Java (JKS is deprecated). Secret targets : off by default, grant read on all Secrets.
spec: # a Bundle : current + previous root
sources:
- useDefaultCAs: true
- configMap: {name: root-current, key: ca.crt}
- configMap: {name: root-previous, key: ca.crt}
target:
configMap: {key: ca-bundle.pem}
additionalFormats: {pkcs12: {key: bundle.p12}}
namespaceSelector: {matchLabels: {trust: corp}}
4.ClusterTrustBundle
- GA in v1.37 (
certificates.k8s.io/v1) : cluster-scoped, CA certs only, readable by every ServiceAccount.v1beta1: deprecated, never served by default. - Signer-unlinked : no colon, plain RBAC. Signer-linked : the name starts with the signer name (
/written as:) then a colon, e.g.example.com:mysigner:foo; writing needsatteston the signer (admissionctbattest). - Mount a projected source by
name, orsignerNamepluslabelSelector:{}matches all, unset none; the Pod will not start unlessoptional: true(docs wrong). - The kubelet keeps the file updated; trust-manager cannot write one yet. Pod Certificates signers are expected to publish here.
kind: ClusterTrustBundle # v1, no colon in name
metadata: {name: corp-root}
spec: {trustBundle: "-----BEGIN CERTIFICATE-----..."}
- clusterTrustBundle: {name: corp-root, path: ca.crt}
5.Runtimes read trust differently
| Runtime | Add a private root | Watch out |
|---|---|---|
| Debian, Ubuntu | a .crt in /usr/local/share/, then update-ca-certificates | other file extensions are silently ignored |
| Go | SSL_CERT_FILE or SSL_CERT_DIR | the pool is cached per process |
| Node.js | NODE_EXTRA_CA_CERTS | read only when the process starts |
| Java | keytool -importcert -cacerts, or a PKCS#12 truststore | restart the JVM |
Python requests : REQUESTS_CA_BUNDLE (certifi, not the OS store). k3s image pulls : registries.yaml tls.ca_file, restart k3s per node.
A root served over plain HTTP is fine only if people compare a published fingerprint; openssl x509 -noout -fingerprint prints SHA-1 without -sha256. Pitfall : reflector mirroring the CA Secret copies the root key too. Do better : publish the SHA-256 fingerprint out of band; fan out only ca.crt.
1.Rotating a root with overlap
- Same subject, different key : old leaves do not validate ("ECDSA verification failure").
- So the new root reaches every store first, and the old one stays until its last leaf expires.
- In OpenBao's words, the hard part is rotating root certificates. It can take months.
2.The rotationPolicy trap
Since cert-manager v1.18.0 a renewal makes a new private key by default (rotationPolicy: Always, locked since v1.20.0). On a CA that is a new root key : trust-manager reading that Secret swaps the root at once.
Always (default) | Never | |
|---|---|---|
| key at renewal | new key and key id, same subject | reuses tls.key; subject, key id unchanged |
| new leaves vs old root cert | fail | verify, until the old notAfter |
| trust stores | need the new root before leaves move | need the renewed cert before the old expires |
| key compromise | each renewal limits exposure | key lives as long as the hierarchy |
Never buys time, not freedom. OpenSSL and Go reject the chain once the old root cert passes notAfter. Fixes : two Bundle sources (current, previous), or Never plus the renewed cert in every store within renewBefore.3.OpenBao PKI as the issuer
- Root outside, intermediate in OpenBao (
internal: no key export;leaf_not_after_behaviorerrortruncate). Classic endpoints : onlypermitted_dns_domains(CEL role templates : full constraints). - cert-manager's
vaultissuer calls the sign path,pki_int/sign/<role>. Roles limit names (allowed_domains). serviceAccountRef: 10-minute tokens for a dedicated ServiceAccount, via a Role tocreateserviceaccounts/token.
kind: ClusterIssuer # cert-manager.io/v1
metadata: {name: bao-int}
spec:
vault:
server: https://bao.example.com:8200
path: pki_int/sign/internal
caBundleSecretRef: {name: bao-ca, key: ca.crt}
auth: {kubernetes: {role: cert-manager,
serviceAccountRef: {name: vault-issuer}}}
Same Vault API, so expected to work, but neither project documents or tests OpenBao with this issuer.
4.Issuer choices
CA issuer : no extra service, key in a Secret, no CRL. Vault issuer + OpenBao : key stays inside, CRLs, one more service. SPIRE : short-lived, SPIFFE IDs, not DNS names. PodCertificateRequest : native, you deploy the signer.
5.Commands and one real cluster
| Command | What it does |
|---|---|
| kubectl -n cert-manager get secret <ca> -o | show CA:TRUE, pathlen, dates |
| kubectl -n cert-manager get certificate <ca> -o | when it will renew, and re-key |
| cmctl renew -n <ns> <certificate> | re-issue now (leaves after a CA change) |
| openssl verify -CAfile root.crt -untrusted int.crt leaf.crt | check a chain |
- Pitfall : one online root that is also an outside trust anchor, for example a cloud IAM Roles Anywhere anchor behind SOPS decryption and OpenBao auto-unseal. When cert-manager renews it with a new key, the outside copy goes stale silently and everything behind it fails at once.
- Do better : an offline root with an online intermediate, alerts on CA expiry and on anchor drift, and a rehearsed rotation with two anchors overlapping.
- Deleting a CA Secret re-keys it at once. Guard it with RBAC and a backup.
Never needs the renewed cert in every store in time.