Certificates and secrets
cert-manager
1.Why cert-manager ?
A Kubernetes add-on that automatically requests, stores and renews TLS certificates. Latest release : v1.21.3.
- Certs expire (Let's Encrypt certs last 90 days or less)
- Manual way : make a CSR, get it signed, paste into a Secret, remember to repeat. Nobody remembers.
- Kubernetes has no "issue and renew" object. Secrets just hold bytes.
- Adds
CertificateandIssueras CRDs, so a cert is declarative like a Deployment
2.Components
Runs as Deployments in the cert-manager Namespace.
- You declare WHAT (Certificate) and WHO signs (Issuer)
- Output : Secret of type
kubernetes.io/tlswithtls.crt,tls.key(andca.crtif the issuer hands over its CA. ACME does not.) - Pods mount it, Ingress and Gateway point at it
Webhook down ?Applying cert-manager objects fails.
-> Wait, or run
-> Wait, or run
cmctl check apiWait, or run
cmctl check api
The pieces
- controllerThe brain. Reconcile loops for Certificate, CertificateRequest, Order, Challenge and Issuers.
- webhookValidates cert-manager resources. Stamps who asked on each CertificateRequest.
- cainjectorInjects CA bundles into webhook configs, CRDs and APIServices.
- acmesolverSmall pod that answers HTTP-01 challenges.
- startupapicheckInstall Job that checks webhook and API are reachable.
- cmctlCLI : status, renew, approve, check api.
3.Issuers and CRDs
Issuer is namespaced, ClusterIssuer is cluster-wide. Same spec, different scope.
- Built in : SelfSigned, CA, Vault, ACME, Venafi (now CyberArk's)
- External issuers plug in via
issuerRefgroup and kind - ACME issuer needs an account key Secret,
privateKeySecretRef
Scope rule
A Certificate in namespace A cannot use an Issuer in namespace B. Use a ClusterIssuer to share.
| Kind | apiVersion | Scope | What it is |
|---|---|---|---|
| Issuer | cert-manager.io/v1 | Namespaced | Signer, one namespace |
| ClusterIssuer | cert-manager.io/v1 | Cluster | Signer, any namespace |
| Certificate | cert-manager.io/v1 | Namespaced | Desired cert, goes to spec.secretName |
| CertificateRequest | cert-manager.io/v1 | Namespaced | One CSR + result, auto-created |
| Order | acme.cert-manager.io/v1 | Namespaced | ACME order (ACME only) |
| Challenge | acme.cert-manager.io/v1 | Namespaced | One domain proof (ACME only) |
In short : Declare a Certificate and an Issuer. cert-manager's controller, webhook and cainjector do the rest, and the result is a TLS Secret.
1.ACME Issuance Flow
Certificate to Secret, step by step. The error always lives at the bottom of the chain.
Certificate issuance (ACME)
CertificateYou apply it.
CertificateRequestController makes it, with the CSR.
Issuer (ACME)Picked via issuerRef.
OrderOne per cert.
ChallengeOne per domain.
Let's EncryptSolver proves the domain.
Signed certReturned to the Request.
Secrettls.crt and tls.key stored.
CA, Vault and SelfSigned skip Order and Challenge. The Request goes straight to signing.
2.Solvers, Renewal, Ingress
- Renews at about 2/3 of lifetime. Default duration is 90 days.
- Controller watches Certificate + Secret. Expired, missing or mismatched triggers a new Request.
- Annotate an Ingress with
cert-manager.and it makes the Certificate for you (ingress-shim)io/ cluster-issuer - Gateway API : install the Gateway CRDs first (or restart cert-manager after), then set
config.gatewayAPI. enabled=true
| HTTP-01 | DNS-01 | |
|---|---|---|
| Proof | Token at /.well-known/acme-challenge | TXT record in DNS |
| Wildcards | No | Yes |
| Public HTTP route | Needed | Not needed |
| DNS API creds | No | Yes (Route53, Cloudflare) |
| Good for | Simple public sites | Wildcards, internal clusters |
3.Example, Commands and Gotchas
kind: ClusterIssuer
metadata: {name: letsencrypt-staging}
spec:
acme:
email: user@example.com
server: <staging URL>
privateKeySecretRef: {name: le-staging-key}
solvers:
- http01: {ingress: {ingressClassName: nginx}}
kind: Certificate
metadata: {name: example-com, namespace: sandbox}
spec:
secretName: example-com-tls
dnsNames: [example.com, www.example.com]
issuerRef:
{name: letsencrypt-staging, kind: ClusterIssuer}
Gotchas
- Test on the STAGING URL (
acme-staging-v02.). Production has rate limits.api. letsencrypt. org - Stuck cert ? Walk down : Certificate, Request, Order, Challenge
- "failed calling webhook" right after install ? Wait for the webhook
- Renewed Secret does not restart Pods. Reload in the app or use a reloader.
- Gateway listeners need protocol HTTPS or TLS, a hostname,
tls.mode Terminate, certificateRefs in the Gateway's namespace
| Command | Description |
|---|---|
helm install cert-manager oci:// | Install with Helm. crds.enabled=true installs the CRDs. |
kubectl get clusterissuer,issuer,certificate -A | List issuers and certs. READY=True is the goal. |
kubectl describe certificate <name> -n <ns> | Conditions and events of a stuck cert |
cmctl status certificate <name> -n <ns> | Detailed status with related resources |
cmctl renew <name> -n <ns> | Force a renewal now (also --all) |
In short : A stuck cert is solved by walking Certificate, Request, Order, Challenge. Test on staging, and use DNS-01 for wildcards.