Certificates and secrets

cert-manager

1.Why cert-manager ?

A Kubernetes add-on that automatically requests, stores and renews TLS certificates. Latest release : v1.21.3.

  • Certs expire (Let's Encrypt certs last 90 days or less)
  • Manual way : make a CSR, get it signed, paste into a Secret, remember to repeat. Nobody remembers.
  • Kubernetes has no "issue and renew" object. Secrets just hold bytes.
  • Adds Certificate and Issuer as CRDs, so a cert is declarative like a Deployment
Manual certs CSR Secret 3am ! forgot to renew, cert expires With cert-manager Certificate TLS Secret renews itself, forever History 2017202020222024 JetstackSandboxIncubatingGraduated

2.Components

Runs as Deployments in the cert-manager Namespace.

  • You declare WHAT (Certificate) and WHO signs (Issuer)
  • Output : Secret of type kubernetes.io/tls with tls.crt, tls.key (and ca.crt if the issuer hands over its CA. ACME does not.)
  • Pods mount it, Ingress and Gateway point at it
Webhook down ?Applying cert-manager objects fails.
-> Wait, or run cmctl check api
Wait, or run cmctl check api
The pieces
  • controllerThe brain. Reconcile loops for Certificate, CertificateRequest, Order, Challenge and Issuers.
  • webhookValidates cert-manager resources. Stamps who asked on each CertificateRequest.
  • cainjectorInjects CA bundles into webhook configs, CRDs and APIServices.
  • acmesolverSmall pod that answers HTTP-01 challenges.
  • startupapicheckInstall Job that checks webhook and API are reachable.
  • cmctlCLI : status, renew, approve, check api.

3.Issuers and CRDs

Issuer is namespaced, ClusterIssuer is cluster-wide. Same spec, different scope.

  • Built in : SelfSigned, CA, Vault, ACME, Venafi (now CyberArk's)
  • External issuers plug in via issuerRef group and kind
  • ACME issuer needs an account key Secret, privateKeySecretRef
Issuer signs Certificate TLS Secret stored in ClusterIssuer = any namespace
Scope rule A Certificate in namespace A cannot use an Issuer in namespace B. Use a ClusterIssuer to share.
KindapiVersionScopeWhat it is
Issuercert-manager.io/v1NamespacedSigner, one namespace
ClusterIssuercert-manager.io/v1ClusterSigner, any namespace
Certificatecert-manager.io/v1NamespacedDesired cert, goes to spec.secretName
CertificateRequestcert-manager.io/v1NamespacedOne CSR + result, auto-created
Orderacme.cert-manager.io/v1NamespacedACME order (ACME only)
Challengeacme.cert-manager.io/v1NamespacedOne domain proof (ACME only)
In short : Declare a Certificate and an Issuer. cert-manager's controller, webhook and cainjector do the rest, and the result is a TLS Secret.

1.ACME Issuance Flow

Certificate to Secret, step by step. The error always lives at the bottom of the chain.

Certificate issuance (ACME)
CertificateYou apply it.
CertificateRequestController makes it, with the CSR.
Issuer (ACME)Picked via issuerRef.
OrderOne per cert.
ChallengeOne per domain.
Let's EncryptSolver proves the domain.
Signed certReturned to the Request.
Secrettls.crt and tls.key stored.

CA, Vault and SelfSigned skip Order and Challenge. The Request goes straight to signing.

2.Solvers, Renewal, Ingress

  • Renews at about 2/3 of lifetime. Default duration is 90 days.
  • Controller watches Certificate + Secret. Expired, missing or mismatched triggers a new Request.
  • Annotate an Ingress with cert-manager.io/cluster-issuer and it makes the Certificate for you (ingress-shim)
  • Gateway API : install the Gateway CRDs first (or restart cert-manager after), then set config.gatewayAPI.enabled=true
HTTP-01DNS-01
ProofToken at /.well-known/acme-challengeTXT record in DNS
WildcardsNoYes
Public HTTP routeNeededNot needed
DNS API credsNoYes (Route53, Cloudflare)
Good forSimple public sitesWildcards, internal clusters
Ingress annotated controller creates (ingress-shim) Certificate fills Secret served as HTTPS

3.Example, Commands and Gotchas

kind: ClusterIssuer
metadata: {name: letsencrypt-staging}
spec:
  acme:
    email: user@example.com
    server: <staging URL>
    privateKeySecretRef: {name: le-staging-key}
    solvers:
    - http01: {ingress: {ingressClassName: nginx}}
kind: Certificate
metadata: {name: example-com, namespace: sandbox}
spec:
  secretName: example-com-tls
  dnsNames: [example.com, www.example.com]
  issuerRef:
    {name: letsencrypt-staging, kind: ClusterIssuer}
Gotchas
  • Test on the STAGING URL (acme-staging-v02.api.letsencrypt.org). Production has rate limits.
  • Stuck cert ? Walk down : Certificate, Request, Order, Challenge
  • "failed calling webhook" right after install ? Wait for the webhook
  • Renewed Secret does not restart Pods. Reload in the app or use a reloader.
  • Gateway listeners need protocol HTTPS or TLS, a hostname, tls.mode Terminate, certificateRefs in the Gateway's namespace
CommandDescription
helm install cert-manager oci://quay.io/jetstack/charts/cert-manager --version v1.21.3 -n cert-manager --create-namespace --set crds.enabled=trueInstall with Helm. crds.enabled=true installs the CRDs.
kubectl get clusterissuer,issuer,certificate -AList issuers and certs. READY=True is the goal.
kubectl describe certificate <name> -n <ns>Conditions and events of a stuck cert
cmctl status certificate <name> -n <ns>Detailed status with related resources
cmctl renew <name> -n <ns>Force a renewal now (also --all)
In short : A stuck cert is solved by walking Certificate, Request, Order, Challenge. Test on staging, and use DNS-01 for wildcards.