Identity and trust

OpenBao: Secrets as a Service

1.What is OpenBao ?

  • The open-source fork of HashiCorp Vault, which is BSL 1.1 from 1.15.0. Forked from Vault's release/1.14.x branch (November 2023). 1.14.8 was the last MPL release.
  • MPL-2.0, an OpenSSF Sandbox project. Latest v2.7.1.
  • API compatible, not storage compatible. X-Vault-* headers, vault:v1: ciphertext, vault.* metrics and vault.hashicorp.com/* annotations stay. CLI bao, env BAO_*.
OpenBao 2.7.1Vault 2.1.2
LicenceMPL-2.0BSL 1.1
CLIbaovault
Tokenss. b. r. by defaulthvs. hvb. hvr.
Cloud sealsplugins only, since 2.7built in
Namespacesyes, since 2.3Enterprise only

2.One request path

client log in auth method kubernetes, jwt, oidc, approle ... token : TTL + policies policy check on the path secrets engine barrier (AES-256-GCM) storage who are you ? short-lived deny by default kv, database ... seal guards the keyring
  • Everything is a path : secret/data/app, database/creds/app, pki/issue/web, auth/kubernetes/login, sys/health.
  • Policies are written against paths. No matching rule means no access.
  • Capabilities : create, read, update, patch, delete, list, scan (OpenBao only, 2.2, a recursive list), sudo.
  • deny always wins over every other capability.
path "secret/data/app/*" { capabilities = ["read"] }

3.Engines and auth methods

EngineWhat it doesSee
KV v2Versioned values : secret/data/..., versions secret/metadata/..., keeps 10ESO
databaseA real DB user per lease (v-openbaouse-...), dropped when it endsCloudNativePG
transitEncryption as a service, the key never leaves. aes256-gcm96, vault:v1:...SOPS key
PKIA CA that issues short-lived certificates on requestPrivate PKI
auth methodsapprole, cert, jwt, oidc, kubernetes, userpass, tokenOIDC, Keycloak
not built inLDAP, Kerberos, RADIUS : plugins since 2.7. AWS, GCP, Azure engines : plugins, never built inplugins

4.Tokens, leases, TTLs

  • serviceRenewable, revocable, accessor, children.
  • batchLight. Not renewable, no accessor.
  • periodicRenews forever within its period.
  • rootFrom init, a root, or generate-root.
  • Default and max lease TTL 768h (32 days). Mounts, roles, tokens only go lower.
  • bao lease renew / revoke [-prefix]. sys/renew, sys/revoke* gone in 2.5.4.

5.Sealed by default

  • A new server is sealed : it cannot read its own data.
  • Shamir : init makes 5 shares, threshold 3. Each restart needs 3 on every node.
  • Auto-unseal (KMS plugin, transit, static, kmip) : the node unseals itself. Init gives recovery keys : they authorize generate-root, never decrypt the root key.

GET /v1/sys/health : 200 active, 429 standby, 503 sealed, 501 uninitialized.

In short : OpenBao is the open fork of Vault. A caller logs in with an auth method, gets a short-lived token whose path policies decide what it may touch, and engines serve static or on-demand secrets behind a sealed barrier.

1.Seals in 2.7

  • 2.7.0 moved awskms, gcpckms, azurekeyvault, ocikms, alicloudkms, pkcs11 into KMS plugins.
  • A same-named plugin "kms" shadows the builtin : no seal migration.
  • Built in : transit, static, kmip, Shamir.
  • Shamir to KMS is a real migration : downtime, backup, both seals up, unseal -migrate.
plugin_directory = "/openbao/plugins"
plugin "kms" "awskms" { command = "openbao-plugin-kms-aws" }
seal "awskms" {
  region     = "us-east-2"
  kms_key_id = "<kms-key-id>"
}
Upgrading to 2.7 with a builtin cloud seal leaves the cluster sealed. Add the plugin stanza and binary on 2.6 first.

2.Storage

raft (Integrated Storage) : HA, 3 or 5 voters. postgresql : HA, the docs now suggest it to newcomers. pebbledb (2.7) : durable, not HA, chart standalone default since 0.30.0. file : removed in 2.7.0, a migrate source until 2.8.0.

3.Raft on a StatefulSet

active Service openbao-active=true KMS plugin seal unseal each Pod openbao-0 active + own PVC openbao-1 standby + own PVC openbao-2 standby + own PVC 8201 : standbys forward writes headless Service a DNS name per Pod, the retry_join targets
  • Each node needs a unique node_id. Set it per Pod, or leave it unset and raft makes a UUID in its data dir. Never hardcode one id in a shared ConfigMap.
  • Quorum is a majority of voters : 2 of 3.
  • Standbys forward writes and, since 2.5, serve reads themselves (eventually consistent).
  • service_registration "kubernetes" labels each Pod openbao-active. Without it : no labels, no active-only Service.
  • Pods, PVCs, stable names : StatefulSets.

4.The Helm chart (0.30.3)

SettingDefault in 0.30.3
ModeStandalone with pebbledb. HA with server.ha.enabled and server.ha.raft.enabled
retry_joinNot in the default raft HCL : add it, or join by hand
Update strategyOnDelete, so you choose the order
authDelegatorOn : OpenBao's ServiceAccount may call TokenReview
InjectorOn, with HashiCorp's vault-k8s 1.7.6 image
TLS, liveness, audit storageOff. Not a production default

Optional : a CSI provider, and a snapshot CronJob (chart 0.22.0), aimed at the active Service since 0.23.5.

5.Upgrades, backups, audit

  1. Back up : a raft snapshot. No downgrade guarantee.
  2. Replace standbys one at a time (-l openbao-active=false), the active Pod last.
  3. Never fail over from a newer to an older version.
  • Snapshots come only from the active node : a standby redirects. No snapshot inspect, list with gunzip -c f.snap | tar tv.
  • Audit devices go in the config audit stanza : the API refuses them since 2.3.2 unless an unsafe switch is set. If every audit device fails, requests fail.
CommandWhat it does
helm install openbao openbao/openbao --set server.ha.enabled=true --set server.ha.raft.enabled=true3-replica raft HA install
bao statusSealed ? Active or standby ?
bao operator initInit : Shamir 5 / 3
bao operator raft list-peersVoters, leader, state
bao operator raft snapshot save <file>Back up, on the active node
bao operator unseal [-migrate]Give one unseal share. -migrate in a seal migration
In short : Run three raft voters with unique node ids and auto-unseal, move cloud seals to KMS plugins before 2.7, upgrade standbys first, and take snapshots from the active node.

1.Kubernetes auth

Pod ServiceAccount token OpenBao auth/kubernetes/ login 1 role + JWT 4 token with role's policies kube-apiserver TokenReview API 2 TokenReview 3 valid : SA, ns 5 read secret/data/app/* with the token
bao auth enable kubernetes
bao write auth/kubernetes/config \
  kubernetes_host=https://kubernetes.default.svc
bao policy write app-read app-read.hcl
bao write auth/kubernetes/role/app \
  bound_service_account_names=app \
  bound_service_account_namespaces=app \
  policies=app-read ttl=1h
  • OpenBao calls the TokenReview API, so its own ServiceAccount needs system:auth-delegator. The chart sets it. A hand-written install must add it, or every login is "permission denied".
  • No reviewer token : the client's JWT is the bearer, and each client SA needs the binding instead.
  • OpenBao re-reads its own SA token every minute, so short-lived projected tokens work (RBAC).
  • A role must bind ServiceAccount names and namespaces.
  • Since 2.7.1 and 2.6.4 the SA JWT is checked again on renewal : a revoked SA token can no longer renew.
  • app-read.hcl : a read-only policy on secret/data/app/*, plus read, list on secret/metadata/app/*.

2.JWT auth, SPIFFE and people

  • JWT auth checks a signed token against one issuer per mount (oidc_discovery_url, jwks_url or fixed keys), like the API server (OIDC). No TokenReview call, no early revocation.
  • SPIFFE : a JWT-SVID from the SPIRE agent, posted to auth/jwt/login. Pin bound_subject=spiffe://<td>/ns/<ns>/sa/<sa>, or any SVID with the right audience gets the role. No SPIFFE-specific method (SPIFFE, SPIRE).
  • People : a second JWT/OIDC mount that trusts the IdP (Keycloak).

3.Ways to get a secret out

PathSecret lands asAuthNote
ESOa Secret, per refreshIntervalESO's SAGitOps-friendly. The value sits in etcd
injectorsidecar files, /vault/secrets/the Pod's SAvault.hashicorp.com/*, webhook failurePolicy: Ignore : webhook down, the Pod starts without secrets
CSI driverfiles in a volumethe Pod's SAprovider: openbao, needs the CSI driver
direct loginin the app's memoryKubernetes or JWTbest for leases and dynamic secrets
Flux SOPSkustomize-controller decryptsKubernetes (Flux 2.9)a transit key as SOPS master key
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata: {name: openbao}
spec:
  provider:
    vault:
      server: http://openbao.openbao:8200
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: external-secrets
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
  • ESO's docs point to the vault provider. A newer provider.openBao (camelCase) uses path, not mountPath.
Seen in practice
  • Lagging patch releases : renewal fixes land in patches. Do better : track the current minor's patches.
  • Cloud KMS auto-unseal ties unsealing to cloud credentials. If a client cert supplies them, its CA now guards unsealing too (Private PKI). Do better : alert on that chain and keep a tested recovery path.
  • Each auth mount fails its own way : Kubernetes auth needs the reviewer's system:auth-delegator binding, or every login fails.
  • Snapshots must hit the active node : a standby redirects to the leader's api_addr. Do better : the active Service, TLS end to end.
In short : Pods log in with their ServiceAccount or a SPIFFE JWT-SVID, then get secrets out via ESO, the injector, CSI, the API or Flux SOPS. Pin JWT roles with bound_subject.