Identity and trust
OpenBao: Secrets as a Service
1.What is OpenBao ?
- The open-source fork of HashiCorp Vault, which is BSL 1.1 from 1.15.0. Forked from Vault's
release/branch (November 2023). 1.14.8 was the last MPL release.1.14.x - MPL-2.0, an OpenSSF Sandbox project. Latest v2.7.1.
- API compatible, not storage compatible.
X-Vault-*headers,vault:v1:ciphertext,vault.*metrics andvault.annotations stay. CLIhashicorp.com/* bao, envBAO_*.
| OpenBao 2.7.1 | Vault 2.1.2 | |
|---|---|---|
| Licence | MPL-2.0 | BSL 1.1 |
| CLI | bao | vault |
| Tokens | s. b. r. by default | hvs. hvb. hvr. |
| Cloud seals | plugins only, since 2.7 | built in |
| Namespaces | yes, since 2.3 | Enterprise only |
2.One request path
- Everything is a path :
secret/,data/ app database/,creds/ app pki/,issue/ web auth/,kubernetes/ login sys/.health - Policies are written against paths. No matching rule means no access.
- Capabilities :
create,read,update,patch,delete,list,scan(OpenBao only, 2.2, a recursive list),sudo. denyalways wins over every other capability.
path "secret/data/app/*" { capabilities = ["read"] }
3.Engines and auth methods
| Engine | What it does | See |
|---|---|---|
| KV v2 | Versioned values : secret/, versions secret/, keeps 10 | ESO |
| database | A real DB user per lease (v-openbaouse-...), dropped when it ends | CloudNativePG |
| transit | Encryption as a service, the key never leaves. aes256-gcm96, vault:v1:... | SOPS key |
| PKI | A CA that issues short-lived certificates on request | Private PKI |
| auth methods | approle, cert, jwt, oidc, kubernetes, userpass, token | OIDC, Keycloak |
| not built in | LDAP, Kerberos, RADIUS : plugins since 2.7. AWS, GCP, Azure engines : plugins, never built in | plugins |
4.Tokens, leases, TTLs
- serviceRenewable, revocable, accessor, children.
- batchLight. Not renewable, no accessor.
- periodicRenews forever within its period.
- rootFrom init, a root, or
generate-root.
- Default and max lease TTL 768h (32 days). Mounts, roles, tokens only go lower.
bao lease renew/revoke [-prefix].sys/renew,sys/revoke*gone in 2.5.4.
5.Sealed by default
- A new server is sealed : it cannot read its own data.
- Shamir : init makes 5 shares, threshold 3. Each restart needs 3 on every node.
- Auto-unseal (KMS plugin,
transit,static,kmip) : the node unseals itself. Init gives recovery keys : they authorizegenerate-root, never decrypt the root key.
GET /v1/sys/health : 200 active, 429 standby, 503 sealed, 501 uninitialized.
In short : OpenBao is the open fork of Vault. A caller logs in with an auth method, gets a short-lived token whose path policies decide what it may touch, and engines serve static or on-demand secrets behind a sealed barrier.
1.Seals in 2.7
- 2.7.0 moved
awskms,gcpckms,azurekeyvault,ocikms,alicloudkms,pkcs11into KMS plugins. - A same-named
plugin "kms"shadows the builtin : no seal migration. - Built in :
transit,static,kmip, Shamir. - Shamir to KMS is a real migration : downtime, backup, both seals up,
unseal -migrate.
plugin_directory = "/openbao/plugins"
plugin "kms" "awskms" { command = "openbao-plugin-kms-aws" }
seal "awskms" {
region = "us-east-2"
kms_key_id = "<kms-key-id>"
}
Upgrading to 2.7 with a builtin cloud seal leaves the cluster sealed. Add the plugin stanza and binary on 2.6 first.
2.Storage
raft (Integrated Storage) : HA, 3 or 5 voters. postgresql : HA, the docs now suggest it to newcomers. pebbledb (2.7) : durable, not HA, chart standalone default since 0.30.0. file : removed in 2.7.0, a migrate source until 2.8.0.
3.Raft on a StatefulSet
- Each node needs a unique
node_id. Set it per Pod, or leave it unset and raft makes a UUID in its data dir. Never hardcode one id in a shared ConfigMap. - Quorum is a majority of voters : 2 of 3.
- Standbys forward writes and, since 2.5, serve reads themselves (eventually consistent).
service_labels each Podregistration "kubernetes" openbao-active. Without it : no labels, no active-only Service.- Pods, PVCs, stable names : StatefulSets.
4.The Helm chart (0.30.3)
| Setting | Default in 0.30.3 |
|---|---|
| Mode | Standalone with pebbledb. HA with server. and server. |
retry_join | Not in the default raft HCL : add it, or join by hand |
| Update strategy | OnDelete, so you choose the order |
authDelegator | On : OpenBao's ServiceAccount may call TokenReview |
| Injector | On, with HashiCorp's vault-k8s 1.7.6 image |
| TLS, liveness, audit storage | Off. Not a production default |
Optional : a CSI provider, and a snapshot CronJob (chart 0.22.0), aimed at the active Service since 0.23.5.
5.Upgrades, backups, audit
- Back up : a raft snapshot. No downgrade guarantee.
- Replace standbys one at a time (
-l openbao-active=false), the active Pod last. - Never fail over from a newer to an older version.
- Snapshots come only from the active node : a standby redirects. No
snapshot inspect, list withgunzip -c f.snap | tar tv. - Audit devices go in the config
auditstanza : the API refuses them since 2.3.2 unless an unsafe switch is set. If every audit device fails, requests fail.
| Command | What it does |
|---|---|
| helm install openbao openbao/ | 3-replica raft HA install |
| bao status | Sealed ? Active or standby ? |
| bao operator init | Init : Shamir 5 / 3 |
| bao operator raft list-peers | Voters, leader, state |
| bao operator raft snapshot save <file> | Back up, on the active node |
| bao operator unseal [-migrate] | Give one unseal share. -migrate in a seal migration |
In short : Run three raft voters with unique node ids and auto-unseal, move cloud seals to KMS plugins before 2.7, upgrade standbys first, and take snapshots from the active node.
1.Kubernetes auth
bao auth enable kubernetes bao write auth/kubernetes/config \ kubernetes_host=https://kubernetes.default.svc bao policy write app-read app-read.hcl bao write auth/kubernetes/role/app \ bound_service_account_names=app \ bound_service_account_namespaces=app \ policies=app-read ttl=1h
- OpenBao calls the TokenReview API, so its own ServiceAccount needs
system:. The chart sets it. A hand-written install must add it, or every login is "permission denied".auth-delegator - No reviewer token : the client's JWT is the bearer, and each client SA needs the binding instead.
- OpenBao re-reads its own SA token every minute, so short-lived projected tokens work (RBAC).
- A role must bind ServiceAccount names and namespaces.
- Since 2.7.1 and 2.6.4 the SA JWT is checked again on renewal : a revoked SA token can no longer renew.
app-read.hcl: a read-only policy onsecret/, plusdata/ app/* read,listonsecret/.metadata/ app/*
2.JWT auth, SPIFFE and people
- JWT auth checks a signed token against one issuer per mount (
oidc_,discovery_ url jwks_urlor fixed keys), like the API server (OIDC). No TokenReview call, no early revocation. - SPIFFE : a JWT-SVID from the SPIRE agent, posted to
auth/. Pinjwt/ login bound_subject=, or any SVID with the right audience gets the role. No SPIFFE-specific method (SPIFFE, SPIRE).spiffe://<td>/ ns/ <ns>/ sa/ <sa> - People : a second JWT/OIDC mount that trusts the IdP (Keycloak).
3.Ways to get a secret out
| Path | Secret lands as | Auth | Note |
|---|---|---|---|
| ESO | a Secret, per refreshInterval | ESO's SA | GitOps-friendly. The value sits in etcd |
| injector | sidecar files, /vault/ | the Pod's SA | vault., webhook failurePolicy: Ignore : webhook down, the Pod starts without secrets |
| CSI driver | files in a volume | the Pod's SA | provider: openbao, needs the CSI driver |
| direct login | in the app's memory | Kubernetes or JWT | best for leases and dynamic secrets |
| Flux SOPS | kustomize-controller decrypts | Kubernetes (Flux 2.9) | a transit key as SOPS master key |
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata: {name: openbao}
spec:
provider:
vault:
server: http://openbao.openbao:8200
path: secret
version: v2
auth:
kubernetes:
mountPath: kubernetes
role: external-secrets
serviceAccountRef:
name: external-secrets
namespace: external-secrets
- ESO's docs point to the
vaultprovider. A newerprovider.(camelCase) usesopenBao path, notmountPath.
Seen in practice
- Lagging patch releases : renewal fixes land in patches. Do better : track the current minor's patches.
- Cloud KMS auto-unseal ties unsealing to cloud credentials. If a client cert supplies them, its CA now guards unsealing too (Private PKI). Do better : alert on that chain and keep a tested recovery path.
- Each auth mount fails its own way : Kubernetes auth needs the reviewer's
system:auth-delegatorbinding, or every login fails. - Snapshots must hit the active node : a standby redirects to the leader's
api_addr. Do better : the active Service, TLS end to end.
In short : Pods log in with their ServiceAccount or a SPIFFE JWT-SVID, then get secrets out via ESO, the injector, CSI, the API or Flux SOPS. Pin JWT roles with
bound_subject.