GitOps
Secrets in Git with SOPS
1.Envelope encryption
- SOPS encrypts values, not files. The YAML shape stays readable.
- One random 256-bit data key per file seals each selected value : AES256-GCM, its own IV, its key path as authenticated data.
- The data key is wrapped once per master key (age, PGP, cloud KMS, Vault or OpenBao, HuaweiCloud KMS). Any one can decrypt.
- Adding a recipient re-wraps the data key. Values stay as they are.
SOPS v3.13.3, age v1.3.2, Flux v2.9.6. Flux basics : Flux CD v2. Short version : GitOps with Flux: Deep Dive.
2.What stays readable
- Keys,
apiVersion,kindandmetadatastay plain, so a diff shows which field changed. Encrypt onlydataandstringData. - Comments : the default
unencrypted_encrypts all. Withsuffix encrypted_regex, only those inside the selected keys. - The MAC covers every value, plain or not. A hand edit of
namespacegives "MAC mismatch" (exit 51).mac_(3.9.0+) covers encrypted values only.only_ encrypted: true
apiVersion: v1
kind: Secret
metadata: {name: webhook-token, namespace: default}
stringData:
token: ENC[AES256_GCM,data:...,iv:...,tag:...,type:str]
sops:
age:
- recipient: age1...
enc: # wrapped data key
encrypted_regex: ^(data|stringData)$
lastmodified: "..."
mac: ENC[...]
version: 3.13.3 # sorted keys, 4-space indent
3..sops.yaml creation rules
- The first matching
path_regexwins. It is read relative to the config file. - The config is found from the current directory upwards, not from the file's folder.
- Overlay trap : a rule with no selector encrypts
apiVersion,kindandmetadatatoo. An overlay's own.sops.yamlcounts only if youcdinto it. sops editandsops setreuse the selector stored in the file.
creation_rules: # Helm values: whole file - path_regex: .*values\.yaml$ age: age1... # Secret manifests: values only - path_regex: clusters/prod/.*\.ya?ml$ encrypted_regex: ^(data|stringData)$ age: age1...
Rule order matters : under the Secret rule a values file stays plain.
4.Everyday commands
| Task | Command |
|---|---|
Make an age key (post-quantum : -pq) | age-keygen -o age.agekey |
| Encrypt a Secret, values only | sops encrypt --age age1... --encrypted-regex '^( |
| Encrypt using .sops.yaml | sops encrypt --in-place s.yaml (run inside the repo) |
| Edit, keeping the MAC valid | sops edit s.yaml |
| Read one value | sops decrypt --extract '["data"]["password"]' s.yaml |
| Set one value (JSON) | sops set s.yaml '["stringData"]["k"]' '"v"' |
| Is it encrypted ? | sops filestatus s.yaml |
| MAC over encrypted values only | sops --mac-only-encrypted encrypt -i s.yaml (global flag, before the subcommand) |
Readable git diff | *.yaml diff=sopsdiffer in .gitattributes, then git config diff. |
.sops.yaml above where you run it.1.Where Flux decrypts
- Only kustomize-controller decrypts. helm-controller never does : it reads Secrets already in the cluster.
- Runs only when
spec.is set on the Kustomization that applies the objects. On the rootdecryption. provider: sops flux-systemit does nothing for its children. - No MAC check in the controller : patches and generators would break it. The CLI does check.
2.The decryption block
| Field or key | Effect |
|---|---|
| provider | Required. Only sops |
| secretRef.name | Key Secret, same namespace |
| serviceAccountName | Secret-less cloud or Vault auth |
| decrypt: Disabled | Skips the post-build pass only |
| substitute: disabled | Keeps ${...} in values |
| watch: Enabled | Key Secret edit starts a run |
Last three : annotations or labels under kustomize. and reconcile..
kind: Kustomization # kustomize.toolkit.fluxcd.io/v1 spec: path: ./apps/production sourceRef: {kind: GitRepository, name: flux-system} decryption: provider: sops secretRef: name: sops-age # entry must end in .agekey
CLI : flux create kustomization ... --decryption-provider=sops --decryption-secret=sops-age.
3.Where the keys come from
| Secret entry | Holds |
|---|---|
| *.agekey | age identities (PQ from Flux 2.9) |
| *.asc | armored PGP key, no passphrase |
| sops.vault-token | Vault or OpenBao token |
| sops.aws-kms | static AWS keys (YAML) |
| sops.azure-kv | service principal or identity |
| sops.gcp-kms | service-account JSON |
Any other name, such as age.key, is ignored without a warning.
- secretRef : a Secret with the entries in the table.
- Global age key :
--sops-age-secret(2.7+). Still needsprovider: sopsper Kustomization. AsecretRefreplaces it, no merge. - Cloud identity : the controller's own, or
serviceAccountNamewith theObjectLevelWorkloadIdentitygate (2.6+, else Stalled). See RBAC. - Vault, in order : static token,
--sops-vault-configmap(2.9+),VAULT_TOKEN. No fallback.
4.Generators and Helm values
- Generator files are decrypted before the build. Format comes from the key name (
values.yaml=). - Helm : generate a Secret from the encrypted values, point
valuesFromat it. The hash suffix upgrades the release on change. - Values also land in Helm release Secrets (
helm get values).
secretGenerator:
- name: podinfo-values
files:
- values.yaml=my-values.enc.yaml
configurations: [kustomizeconfig.yaml]
# kustomizeconfig.yaml: nameReference
# Secret -> spec/valuesFrom/name (HelmRelease)
5.No safety net
Decryption off, ciphertext under stringData : the Secret goes live with the raw ENC[...] text and the Kustomization says Ready=True. ConfigMaps and generated Secrets too. kubectl apply -k never decrypts.
- Normalize drops the top-level
sopsfield, so the guard ("is SOPS encrypted, configuring decryption is required") only fires for ciphertext underdata:. - Land
decryptionin an earlier commit than the first encrypted file.
provider: sops is set, with keys found by name or cloud identity. Nothing refuses ciphertext.1.Picking a master key
| Backend | In .sops.yaml | Flux gets it from | Watch out |
|---|---|---|---|
| age | age: age1... | *.agekey entry or global key | Simplest. PQ keys need Flux 2.9, sops 3.12.1+ |
| PGP | pgp: <fingerprint> | *.asc entry | Not deprecated, but docs prefer age |
| Cloud KMS | kms, gcp_kms ... | creds entry or workload identity | No key file. KMS must answer every run |
| Vault, OpenBao | hc_ | token, or Kubernetes auth | Role name fixed to {namespace}_{sa} |
2.Keyless AWS KMS
- One KMS key, no age, no static AWS keys. Kustomizations set only
provider: sops, so the controller uses its own identity. - A sidecar,
aws-signing-helper serve, plays the EC2 metadata service.AWS_points the SDK at it.EC2_ METADATA_ SERVICE_ ENDPOINT - It signs with a cert-manager client cert and gets short-lived creds for a
kms:Decrypt-only role. - Fresh cluster : SOPS Kustomizations retry until the CA exists.
3.Rotation
updatekeys: recipients from.sops.yaml, same data key.rotate: new data key, every value re-encrypted. Do it after removing a recipient.- Leaked key (docs order) : drop it, updatekeys, rotate, commit, then change the real passwords. History still opens with the old key.
$ sops updatekeys -y secrets/*.yaml $ sops rotate -i secrets/db.yaml # new key Secret in the cluster, then $ flux reconcile ks apps
A key Secret edit starts no run unless labelled reconcile..
4.When it breaks
| Symptom | Cause | Fix |
|---|---|---|
| "failed to load age identities ... no master key was able to decrypt" | No *.agekey entry loaded: misnamed, or a secretRef hides the global key | Rename to identity.agekey |
| "MAC mismatch" (CLI, exit 51) | Hand edit, or decrypting merged build output | sops edit, or decrypt the patch file |
| "illegal base64 data at input byte 3" | Committed **SOPS** preview output, or plain text under data: | Use stringData (Secrets). Never commit previews |
| "cipher: message authentication failed" | Plain patch onto an encrypted object | Encrypt the patch too |
| Secret value changed after decrypt | ${...} was substituted | substitute: disabled or $${var} |
Ready=True, ENC[...] live | No decryption on that Kustomization | Add provider: sops |
flux build and flux diff never decrypt : Secret values show as **SOPS**, ConfigMaps as ENC[...].
5.SOPS or something else ?
| SOPS in Flux | Sealed Secrets | External Secrets | |
|---|---|---|---|
| In Git | ciphertext in normal files | a SealedSecret | a reference, no values |
| Decrypts | kustomize-controller | its own controller | pulls from a store |
| Rotation | per file, by hand | by hand (key renewal is not rotation) | in the store |
| Needs live | KMS or Vault at reconcile | only its controller | the store, at sync |
More on the third option in External Secrets. Argo CD needs a plugin such as ksops for SOPS, and its docs caution against secret-generating plugins.