GitOps

Secrets in Git with SOPS

1.Envelope encryption

  • SOPS encrypts values, not files. The YAML shape stays readable.
  • One random 256-bit data key per file seals each selected value : AES256-GCM, its own IV, its key path as authenticated data.
  • The data key is wrapped once per master key (age, PGP, cloud KMS, Vault or OpenBao, HuaweiCloud KMS). Any one can decrypt.
  • Adding a recipient re-wraps the data key. Values stay as they are.

SOPS v3.13.3, age v1.3.2, Flux v2.9.6. Flux basics : Flux CD v2. Short version : GitOps with Flux: Deep Dive.

plaintext value stringData.token data key random, 256-bit, per file AES256-GCM ENC[AES256_GCM,...] own IV, key path as AAD per recipient age PGP AWS KMS Vault same file in Git ENC values + sops: block (wrapped keys, MAC, lastmodified) any one master key unwraps the data key

2.What stays readable

  • Keys, apiVersion, kind and metadata stay plain, so a diff shows which field changed. Encrypt only data and stringData.
  • Comments : the default unencrypted_suffix encrypts all. With encrypted_regex, only those inside the selected keys.
  • The MAC covers every value, plain or not. A hand edit of namespace gives "MAC mismatch" (exit 51). mac_only_encrypted: true (3.9.0+) covers encrypted values only.
apiVersion: v1
kind: Secret
metadata: {name: webhook-token, namespace: default}
stringData:
    token: ENC[AES256_GCM,data:...,iv:...,tag:...,type:str]
sops:
    age:
        - recipient: age1...
          enc: # wrapped data key
    encrypted_regex: ^(data|stringData)$
    lastmodified: "..."
    mac: ENC[...]
    version: 3.13.3  # sorted keys, 4-space indent

3..sops.yaml creation rules

  • The first matching path_regex wins. It is read relative to the config file.
  • The config is found from the current directory upwards, not from the file's folder.
  • Overlay trap : a rule with no selector encrypts apiVersion, kind and metadata too. An overlay's own .sops.yaml counts only if you cd into it.
  • sops edit and sops set reuse the selector stored in the file.
creation_rules:
  # Helm values: whole file
  - path_regex: .*values\.yaml$
    age: age1...
  # Secret manifests: values only
  - path_regex: clusters/prod/.*\.ya?ml$
    encrypted_regex: ^(data|stringData)$
    age: age1...

Rule order matters : under the Secret rule a values file stays plain.

4.Everyday commands

TaskCommand
Make an age key (post-quantum : -pq)age-keygen -o age.agekey
Encrypt a Secret, values onlysops encrypt --age age1... --encrypted-regex '^(data|stringData)$' --in-place s.yaml
Encrypt using .sops.yamlsops encrypt --in-place s.yaml (run inside the repo)
Edit, keeping the MAC validsops edit s.yaml
Read one valuesops decrypt --extract '["data"]["password"]' s.yaml
Set one value (JSON)sops set s.yaml '["stringData"]["k"]' '"v"'
Is it encrypted ?sops filestatus s.yaml
MAC over encrypted values onlysops --mac-only-encrypted encrypt -i s.yaml (global flag, before the subcommand)
Readable git diff*.yaml diff=sopsdiffer in .gitattributes, then git config diff.sopsdiffer.textconv "sops decrypt"
In short : SOPS seals each value with one data key per file, wraps that key for every recipient, keeps the YAML shape and metadata readable, and picks its rule from the nearest .sops.yaml above where you run it.

1.Where Flux decrypts

  • Only kustomize-controller decrypts. helm-controller never does : it reads Secrets already in the cluster.
  • Runs only when spec.decryption.provider: sops is set on the Kustomization that applies the objects. On the root flux-system it does nothing for its children.
  • No MAC check in the controller : patches and generators would break it. The CLI does check.
source artifact 1. decrypt sources 2. kustomize build 3. decrypt objects 4. postBuild substitution 5. normalize 6. guard, apply generator files, envs, local patches, in place any kind with sops.mac, marked Secret data sees plaintext stringData into data, unknown fields dropped no MAC check anywhere here

2.The decryption block

Field or keyEffect
providerRequired. Only sops
secretRef.nameKey Secret, same namespace
serviceAccountNameSecret-less cloud or Vault auth
decrypt: DisabledSkips the post-build pass only
substitute: disabledKeeps ${...} in values
watch: EnabledKey Secret edit starts a run

Last three : annotations or labels under kustomize.toolkit.fluxcd.io/ and reconcile.fluxcd.io/.

kind: Kustomization  # kustomize.toolkit.fluxcd.io/v1
spec:
  path: ./apps/production
  sourceRef: {kind: GitRepository, name: flux-system}
  decryption:
    provider: sops
    secretRef:
      name: sops-age  # entry must end in .agekey

CLI : flux create kustomization ... --decryption-provider=sops --decryption-secret=sops-age.

3.Where the keys come from

Secret entryHolds
*.agekeyage identities (PQ from Flux 2.9)
*.ascarmored PGP key, no passphrase
sops.vault-tokenVault or OpenBao token
sops.aws-kmsstatic AWS keys (YAML)
sops.azure-kvservice principal or identity
sops.gcp-kmsservice-account JSON

Any other name, such as age.key, is ignored without a warning.

Three sources
  • secretRef : a Secret with the entries in the table.
  • Global age key : --sops-age-secret (2.7+). Still needs provider: sops per Kustomization. A secretRef replaces it, no merge.
  • Cloud identity : the controller's own, or serviceAccountName with the ObjectLevelWorkloadIdentity gate (2.6+, else Stalled). See RBAC.
  • Vault, in order : static token, --sops-vault-configmap (2.9+), VAULT_TOKEN. No fallback.

4.Generators and Helm values

  • Generator files are decrypted before the build. Format comes from the key name (values.yaml=).
  • Helm : generate a Secret from the encrypted values, point valuesFrom at it. The hash suffix upgrades the release on change.
  • Values also land in Helm release Secrets (helm get values).
secretGenerator:
  - name: podinfo-values
    files:
      - values.yaml=my-values.enc.yaml
configurations: [kustomizeconfig.yaml]
# kustomizeconfig.yaml: nameReference
# Secret -> spec/valuesFrom/name (HelmRelease)

5.No safety net

Decryption off, ciphertext under stringData : the Secret goes live with the raw ENC[...] text and the Kustomization says Ready=True. ConfigMaps and generated Secrets too. kubectl apply -k never decrypts.

Why, and the fix
  • Normalize drops the top-level sops field, so the guard ("is SOPS encrypted, configuring decryption is required") only fires for ciphertext under data:.
  • Land decryption in an earlier commit than the first encrypted file.
In short : kustomize-controller decrypts between build and apply, only where provider: sops is set, with keys found by name or cloud identity. Nothing refuses ciphertext.

1.Picking a master key

BackendIn .sops.yamlFlux gets it fromWatch out
ageage: age1...*.agekey entry or global keySimplest. PQ keys need Flux 2.9, sops 3.12.1+
PGPpgp: <fingerprint>*.asc entryNot deprecated, but docs prefer age
Cloud KMSkms, gcp_kms ...creds entry or workload identityNo key file. KMS must answer every run
Vault, OpenBaohc_vault_transit_uritoken, or Kubernetes authRole name fixed to {namespace}_{sa}

2.Keyless AWS KMS

  • One KMS key, no age, no static AWS keys. Kustomizations set only provider: sops, so the controller uses its own identity.
  • A sidecar, aws-signing-helper serve, plays the EC2 metadata service. AWS_EC2_METADATA_SERVICE_ENDPOINT points the SDK at it.
  • It signs with a cert-manager client cert and gets short-lived creds for a kms:Decrypt-only role.
  • Fresh cluster : SOPS Kustomizations retry until the CA exists.
cert-manager private ClusterIssuer client cert Secret client auth, 1 year kustomize-controller Pod mounted manager SOPS + AWS SDK signing helper 127.0.0.1:9911 IMDS IAM Roles Anywhere cert in, temp creds out kms:Decrypt AWS KMS unwraps the data key no static AWS keys in Git or the cluster

3.Rotation

  • updatekeys : recipients from .sops.yaml, same data key.
  • rotate : new data key, every value re-encrypted. Do it after removing a recipient.
  • Leaked key (docs order) : drop it, updatekeys, rotate, commit, then change the real passwords. History still opens with the old key.
$ sops updatekeys -y secrets/*.yaml
$ sops rotate -i secrets/db.yaml
# new key Secret in the cluster, then
$ flux reconcile ks apps

A key Secret edit starts no run unless labelled reconcile.fluxcd.io/watch: Enabled.

4.When it breaks

SymptomCauseFix
"failed to load age identities ... no master key was able to decrypt"No *.agekey entry loaded: misnamed, or a secretRef hides the global keyRename to identity.agekey
"MAC mismatch" (CLI, exit 51)Hand edit, or decrypting merged build outputsops edit, or decrypt the patch file
"illegal base64 data at input byte 3"Committed **SOPS** preview output, or plain text under data:Use stringData (Secrets). Never commit previews
"cipher: message authentication failed"Plain patch onto an encrypted objectEncrypt the patch too
Secret value changed after decrypt${...} was substitutedsubstitute: disabled or $${var}
Ready=True, ENC[...] liveNo decryption on that KustomizationAdd provider: sops

flux build and flux diff never decrypt : Secret values show as **SOPS**, ConfigMaps as ENC[...].

5.SOPS or something else ?

SOPS in FluxSealed SecretsExternal Secrets
In Gitciphertext in normal filesa SealedSecreta reference, no values
Decryptskustomize-controllerits own controllerpulls from a store
Rotationper file, by handby hand (key renewal is not rotation)in the store
Needs liveKMS or Vault at reconcileonly its controllerthe store, at sync

More on the third option in External Secrets. Argo CD needs a plugin such as ksops for SOPS, and its docs caution against secret-generating plugins.

In short : Pick age or a cloud KMS, avoid static credentials, rotate with updatekeys and rotate, and read the error first : most failures are a key name, a MAC or a missing switch.