GitOps

GitOps with Flux: Deep Dive

1.Repo layout

Running Flux for many apps and clusters, at v2.9.6. The basics are in Flux CD v2.

  • apps/ : a base plus one overlay per env. infrastructure/ : controllers, then configs.
  • clusters/<env>/ : only Flux objects, such as Kustomizations that point at that env's overlays.
  • Env differences : Kustomize overlays, or .spec.patches on a Flux Kustomization.
One fleet repo
  • fleet/
    • apps/ : base/, staging/, production/
    • infrastructure/ : controllers/, configs/
    • clusters/staging/, clusters/production/
      • flux-system/ : written by bootstrap
      • artifacts.yaml, infrastructure.yaml, apps.yaml

2.Promotion

LayoutHow a change moves on
MonorepoOverlay bump. Big PRs can hide prod
Repo per envCross-repo PRs. Hardest for infra
Repo per teamPlatform onboards each team repo
Repo per appConfig repo points at it by semver
Trunk-based flow
  • Staging gets new app versions from image automation.
  • Production gets them by pull request.
  • Infra goes to staging first, then on after e2e tests.
  • Helm : a passing staging upgrade fires an Alert to a githubdispatch Provider. CI reads metadata.revision and opens the prod PR.

3.Layers with dependsOn

  • Controllers and CRDs, then their custom resources, then apps. All with prune: true.
  • infra-configs has no wait : apps starts once configs are applied, healthy or not.
  • ArtifactGenerator slices the monorepo. Its revision is a content hash, so a commit outside apps/ leaves apps idle.
  • Needs source-watcher. ExternalArtifact gate off : Stalled AccessDenied.
GitRepository flux-system, monorepo ArtifactGenerator source-watcher infrastructure ExternalArtifact apps ExternalArtifact ./controllers infra-controllers wait: true ./configs infra-configs no wait dependsOn apps wait: true red arrow = dependsOn (waits for Ready)

4.What Ready means

Dependency setsIts Ready means
nothingapplied, no more
healthCheckslisted objects healthy
wait: trueevery object healthy
only CEL exprsapplied (CEL unused)

Only the dependency's wait counts, not the dependent's.

The dependsOn check
  • Reads the API server, not the cache : generation observed and Ready=True.
  • Same source : the dependency must have applied that revision too.
  • No watch : a blocked dependent re-checks every 30s (--requeue-dependency).
  • readyExpr replaces the built-in check. Repeat the Ready test in it.
  • Health : kstatus or a CEL expr, polled every 5s until timeout. Failed ends it early.

5.Interval is not the delay

A run starts at once on a new source revision, a spec change, flux reconcile or a Receiver. Also on an edit to a ConfigMap or Secret it uses in substituteFrom, decryption or kubeConfig, if labelled reconcile.fluxcd.io/watch: Enabled. Labelling anything else does nothing.

The three timers
  • intervalfull cycle that fixes drift. Shorter = more API load, not faster delivery
  • retryIntervaldefaults to interval. Flat, no backoff
  • timeoutinterval minus 30s (1h : 59m30s)
In short : Keep apps, infrastructure and clusters apart, promote by pull request, chain layers with dependsOn, and remember Ready means healthy only when the dependency waits.

1.Apply in stages, then prune

  • An SSA dry-run diff first. Only changed objects are sent, as manager kustomize-controller.
  • Stages put CRDs and Namespaces first.
  • Prune deletes what left the build (.status.inventory). Since 2.9 failed prunes retry.
  • Moving a pruned Kustomization recreates its workloads. Set prune: false first.
1. CRDs, Namespaces, ClusterRoles cluster-wide first wait 2. kinds ending in Class StorageClass, GatewayClass ... wait 3. custom stage kinds --custom-apply-stage-kinds (2.8+) no wait 4. everything else Deployments, Services ... prune health stage waits poll every 2s, up to 60s only with wait or healthChecks

2.Why kubectl edits come back

  • Before each apply Flux takes over the fields of every field manager whose name starts with kubectl. Then Git wins : extra fields go, Git's values return.
  • Spared : /status and /scale fields, ssa: Merge objects, other managers.
  • rollout restart rolls twice. Add --field-manager=flux-client-side-apply.

Table keys live under kustomize.toolkit.fluxcd.io/.

Object keyEffect
ssa: OverrideDefault. Git wins
ssa: MergeKeeps others' fields, not atomic lists
ssa: IfNotPresentCreate once, never patch
ssa: IgnoreSkipped, health too
prune: DisabledKept when it leaves Git
reconcile: disabledNo apply, no prune
force: EnabledRecreate if immutable

3.Ignore rules (2.9)

  • .spec.ignore : JSON pointer paths plus an optional target. No target means every object.
  • kubectl edit and the HPA (via /scale) are Update-type, so their values are adopted.
  • HPA : commit a starting replicas and ignore /spec/replicas.
  • An ignored path keeps hand edits forever. Scope it.
ignored path drifted found during a run another Apply-type manager owns the field ? yes no Strip drop it, Flux lets go Adopt copy the live value in on create Git's value seeds the field

4.SOPS and variables

  • Encrypt only data and stringData.
  • secretRef keys : *.agekey, *.asc, sops.*. It replaces the global age key.
  • No secretRef : global age key or KMS identity.
  • No decryption : ENC[...] is applied as is. See Secrets in Git with SOPS or External Secrets.

Test : kustomize build . | flux envsubst --strict. flux debug ks <n> --show-vars prints secrets.

decryption:
  provider: sops
  secretRef: {name: sops-age}
postBuild:
  substitute: {env: prod}  # wins
  substituteFrom:  # later wins
  - {kind: ConfigMap, name: vars}

2.9 strict : a ${var} with no value and no default fails the build, when substitution runs. ${var:=x} is fine.

5.Tenant lockdown

  • --no-cross-namespace-refs=true : a sourceRef to another namespace stalls (AccessDenied). dependsOn is not checked.
  • --no-remote-bases=true, and --default-service-account=default on kustomize and helm (source-watcher from 2.10).
  • flux-system keeps cluster-admin through serviceAccountName: kustomize-controller. See RBAC.
flux-system SA kustomize-controller applies Tenant layer Namespace apps, SA flux RoleBinding : cluster-admin Tenant repo GitRepository in apps Kustomization in apps serviceAccountName: flux dependsOn impersonates SA flux objects in apps namespace only Refused CRDs, cluster-scoped, other-namespace sourceRef
In short : Flux applies in stages, prunes by inventory, takes kubectl-made fields back unless you opt out or ignore them, decrypts SOPS at build time, and fences tenants with impersonation.

1.OCI artifacts, signed

  • Gitless : CI pushes manifests to a registry. Controllers pull only from there.
  • --revision=<branch>@sha1:<sha> returns as lastAppliedOriginRevision, so commit status works.
  • Verify with cosign (keyless matchOIDCIdentity) or notation. Prod refs : semver or digest.
CI flux push artifact cosign sign, flux tag OCI registry artifact + tag + signature poll OCIRepository verify : cosign / notation verified Kustomization lastAppliedOriginRevision VerificationError nothing is pulled CI can also call flux trigger receiver

2.HelmRelease defaults that bite

FieldDefaultChange it to
*.strategyRemediateOnFailure, 0 retriesRetryOnFailure : 5m, no rollback
upgrade.crdsCreate : never updates CRDsCreateReplace
driftDetection.modedisabledenabled
postRenderStrategycombined (2.9) : sees hooksnohooks
upgrade.serverSideApplyauto : keeps the old methodenabled
chartRef (OCI)version+digest : re-push upgradesDisableChartDigestTracking

3.Receivers and latency

  • No webhooks : scan interval + Git poll interval + push and rollout. 5m + 1m is about 6 min.
  • Receivers on both edges : seconds plus rollout.
  • Never on the path : Kustomization, HelmRelease and automation intervals.
  • An automation sees policies in its namespace only, and never pokes its GitRepository.
registry scan interval ImageRepository watch ImagePolicy watch, same ns ImageUpdateAutomation push Git poll interval GitRepository watch Kustomization apply rollout + health a Receiver replaces either red timer
Receivers (v1)
  • Patch requestedAt on the sources they list. Path in .status.webhookPath.
  • 2.9 generic-oidc : no Secret. Pin the token claims.
  • Rate-limit the public Ingress.
Alerts (still v1beta3)
  • Commit status : a github or gitlab Provider + an Alert.
  • Same event dropped for 5m. Big fleets : DisableCommitStatusEvent.
  • Chat : eventSeverity: error.

4."My change is not deployed"

Ready reasonLikely causeLook with
ArtifactFailedSource not Ready : auth, ref, verifyflux get sources all -A
BuildFailed (strict mode)A ${var} with no valueflux debug ks <n> --show-vars
DependencyNotReadyDependency not Ready, or behindflux get ks <dependency>
HealthCheckFailedRollout slower than timeout, or brokenkubectl describe the object
Stalled AccessDeniedCross-namespace ref, or gate offflux debug ks <n> --show-status
HelmRelease RetriesExceeded0 retries by default : one failure stallsfix, then flux reconcile hr <n> --reset

Start with flux get all -A --status-selector ready=false. Owner : flux trace -n <ns> deployment <n>. Preview : flux diff ks <n> --path ./dir.

5.Scale

  • One work queue per kind, sized by --concurrent.
  • Shards : label a source and its users sharding.fluxcd.io/key.
  • Upgrades : FluxInstance "2.9.x" floats patches.
cluster.sizeWorkersLimitsDep. requeue
small51 CPU, 512Mi10s
medium102 CPU, 1Gi5s
large203 CPU, 3Gi5s

Rows set kustomize and helm. Plain defaults : kustomize 4, helm 4, source 2.

In short : Ship signed OCI artifacts, set Helm remediation on purpose, put Receivers on both edges, read the Ready reason first, then scale with workers and shards.