GitOps
GitOps with Flux: Deep Dive
1.Repo layout
Running Flux for many apps and clusters, at v2.9.6. The basics are in Flux CD v2.
- apps/ : a base plus one overlay per env. infrastructure/ : controllers, then configs.
- clusters/<env>/ : only Flux objects, such as Kustomizations that point at that env's overlays.
- Env differences : Kustomize overlays, or
.spec.patcheson a Flux Kustomization.
- fleet/
- apps/ : base/, staging/, production/
- infrastructure/ : controllers/, configs/
- clusters/staging/, clusters/production/
- flux-system/ : written by bootstrap
- artifacts.yaml, infrastructure.yaml, apps.yaml
2.Promotion
| Layout | How a change moves on |
|---|---|
| Monorepo | Overlay bump. Big PRs can hide prod |
| Repo per env | Cross-repo PRs. Hardest for infra |
| Repo per team | Platform onboards each team repo |
| Repo per app | Config repo points at it by semver |
- Staging gets new app versions from image automation.
- Production gets them by pull request.
- Infra goes to staging first, then on after e2e tests.
- Helm : a passing staging upgrade fires an Alert to a
githubdispatchProvider. CI readsmetadata.revisionand opens the prod PR.
3.Layers with dependsOn
- Controllers and CRDs, then their custom resources, then apps. All with
prune: true. - infra-configs has no wait : apps starts once configs are applied, healthy or not.
- ArtifactGenerator slices the monorepo. Its revision is a content hash, so a commit outside apps/ leaves apps idle.
- Needs source-watcher. ExternalArtifact gate off : Stalled AccessDenied.
4.What Ready means
| Dependency sets | Its Ready means |
|---|---|
| nothing | applied, no more |
healthChecks | listed objects healthy |
wait: true | every object healthy |
| only CEL exprs | applied (CEL unused) |
Only the dependency's wait counts, not the dependent's.
- Reads the API server, not the cache : generation observed and Ready=True.
- Same source : the dependency must have applied that revision too.
- No watch : a blocked dependent re-checks every 30s (
--requeue-dependency). readyExprreplaces the built-in check. Repeat the Ready test in it.- Health : kstatus or a CEL expr, polled every 5s until timeout. Failed ends it early.
5.Interval is not the delay
A run starts at once on a new source revision, a spec change, flux reconcile or a Receiver. Also on an edit to a ConfigMap or Secret it uses in substituteFrom, decryption or kubeConfig, if labelled reconcile.. Labelling anything else does nothing.
- intervalfull cycle that fixes drift. Shorter = more API load, not faster delivery
- retryIntervaldefaults to interval. Flat, no backoff
- timeoutinterval minus 30s (1h : 59m30s)
1.Apply in stages, then prune
2.Why kubectl edits come back
- Before each apply Flux takes over the fields of every field manager whose name starts with kubectl. Then Git wins : extra fields go, Git's values return.
- Spared : /status and /scale fields,
ssa: Mergeobjects, other managers. rollout restartrolls twice. Add--field-manager=flux-client-side-apply.
Table keys live under kustomize..
| Object key | Effect |
|---|---|
| ssa: Override | Default. Git wins |
| ssa: Merge | Keeps others' fields, not atomic lists |
| ssa: IfNotPresent | Create once, never patch |
| ssa: Ignore | Skipped, health too |
| prune: Disabled | Kept when it leaves Git |
| reconcile: disabled | No apply, no prune |
| force: Enabled | Recreate if immutable |
3.Ignore rules (2.9)
.spec.ignore: JSON pointer paths plus an optional target. No target means every object.- kubectl edit and the HPA (via /scale) are Update-type, so their values are adopted.
- HPA : commit a starting replicas and ignore
/spec/replicas. - An ignored path keeps hand edits forever. Scope it.
4.SOPS and variables
- Encrypt only
dataandstringData. - secretRef keys :
*.agekey,*.asc,sops.*. It replaces the global age key. - No secretRef : global age key or KMS identity.
- No decryption :
ENC[...]is applied as is. See Secrets in Git with SOPS or External Secrets.
Test : kustomize build . | flux envsubst --strict. flux debug ks <n> --show-vars prints secrets.
decryption:
provider: sops
secretRef: {name: sops-age}
postBuild:
substitute: {env: prod} # wins
substituteFrom: # later wins
- {kind: ConfigMap, name: vars}
2.9 strict : a ${var} with no value and no default fails the build, when substitution runs. ${var:=x} is fine.
5.Tenant lockdown
--no-cross-namespace-refs=true: a sourceRef to another namespace stalls (AccessDenied). dependsOn is not checked.--no-remote-bases=true, and--default-service-account=defaulton kustomize and helm (source-watcher from 2.10).- flux-system keeps cluster-admin through
serviceAccountName: kustomize-controller. See RBAC.
2.HelmRelease defaults that bite
| Field | Default | Change it to |
|---|---|---|
| *.strategy | RemediateOnFailure, 0 retries | RetryOnFailure : 5m, no rollback |
| upgrade.crds | Create : never updates CRDs | CreateReplace |
| driftDetection.mode | disabled | enabled |
| postRenderStrategy | combined (2.9) : sees hooks | nohooks |
| upgrade.serverSideApply | auto : keeps the old method | enabled |
| chartRef (OCI) | version+digest : re-push upgrades | DisableChartDigestTracking |
3.Receivers and latency
- No webhooks : scan interval + Git poll interval + push and rollout. 5m + 1m is about 6 min.
- Receivers on both edges : seconds plus rollout.
- Never on the path : Kustomization, HelmRelease and automation intervals.
- An automation sees policies in its namespace only, and never pokes its GitRepository.
- Patch
requestedAton the sources they list. Path in..status. webhookPath - 2.9
generic-oidc: no Secret. Pin the token claims. - Rate-limit the public Ingress.
- Commit status : a
githuborgitlabProvider + an Alert. - Same event dropped for 5m. Big fleets :
DisableCommitStatusEvent. - Chat :
eventSeverity: error.
4."My change is not deployed"
| Ready reason | Likely cause | Look with |
|---|---|---|
| ArtifactFailed | Source not Ready : auth, ref, verify | flux get sources all -A |
| BuildFailed (strict mode) | A ${var} with no value | flux debug ks <n> --show-vars |
| DependencyNotReady | Dependency not Ready, or behind | flux get ks <dependency> |
| HealthCheckFailed | Rollout slower than timeout, or broken | kubectl describe the object |
| Stalled AccessDenied | Cross-namespace ref, or gate off | flux debug ks <n> --show-status |
| HelmRelease RetriesExceeded | 0 retries by default : one failure stalls | fix, then flux reconcile hr <n> --reset |
Start with flux get all -A --status-selector ready=false. Owner : flux trace -n <ns> deployment <n>. Preview : flux diff ks <n> --path ./dir.
5.Scale
- One work queue per kind, sized by
--concurrent. - Shards : label a source and its users
sharding..fluxcd. io/ key - Upgrades : FluxInstance
"2.9.x"floats patches.
| cluster. | Workers | Limits | Dep. requeue |
|---|---|---|---|
| small | 5 | 1 CPU, 512Mi | 10s |
| medium | 10 | 2 CPU, 1Gi | 5s |
| large | 20 | 3 CPU, 3Gi | 5s |
Rows set kustomize and helm. Plain defaults : kustomize 4, helm 4, source 2.