GitOps and delivery

Argo CD

1.What is Argo CD ?

A GitOps continuous delivery tool for Kubernetes. The desired state lives in Git (or a Helm or OCI repo). A controller keeps comparing it with the live cluster, and can sync the cluster back to it.

  • Pull model : Argo CD pulls from Git, so CI needs no cluster credentials.
  • Renders Kustomize, Helm, Jsonnet, plain YAML, or a plugin.
  • Extras : web UI, CLI, SSO and its own RBAC on top of Kubernetes.
  • Lives in argocd, drives one or many clusters.

2.Words to know

WordMeaning
Target stateWhat Git says should run
Live stateWhat really runs in the cluster
Sync statusSynced, OutOfSync or Unknown
SyncMake live match target (apply)
RefreshRe-read Git and diff, no apply. Hard refresh also drops the cache
HealthIs it running and serving ? A separate answer : Synced can still be Degraded

Argo project, CNCF graduated 2022-12-06. Latest v3.5.4 (2026-10-06), tested on Kubernetes v1.33 to v1.36. The three newest minors get patches. 3.6 is a release candidate.

3.The parts

A few workloads; Argo's own share one image (quay.io/argoproj/argocd). Real state lives in CRs, ConfigMaps and Secrets.

  • repo-server clones and renders. No Kubernetes rights, no stored credentials.
  • application-controller is a StatefulSet and a classic controller. Scale it to shard clusters.
  • notifications-controller sends alerts. dex is optional with direct OIDC. redis is safe to lose.
YouUI, CLI, CI ApplicationSetcontroller Git repoor Helm, OCI login, sync creates argocd-serverAPI, SSO, RBAC Applicationdesired + status writes login watch pulls,caches 24h dexSSO broker app controllerdiff, sync, health repo-serverrenders YAML asks watch, apply rediscache only Cluster(s)in-cluster + registered remote clusters

4.Detection vs correction

Argo CD always notices drift. Fixing it is opt-in, one switch at a time.

  1. Every 120s (plus up to 60s jitter), or at once on a Git webhook, the controller refreshes each app.
  2. repo-server renders the manifests, the controller diffs them with the live objects, and marks the app Synced or OutOfSync. That is detection.
  3. Correction is a sync, and it is manual by default : press Sync, or run argocd app sync.
syncPolicyNew commit in GitFile removed from GitSomeone runs kubectl edit
none (default)OutOfSync, waits for youOutOfSync, waits for youOutOfSync, waits for you
automatedSynced by itselfKeeps running, stays OutOfSyncStays OutOfSync, not reverted
+ prune: trueSynced by itselfDeleted from the clusterStays OutOfSync, not reverted
+ selfHeal: trueSynced by itselfDeleted only if prune is on tooReverted to Git

OpenGitOps asks for "continuously reconciled". Out of the box Argo CD only observes continuously. Flux fixes drift on every interval.

5.Turning it on

spec:
  syncPolicy:
    automated:
      enabled: true   # 3.1+, unset = on
      prune: true     # default false
      selfHeal: true  # default false

CLI : argocd app set APP --sync-policy automated --auto-prune --self-heal. enabled: false pauses auto-sync but keeps the flags.

Fine print
  • Auto-sync tries once per commit SHA (plus parameters). A SHA that failed is not tried again.
  • Self-heal waits a backoff : 2s, times 3, max 300s. Docs saying "5 seconds" are stale.
  • No retry block : auto-sync retries 5 times (5s doubling, max 3m). A retry block without limit : 0 retries.
  • No rollback while auto-sync is on.
In short : repo-server renders Git, the controller diffs it with the cluster every few minutes and flags OutOfSync, and nothing is fixed until you sync or turn on automated, prune and selfHeal.

1.The three CRDs

All are argoproj.io/v1alpha1 only (no beta, no v1), namespaced, and live in argocd by default. See CRDs.

Kind (short)JobKey fields
Application (app)Source(s) to one destinationproject, source, destination, syncPolicy
ApplicationSet (appset)Template that stamps out many Applicationsgenerators, template, strategy
AppProject (appproj)Guard rails : what may go wheresourceRepos, destinations, roles, syncWindows

2.An Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
  finalizers: [resources-finalizer.argocd.argoproj.io]
spec:
  project: default
  source:
    repoURL: https://github.com/my-org/apps.git
    path: guestbook   # targetRevision unset = HEAD
  destination:
    server: https://kubernetes.default.svc
    namespace: guestbook
  • targetRevision : a branch, a tag, a semver range (matches tags only) or a commit SHA.
  • sources : a list instead of source, e.g. a Helm chart plus values from your repo.
  • destination : server URL or cluster name, never both.
  • finalizer : without it, deleting the Application leaves every resource running. See finalizers.
  • project : required. The default project allows anything. A project that may deploy into argocd is as good as admin.

3.Sync phases and waves

  1. Order : phase, wave (default 0, negatives allowed), kind (Namespaces first, custom resources last), name.
  2. Apply the first wave that is not yet synced and healthy.
  3. Wait for Healthy, pause 2s, next wave. Prunes go in reverse wave order.
PreSynchooks Sync phase wave-1 wave0 wave1 each : apply, wait Healthy, 2s prunes in reverse wave order PostSynchooks any failure SyncFail hooks

4.Hooks

kind: Job   # batch/v1
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
    argocd.argoproj.io/sync-wave: "-1"
spec: ...    # a normal Job
  • Kinds : PreSync, Sync, PostSync, SyncFail, Skip. On app deletion : PreDelete (3.3+), PostDelete.
  • No delete policy means BeforeHookCreation. Hooks skip a selective sync.
  • Helm : Argo CD runs helm template, so helm ls shows nothing. pre-install / pre-upgrade become PreSync, post-install / post-upgrade become PostSync.

5.Health

KindHealthy when
Workloadsupdated replicas = desired (rollouts)
LB Service, IngressLB hostname or IP in status
PVCphase is Bound
Job, CronJobComplete. A failed Job, or a CronJob whose last run failed (3.2+), is Degraded
Custom resources238 bundled Lua checks, or yours
Applicationno check (since 1.8)
  • App health = worst direct child. Best to worst : Healthy, Suspended, Progressing, Missing, Degraded, Unknown.
  • Own check : Lua in argocd-cm, key resource.customizations.health.<group>_<kind>.
  • App of apps : add one for argoproj.io_Application, or waves never wait for a child app.
In short : An Application maps its source(s) to one destination inside a project's guard rails, and a sync runs PreSync, then each wave until Healthy, then PostSync.

1.ApplicationSets

Generators make parameter sets. The template is rendered once per set into an Application.

GeneratorParams fromRe-reads
listfixed valuesnever
clustersclusters, by labelSecret events
gitdirs or files in a repoevery 3 min
matrix, mergechild generators combinedfastest child
pullRequestopen PRsevery 30 min

Also scmProvider, clusterDecisionResource, plugin. 3.6 (RC) adds oci. RollingSync (Beta) syncs the apps step by step and turns their auto-sync off.

git generatordirs apps/a, apps/b clusters generatorlabel env: prod matrixevery pair params : path, server templateone Application per pair create, update, delete a on c1 a on c2 b on c1 b on c2 2 dirs x 2 clusters = 4 Applications
  • Bootstrap : since 3.4 the docs recommend it, with the cluster generator, over app of apps (Cluster API).
  • Admin only : it can create Applications in any project. Put a template variable in project only if admins own every generator source.

2.Install : server-side since 3.3

$ kubectl create namespace argocd
$ kubectl apply -n argocd --server-side --force-conflicts \
    -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.4/manifests/install.yaml
Why server-side ?Client-side apply keeps a copy of the object in an annotation. The ApplicationSet CRD (about 343 KB as JSON) is over the 262,144-byte limit, so plain kubectl apply fails. See SSA.
ManifestFor
install.yamltrying it, not production
ha/install.yamlproduction, 3+ nodes
namespace-installone namespace, no CRDs
core-install.yamlno API server, UI or SSO
  • --force-conflicts overwrites live edits. Customize with Kustomize patches.
  • Self-managed Argo CD needs ServerSideApply=true.

3.Commands

CommandDoes
argocd admin initial-password -n argocdFirst admin password. Change it, then delete its Secret
kubectl port-forward svc/argocd-server -n argocd 8080:443Reach the API unexposed, then argocd login localhost:8080
argocd app get my-app --hard-refreshStatus. Hard refresh also drops the manifest cache
argocd app diff my-appGit vs live. Exit 0 same, 1 different, 2 error
argocd app sync my-app --prune --dry-runPreview a sync that also deletes extras. Drop --dry-run to run it
argocd app history my-app, argocd app rollback my-app IDRoll back to a past sync. Not while auto-sync is on
argocd appset generate appset.yamlShow the Applications an ApplicationSet would make
argocd cluster add CONTEXTRegister a cluster (adds an admin-level argocd-manager SA)

4.Argo CD vs Flux

TopicArgo CDFlux
UIBuilt inIn Flux Operator
UsersOwn SSO + RBACKubernetes RBAC
ApplyClient-side, SSA opt-inServer-side apply
DriftSeen always, fixed if selfHealFixed every interval
Helmhelm template, no releaseReal Helm releases

Both CNCF graduated in 2022. New image tags into Git : Argo CD Image Updater, a separate argoproj-labs controller.

3.0 tightened RBAC defaults. 3.5 renders with Helm 4 only. 3.6 (RC) diffs ServerSideApply apps server-side.

5.Gotchas

  • ignoreDifferences only hides the diff. Sync still writes those fields unless RespectIgnoreDifferences=true.
  • An HPA or KEDA owns replicas ? Leave replicas out of Git.
  • Secrets injected while rendering are cached in plain text in Redis. Keep them in the cluster with External Secrets.
  • Since 3.0 Argo CD tracks resources with an annotation, not a label. Sync every app once after upgrading.
In short : ApplicationSets stamp out Applications per cluster or folder, install and upgrade go server-side since 3.3, and Argo CD is the UI-and-SSO cousin of Flux.