Core Kubernetes · 10

StatefulSets and DaemonSets

1.Why a StatefulSet ?

Deployment Pods are interchangeable. A StatefulSet is a set of Pods with names. It promises four things. The first two survive a reschedule :

  1. a fixed name and DNS name per Pod
  2. its own storage per Pod
  3. ordered start and scaling
  4. ordered rolling updates

Need none of them ? Use a Deployment (Deployments and Scaling). Stable apps/v1 since v1.9.

A Pod in aDeploymentStatefulSet
Name<rs>-<random><sts>-<ordinal>
Replaced bya new Pod, new namesame name, same PVC
DNS namenone per Podone per Pod
Storageshared PVC or noneown PVC per Pod
Orderany0 first, top down
Not an operator : no leader election, replication, failover or backups. That is what Operators add (see CloudNativePG).

2.Identity : name, DNS name, PVC

you create it Service nginx clusterIP: None a DNS name per Pod web-0 web-0.nginx web-1 web-1.nginx web-2 web-2.nginx mounts www-web-0 www-web-1 www-web-2 web-1 moves : same name, same DNS name, same PVC
PieceWhat web-1 gets
Name, hostnameweb-1
DNS nameweb-1.nginx.<ns>.svc.cluster.local
PVCwww-web-1 : template, set, ordinal
Labelsapps.kubernetes.io/pod-index: "1", statefulset.kubernetes.io/pod-name: web-1

The pod-name label lets a Service pick one replica. ordinals.start (GA v1.31) shifts the range : start 1 gives db-1 to db-3.

3.The headless Service

  • You create it and name it in serviceName. Deleting the set leaves it behind.
  • Optional since v1.33, but no DNS names without it. Nobody checks that it exists.
  • DNS lists Ready Pods only. publishNotReadyAddresses: true lets peers meet while booting.
  • Use names, not Pod IPs : IPs change. See Services.

4.Own PVC, in short

  • volumeClaimTemplates : one PVC per template per ordinal. A replaced Pod mounts the same one.
  • PVCs stay on scale down and delete, unless the retention policy (GA v1.32) says Delete.
  • Templates are immutable : grow each PVC by hand (Volumes and Storage).
  • Zonal disks : use WaitForFirstConsumer.

5.Order : OrderedReady vs Parallel

OrderedReady (default)Parallel
Scale up0 up, each waits for lower Pods Running and Readyat once
Scale downfrom the top, each waits until the previous is goneat once
Sick Podscale down waits, unless the sick one goes nextno wait
Change itimmutable : orphan delete, recreate
scale up web-0 Ready ? web-1 Ready ? web-2 scale down web-2 gone ? web-1 gone ? web-0 Parallel : all at once, no waiting

minReadySeconds makes each step wait for Available too. A Failed Pod (eviction, reboot) comes back with the same name.

6.At most one web-1

Never two Pods with one name : a Pod on a lost node stays Terminating, and no twin starts. Ways out, safest first :

  1. Machine really gone : delete the Node object.
  2. Node confirmed shut down : taint it node.kubernetes.io/out-of-service. Pods are force-deleted, volumes detach. Remove the taint after.
  3. Last resort : force delete the Pod. Risks split brain.
Careful
  • Never terminationGracePeriodSeconds: 0.
  • Deleting the set is not ordered : scale to 0 first.
  • With an HPA, leave replicas out of the manifest.
In short : A StatefulSet gives each Pod an ordinal name, a DNS name via the headless Service you create, and its own PVC, then starts Pods from 0 and removes them from the top.

1.Rolling update and partition

A template change makes a new ControllerRevision. RollingUpdate (default) replaces Pods from the highest ordinal down : same name, same PVC, new template.

  • Each step waits for Running and Ready, plus minReadySeconds (Rollouts and Probes).
  • partition (default 0) : only Pods at or above it update. Lower ones stay old, even when deleted.
  • It counts from ordinals.start : with start 5, partition 1 updates ord-6 and up.
3rd 2nd 1st web-0 rev 1 to 2 web-1 rev 1 to 2 web-2 rev 1 to 2 Ready ? Ready ? staged, with partition : Stage partition: 3 Canary partition: 2 Finish partition: 0 nothing moves only web-2 web-1, web-0

2.Three update strategies

RollingUpdate (default)OnDeleteRecreate (alpha, v1.37)
Who replacescontroller, top downyou, by deleting Podscontroller, all at once
Old and newside by sideside by side, no limitnever : downtime
rollout statusworksrefusesrefuses
Switch withdefault"rollingUpdate": nullsame, plus a feature gate

OnDelete : scaled-up Pods get the new template too. Recreate needs StatefulSetRecreateStrategy on kube-apiserver and kube-controller-manager; beta is planned for v1.38, not promised.

3.maxUnavailable (Beta)

  • Several Pods per step. On by default in v1.37 (default 1). In v1.35.4+ and v1.36 the field is dropped silently.
  • A percent rounds down, minimum 1. The docs say up.
  • OrderedReady : several go, but they return one at a time. Parallel : together. Every unavailable Pod counts.
maxUnavailablePods per step
1 (default)1
30%1
60%2
70%2

4 replicas, seen on v1.37.1. 60% again on a kind cluster.

4.The stuck rollout

A template that never gets Ready stops an OrderedReady rollout. Old Pods keep serving.

  • rollout undo alone is not enough : the controller waits on the stuck Pod forever.
  • Delete it : it comes back on the reverted template. Parallel (v1.37) and Recreate do this for you.
  • Upstream : working as intended, a Pod may hold data.
set image : bad tag new revision web-2 not Ready rollout stops rollout undo template reverted web-2 still stuck same bad revision delete pod web-2 your manual step web-2 recreated rollout goes on

5.Revisions and commands

CommandWhat it is for
kubectl get pods -l app=nginx -L controller-revision-hashWhich Pod runs which revision
kubectl patch sts web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'Canary : only web-2 takes the new template
kubectl rollout status sts/web --timeout=10mThe only deadline you get. Checks Ready, not minReadySeconds
kubectl rollout undo sts/web --to-revision=3Pod template only. The old revision returns with a new number
kubectl annotate sts/web kubernetes.io/change-cause='bump nginx'CHANGE-CAUSE in rollout history (--record is deprecated)

10 ControllerRevisions kept. No progressDeadlineSeconds, rollout pause fails. PDBs limit drains, not updates. GitOps ? Revert in Git.

In short : Updates go from the highest ordinal down, partition stages a canary, OnDelete waits for you, and a Pod stuck on a bad template must be deleted by hand after the revert.

1.DaemonSet : one Pod per node

A DaemonSet has no replica count : the list of eligible nodes is the count. A node joins, it gets a Pod. It leaves, its Pod is garbage-collected.

  • For node work : CNI and CSI node plugins, log and metrics agents, kube-proxy.
  • No /scale : no HPA, kubectl scale ds fails.
  • --cascade=orphan keeps its Pods for the next DaemonSet with that selector.
DaemonSet agent one Pod per eligible node node-1 Ready agent node-2 cordoned agent tolerated node-3 new, NotReady no Pod not tolerated cp-1 control plane no Pod not tolerated

2.Which nodes get a Pod ?

  1. nodeName in the template, if set, must match.
  2. nodeSelector and required node affinity must match.
  3. Every NoSchedule and NoExecute taint must be tolerated.

CPU and memory are not checked : a full node leaves the Pod Pending. Free room, or use priorityClassName: system-node-critical. Each Pod is pinned with matchFields metadata.name and the scheduler binds it (Advanced Scheduling).

Added tolerationEffectSo the Pod
not-ready, unreachableNoExecutenever evicted
disk, memory, pid pressureNoScheduleon busy nodes
unschedulableNoScheduleon cordoned nodes
network-unavailableNoSchedulehostNetwork only
Not added
  • not-ready:NoSchedule : a NotReady node gets no new Pod. CNI and kube-proxy tolerate it themselves.
  • The control-plane taint : add it yourself.

3.Updates : maxUnavailable or maxSurge

  • Default : maxUnavailable 1, maxSurge 0. Percents round up.
  • maxSurge needs maxUnavailable: 0. Old goes once new is Available.
  • OnDelete : only deleted Pods and new nodes update.
  • Bad template ? Push a fixed one : broken Pods go first, and use up the budget.
maxUnavailable: 1 old Pod rev 1 no Pod a gap new Pod rev 2 maxSurge: 1 old Pod rev 1 old + new 2 Pods new Pod old removed one node, during the update

4.Deployment vs StatefulSet vs DaemonSet

DeploymentStatefulSetDaemonSet
How manyreplicas, HPAreplicas, HPAone per eligible node
Storageshared PVC or noneown PVC per ordinalhostPath, CSI node mounts
OrdernoneOrderedReady or Parallelnone
StrategiesRollingUpdate, RecreateRollingUpdate, OnDelete, Recreate (alpha)RollingUpdate, OnDelete
Deadline, pauseyesnono
Good forstateless web, APIsdatabases, quorum, brokersCNI, CSI, log and metrics agents

5.Troubleshooting

SymptomCheckLikely cause, fix
Stuck at N-1/Nkubectl get pods -L controller-revision-hashbad template : revert, delete the stuck Pod
No StatefulSet Podskubectl describe sts webname over 52 characters, Pod Security, quota
Pending after a movekubectl describe pod web-2its volume is pinned to another zone
DESIRED too lownode taints and labelsuntolerated taint, or nodeSelector
DaemonSet CURRENT 0kubectl describe ds agentPod Security : use a privileged namespace (Pod Security)
Drain refuseskubectl drain outputadd --ignore-daemonsets : they tolerate cordon
DaemonSet rollout hangsevents, dead nodeshostPort with maxSurge, or a Pod stuck Terminating
In short : A DaemonSet runs one Pod on each node its selectors and taints allow, tolerates node trouble by itself, and updates node by node : off then on, or side by side with maxSurge.