Core Kubernetes · 10
StatefulSets and DaemonSets
1.Why a StatefulSet ?
Deployment Pods are interchangeable. A StatefulSet is a set of Pods with names. It promises four things. The first two survive a reschedule :
- a fixed name and DNS name per Pod
- its own storage per Pod
- ordered start and scaling
- ordered rolling updates
Need none of them ? Use a Deployment (Deployments and Scaling). Stable apps/v1 since v1.9.
| A Pod in a | Deployment | StatefulSet |
|---|---|---|
| Name | <rs>-<random> | <sts>-<ordinal> |
| Replaced by | a new Pod, new name | same name, same PVC |
| DNS name | none per Pod | one per Pod |
| Storage | shared PVC or none | own PVC per Pod |
| Order | any | 0 first, top down |
2.Identity : name, DNS name, PVC
| Piece | What web-1 gets |
|---|---|
| Name, hostname | web-1 |
| DNS name | web-1. |
| PVC | www-web-1 : template, set, ordinal |
| Labels | apps., statefulset. |
The pod-name label lets a Service pick one replica. ordinals.start (GA v1.31) shifts the range : start 1 gives db-1 to db-3.
3.The headless Service
- You create it and name it in
serviceName. Deleting the set leaves it behind. - Optional since v1.33, but no DNS names without it. Nobody checks that it exists.
- DNS lists Ready Pods only.
publishNotReadyAddresses: truelets peers meet while booting. - Use names, not Pod IPs : IPs change. See Services.
4.Own PVC, in short
volumeClaimTemplates: one PVC per template per ordinal. A replaced Pod mounts the same one.- PVCs stay on scale down and delete, unless the retention policy (GA v1.32) says Delete.
- Templates are immutable : grow each PVC by hand (Volumes and Storage).
- Zonal disks : use
WaitForFirstConsumer.
5.Order : OrderedReady vs Parallel
| OrderedReady (default) | Parallel | |
|---|---|---|
| Scale up | 0 up, each waits for lower Pods Running and Ready | at once |
| Scale down | from the top, each waits until the previous is gone | at once |
| Sick Pod | scale down waits, unless the sick one goes next | no wait |
| Change it | immutable : orphan delete, recreate | |
minReadySeconds makes each step wait for Available too. A Failed Pod (eviction, reboot) comes back with the same name.
6.At most one web-1
Never two Pods with one name : a Pod on a lost node stays Terminating, and no twin starts. Ways out, safest first :
- Machine really gone : delete the Node object.
- Node confirmed shut down : taint it
node.. Pods are force-deleted, volumes detach. Remove the taint after.kubernetes. io/ out-of-service - Last resort : force delete the Pod. Risks split brain.
- Never
terminationGracePeriodSeconds: 0. - Deleting the set is not ordered : scale to 0 first.
- With an HPA, leave
replicasout of the manifest.
1.Rolling update and partition
A template change makes a new ControllerRevision. RollingUpdate (default) replaces Pods from the highest ordinal down : same name, same PVC, new template.
- Each step waits for Running and Ready, plus
minReadySeconds(Rollouts and Probes). partition(default 0) : only Pods at or above it update. Lower ones stay old, even when deleted.- It counts from
ordinals.start: with start 5, partition 1 updates ord-6 and up.
2.Three update strategies
| RollingUpdate (default) | OnDelete | Recreate (alpha, v1.37) | |
|---|---|---|---|
| Who replaces | controller, top down | you, by deleting Pods | controller, all at once |
| Old and new | side by side | side by side, no limit | never : downtime |
| rollout status | works | refuses | refuses |
| Switch with | default | "rollingUpdate": null | same, plus a feature gate |
OnDelete : scaled-up Pods get the new template too. Recreate needs StatefulSetRecreateStrategy on kube-apiserver and kube-controller-manager; beta is planned for v1.38, not promised.
3.maxUnavailable (Beta)
- Several Pods per step. On by default in v1.37 (default 1). In v1.35.4+ and v1.36 the field is dropped silently.
- A percent rounds down, minimum 1. The docs say up.
- OrderedReady : several go, but they return one at a time. Parallel : together. Every unavailable Pod counts.
| maxUnavailable | Pods per step |
|---|---|
| 1 (default) | 1 |
| 30% | 1 |
| 60% | 2 |
| 70% | 2 |
4 replicas, seen on v1.37.1. 60% again on a kind cluster.
4.The stuck rollout
A template that never gets Ready stops an OrderedReady rollout. Old Pods keep serving.
rollout undoalone is not enough : the controller waits on the stuck Pod forever.- Delete it : it comes back on the reverted template. Parallel (v1.37) and Recreate do this for you.
- Upstream : working as intended, a Pod may hold data.
5.Revisions and commands
| Command | What it is for |
|---|---|
kubectl get pods -l app=nginx -L controller-revision-hash | Which Pod runs which revision |
kubectl patch sts web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}' | Canary : only web-2 takes the new template |
kubectl rollout status sts/web --timeout=10m | The only deadline you get. Checks Ready, not minReadySeconds |
kubectl rollout undo sts/web --to-revision=3 | Pod template only. The old revision returns with a new number |
kubectl annotate sts/web kubernetes.io/change-cause='bump nginx' | CHANGE-CAUSE in rollout history (--record is deprecated) |
10 ControllerRevisions kept. No progressDeadlineSeconds, rollout pause fails. PDBs limit drains, not updates. GitOps ? Revert in Git.
partition stages a canary, OnDelete waits for you, and a Pod stuck on a bad template must be deleted by hand after the revert.1.DaemonSet : one Pod per node
A DaemonSet has no replica count : the list of eligible nodes is the count. A node joins, it gets a Pod. It leaves, its Pod is garbage-collected.
- For node work : CNI and CSI node plugins, log and metrics agents, kube-proxy.
- No
/scale: no HPA,kubectl scale dsfails. --cascade=orphankeeps its Pods for the next DaemonSet with that selector.
2.Which nodes get a Pod ?
nodeNamein the template, if set, must match.nodeSelectorand required node affinity must match.- Every NoSchedule and NoExecute taint must be tolerated.
CPU and memory are not checked : a full node leaves the Pod Pending. Free room, or use priorityClassName: system-node-critical. Each Pod is pinned with matchFields metadata.name and the scheduler binds it (Advanced Scheduling).
| Added toleration | Effect | So the Pod |
|---|---|---|
| not-ready, unreachable | NoExecute | never evicted |
| disk, memory, pid pressure | NoSchedule | on busy nodes |
| unschedulable | NoSchedule | on cordoned nodes |
| network-unavailable | NoSchedule | hostNetwork only |
not-ready:NoSchedule: a NotReady node gets no new Pod. CNI and kube-proxy tolerate it themselves.- The control-plane taint : add it yourself.
3.Updates : maxUnavailable or maxSurge
- Default : maxUnavailable 1, maxSurge 0. Percents round up.
maxSurgeneedsmaxUnavailable: 0. Old goes once new is Available.- OnDelete : only deleted Pods and new nodes update.
- Bad template ? Push a fixed one : broken Pods go first, and use up the budget.
4.Deployment vs StatefulSet vs DaemonSet
| Deployment | StatefulSet | DaemonSet | |
|---|---|---|---|
| How many | replicas, HPA | replicas, HPA | one per eligible node |
| Storage | shared PVC or none | own PVC per ordinal | hostPath, CSI node mounts |
| Order | none | OrderedReady or Parallel | none |
| Strategies | RollingUpdate, Recreate | RollingUpdate, OnDelete, Recreate (alpha) | RollingUpdate, OnDelete |
| Deadline, pause | yes | no | no |
| Good for | stateless web, APIs | databases, quorum, brokers | CNI, CSI, log and metrics agents |
5.Troubleshooting
| Symptom | Check | Likely cause, fix |
|---|---|---|
| Stuck at N-1/N | kubectl get pods -L controller-revision-hash | bad template : revert, delete the stuck Pod |
| No StatefulSet Pods | kubectl describe sts web | name over 52 characters, Pod Security, quota |
| Pending after a move | kubectl describe pod web-2 | its volume is pinned to another zone |
| DESIRED too low | node taints and labels | untolerated taint, or nodeSelector |
| DaemonSet CURRENT 0 | kubectl describe ds agent | Pod Security : use a privileged namespace (Pod Security) |
| Drain refuses | kubectl drain output | add --ignore-daemonsets : they tolerate cordon |
| DaemonSet rollout hangs | events, dead nodes | hostPort with maxSurge, or a Pod stuck Terminating |