Storage internals

Volume Topology and Scheduling

1.Where a volume can live

  • Topology = the PV spec.nodeAffinity. Terms are ORed, expressions in a term ANDed.
  • CSINode topologyKeys are key names. The values are Node labels.
  • Immediate is the default. The mode is immutable.
ModeImmediateWFFC
Volume madePVC createdPod scheduled
Zone fromprovisionerscheduler's node
Capacityneverif storageCapacity
Pod, unboundPendingnormal
Riskfull zonestale selected-node

2.VolumeBinding in the framework

  • A missing class or a pre-bound claim counts as immediate.
  • bindTimeoutSeconds: 0 fails at once, not "no wait". The waiting node shows as NOMINATED NODE.
  • UnschedulableAndUnresolvable : evicting cannot help, so no preemption (Advanced Scheduling).
PreFilter : bound / delayed / immediate Filter per node : PV affinity, smallest static PV, allowedTopologies + capacity Score : free capacity (weight 1) Reserve : assume, cache only PreBind : selected-node, poll 1s, 600s Pod bound Unschedulable AndUnresolvable = no preemption

3.The handshake, end to end

PVC events : WaitForFirstConsumer, WaitForPodScheduled, ExternalProvisioning. Basics : Volumes and Storage. A nodeName pin skips the scheduler : a bound PV fails MountVolume.NodeAffinity check failed, a WFFC claim never binds.

kubelet node-b attach, mount kube-scheduler VolumeBinding PV nodeAffinity PVC data-0 WFFC class CSINode node-b topologyKeys external-provisioner Node node-b labels (values) CSI driver 1 selected-node 2 watch 3 keys 3 values 4 CreateVolume (requisite, preferred) 5 accessible_topology [{zone: b}] 6 zone In [b] 7 bind 8 Pod bound 9 affinity re-check

4.What CreateVolume receives

  • Plain WFFC : requisite is the aggregated topology, every Node with the key labels, driver or not.
  • allowedTopologies keys must be driver keys, or ... is not in requisite.
  • Node-local storage : --strict-topology.
Node in zone brequisitepreferred
WFFC, plaina, b, cb, c, a
WFFC, strictbb
WFFC, allowed a, ba, bb, a
Immediatea, b, chash(set) + ordinal
Immediate, topology offnonenone

5.Provisioning fails on the node

  • Only RESOURCE_EXHAUSTED (or an uncached CSINode NotFound) frees the Pod.
  • v6.0.0 and v6.1.0 could retry in the wrong zone : use v6.0.1, v6.1.1, v6.2.0+.
CreateVolumeProvisionerPod
RESOURCE_EXHAUSTEDdrops the annotationrescheduled
INVALID_ARGUMENT5m slow retrypinned
anything elsesame node, backoffpinned, 600s

6.Commands

CommandShows
kubectl get pvc <pvc> -o jsonpath='{.metadata.annotations.volume\.kubernetes\.io/selected-node}'Node picked (empty if static)
kubectl get pod <pod> -o jsonpath='{.status.conditions[?(@.type=="PodScheduled")].message}'Current verdict (events lag)
kubectl get events --field-selector involvedObject.name=<pvc>PVC events, ProvisioningFailed
kubectl describe pv <pv>Node Affinity, Term 0
In short : With WaitForFirstConsumer the scheduler picks the node and the driver's answer becomes the PV's nodeAffinity. With Immediate the volume picks first and the Pod must follow.

1.Storage capacity tracking

  • Opt in : CSIDriver storageCapacity: true and provisioner --enable-capacity. Objects land in the provisioner's namespace.
  • Only new volumes of WFFC classes are checked. No match : did not have enough free storage.
CSI driver external-provisioner GetCapacity per segment create / update, every 1m and after create, delete CSIStorageCapacity class, nodeTopology, capacity reads kube-scheduler Filter + Score

A hint only, no bookkeeping, unlike Dynamic Resource Allocation.

2.Why the numbers are only hints

  • Each claim checked alone, new volumes not subtracted, data up to 1m old.
  • Flag true before any object exists blocks every node : ship false, then flip.
  • Matched by class name only : write access to csistoragecapacities steers scheduling (RBAC).
FieldRuleUsed by
nodeTopologynil = none, {} = allFilter
storageClassNamename onlyFilter
capacityavailable_capacityFilter, Score
maximumVolumeSizeoptionalFilter first

The first two are immutable. Flag turned off : PVC Pods wait for the 5 min flush.

3.Storage scoring in v1.37

  • StorageCapacityScoring : beta and on.
  • Dynamic : most free space wins. Static : the node whose smallest fitting PV is largest wins.
  • Weight 1. Gate off with a shape set : kube-scheduler will not start.
Claim, space%Def.Inv.
10Gi, 100Gi109010
10Gi, 50Gi208020
50Gi, fit 200Gi2575-
50Gi, fit 100Gi5050-
50Gi, PVs 60 + 500Gi8317-

4.Attach slots per node

  • NodeVolumeLimits checks unique volumes against CSINode allocatable.count (in-tree EBSLimits and co : removed v1.32). Over : exceed max volume count, preemption may run.
  • nodeAllocatableUpdatePeriodSeconds (>= 10, GA v1.36) and external-attacher v4.10.0+ : a code 8 attach fails the Pod for a new one.
CSI driver max_volumes_per_node kubelet NodeGetInfo every period CSINode allocatable.count scheduler NodeVolumeLimits external-attacher ControllerPublish RESOURCE_ EXHAUSTED code 8 kubelet : refresh count Pod Failed VolumeAttachmentLimitExceeded controller makes a new Pod

5.Nodes without the driver

  • VolumeLimitScaling (beta, on) : the flag rejects driverless nodes : <driver> CSI driver is not installed on the node.
  • Only with CA --enable-csi-node-aware-scheduling=true (1.35.0+, default false) or no autoscaler. Karpenter : out of scope.
kind: CSIDriver   # name: ebs.csi.aws.com
spec:
  attachRequired: true
  nodeAllocatableUpdatePeriodSeconds: 60
  preventPodSchedulingIfMissing: true
# CA flag first, or no autoscaler

6.Commands

CommandShows
kubectl describe csinode <node>Node ID, count, topology keys
kubectl get csidriver <d> -o jsonpath='{.spec.preventPodSchedulingIfMissing} {.spec.nodeAllocatableUpdatePeriodSeconds}'The two fields get hides
kubectl get csistoragecapacities -A -o custom-columns='SC:.storageClassName,CAP:.capacity,MAX:.maximumVolumeSize,TOPO:.nodeTopology.matchLabels'What the scheduler matches
kubectl get volumeattachments -o custom-columns='NODE:.spec.nodeName,CODE:.status.attachError.errorCode'Code 8 = RESOURCE_EXHAUSTED
In short : The scheduler counts two scarce things per node : free space, from CSIStorageCapacity hints, and attach slots, from CSINode counts. v1.37 scores space by default.

1.Who says no, and can preemption help ?

  • Each node reports only its first failing filter, so one reason can hide another. Alerts on had volume node affinity conflict (renamed v1.33) are silent.
PluginReason, node(s) ...StatusPreempt ?
VolumeBindingdidn't match PersistentVolume's node affinity, didn't find available persistent volumes to bind, did not have enough free storage. PreFilter : pod has unbound immediate PersistentVolumeClaims...Unresolvablenever
VolumeZonehad no available volume zone (legacy PV zone labels only)...Unresolvablenever
NodeVolumeLimitsexceed max volume count, <driver> CSI driver is not installedUnschedulablepossible
VolumeRestrictionsRWOP claim already in-use by another podUnschedulablepossible

...Unresolvable = UnschedulableAndUnresolvable, also given by the last two for a missing PVC.

2.Reading one FailedScheduling message

Format : Advanced Scheduling. A PreFilter failure gives one reason. A trailing . not found in v1.37 is DRA noise.

0/3 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 2 node(s) had untolerated taint(s). preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. all nodes, control plane too first failing filter per node volume reasons never preempt

3.Node-local storage

  • local-static-provisioner v2.9.0 : one PV per disk, can drop a startup taint once disks are found.
  • Node-local CSI (TopoLVM, host-path) : strict topology, provisioner central or per node.
  • Node deleted : PV stays, Pod Pending. Same name reused : wrong disk.
local-staticNode-local CSI
Volumespre-made diskscarved per claim
Keykubernetes.io/hostnamedriver key
Classno-provisioner + WFFCdriver + WFFC
Scheduler seesPV sizesCSIStorageCapacity
v1.37 scorelargest best fitmost free space
Node lossnode-cleanup (opt-in)force-delete PV

4.A zone dies under a StatefulSet

  • A bound replica fails PV affinity everywhere, spare PVs or not. Dropping its PVC still hits zone spread unless nodeTaintsPolicy: Honor (GA v1.33) skips tainted nodes. Old Pod gone first : Volume Operations.
  • Or restore (Snapshots, Clones and Backup), or go regional, widen PV affinity (alpha MutablePVNodeAffinity).
created : db-0 zone-b, db-1 zone-a zone-a lost (NoExecute taint) db-1 Pending : PV node affinity delete PVC data-db-1 and Pod db-1 nodeTaintsPolicy unset : didn't match pod topology spread constraints Honor : binds spare PV in zone-b, runs observed, v1.37.0

5.Autoscalers and volume zones

  • cluster-autoscaler : group per zone. Per-node CSIStorageCapacity blocks scale-up : add one for template-node=true.
  • Karpenter : adds a bound PV's zone, picks one allowedTopologies zone at random, no region topology (Karpenter).
  • Attach limits : CA --enable-csi-node-aware-scheduling. Karpenter is out of KEP-5030's scope.

6.Commands

CommandShows
kubectl get events --field-selector involvedObject.name=<pod>,reason=FailedSchedulingReason history
kubectl get pv -o custom-columns='PV:.metadata.name,KEYS:.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].key'PV topology keys
kubectl delete pv <pv> --wait=false --grace-period=0 --force, then clear its finalizersDrop a dead node's PV
kubectl -n kube-system logs <scheduler> | grep 'Plugin scored node for pod'Scores (--v=10)
In short : Binding and zone rejections never preempt, attach-limit and RWOP ones can. Plan for node and zone loss : nodeTaintsPolicy: Honor, a node group per zone, a restore path.