Storage internals

Volume Operations and Failure Modes

1.Who writes what

  • Basics : Volumes and Storage. Gate : PersistentVolumeClaimResize admission (on). Its only dynamically provisioned pvc can be resized misleads : static PVs on an expandable class grow.
  • CSI : external-resizer does the controller step and status, the kubelet the node step.
user edits PVC spec size or VAC name kube-apiserver validation, Resize, Quota kcm expander ExternalExpanding, hands off watch external-resizer controller step calls CSI controller ControllerExpandVolume ControllerModifyVolume writes PVC status allocatedResources, statuses, conditions (+ PV size, VAC) NodeResizePending status.capacity kubelet online or at next mount calls CSI node plugin NodeExpandVolume

2.Expansion, step by step

  • Resizer : allocatedResources, condition Resizing, then ControllerExpandVolume (10s timeout). Node step needed : condition FileSystemResizePending.
  • Offline-only driver : FAILED_PRECONDITION while published. The resizer waits for zero Pods.
  • An unused PVC never finishes : Waiting for user to (re-)start a pod.
(no key) resizer ControllerResizeInProgress resizer NodeResizePending kubelet NodeResizeInProgress kubelet key removed, capacity updated ControllerResizeInfeasible NodeResizeInfeasible lower the request red = driver refused

3.Status values

Exactly five are accepted. The API reference still names ControllerResizeFailed and NodeResizeFailed : both are rejected.

ValueByMeans
ControllerResizeInProgressresizerstarted
ControllerResizeInfeasibleresizerdriver refused
NodeResizePendingresizergrow the FS
NodeResizeInProgresskubeletnode call running
NodeResizeInfeasiblekubeletnode refused

Infeasible : InvalidArgument, OutOfRange, NotFound. Timeouts : still running.

4.Undo a typo (GA v1.34)

  • Lower to anything strictly above status.capacity. Quota (max of spec and allocated) drops with it.
  • No help while ControllerResizeInProgress holds, or once only the node step failed.
  • Editing a PV's capacity cancels the resize. Raising the PVC to match then fakes a grown backend.
$ kubectl patch pvc data ... 1000Gi
1000Gi ControllerResizeInfeasible 10Gi
$ kubectl patch pvc data ... 100Gi  # ok
$ ... 10Gi  # Forbidden: can not be less

5.VolumeAttributesClass

  • Edit IOPS of a Bound PVC. GA v1.34, v1beta1 fails on v1.37.
  • Immutable classes, no default, vac-protection on each.
  • Cancel = old class back. get pvc : desired, get pv : applied.
  • A VAC quota checks create and spec. Status can pass hard.
ExpansionVAC modify
Spec fieldrequests.storageVAC name
Allowed onexpandable classMODIFY_VOLUME
ProgressallocatedResource*modifyVolumeStatus
ControllerControllerExpandControllerModify
Node stepNodeExpandVolumenone
Infeasible3 codesINVALID_ARGUMENT
Undolower, > capacityprevious class

6.Commands

CommandShows
kubectl get pvc data -o jsonpath='{.spec.resources.requests.storage}{" alloc="}{.status.allocatedResources.storage}{" state="}{.status.allocatedResourceStatuses.storage}{" cap="}{.status.capacity.storage}{"\n"}'State in one line. Read it first
kubectl events --for pvc/dataResize and VolumeModify events
kubectl get pvc data -o jsonpath='{.status.currentVolumeAttributesClassName}{" "}{.status.modifyVolumeStatus}{"\n"}'Applied class, change in flight
kubectl -n <ns> get resourcequota <q> -o jsonpath='{.status.used}{" / "}{.status.hard}{"\n"}'VAC quota over hard
In short : external-resizer runs both changes. Infeasible = driver refused : change the spec back.

1.Two hidden walks

  • A mount can walk the whole volume twice : kubelet fsGroup chown, runtime SELinux relabel. Both grow with file count.
  • VOLUME_MOUNT_GROUP driver : gets volume_mount_group, no walk, both policy fields ignored.
VOLUME_MOUNT_GROUP driver ? read-only mount ? CSIDriver fsGroupPolicy default : fsType, RWO(P) ? OnRootMismatch, root matches ? recursive walk no no yes no File driver does it yes skip the walk yes None no yes

2.What the fsGroup walk does

  • Lchown to the GID, OR in 0660 (0440 ro), setgid on dirs. Never removes bits.
  • OnRootMismatch skips if the root has the GID, 0770 (0550 ro) and setgid.
  • 30s : VolumePermissionChangeInProgress, then every 60s (CSI only).
  • RWX skipped by default : NFS needs File.
LeverWhereEffect
OnRootMismatchPodskip if root matches
Always, or unsetPodwalk every mount
fsGroupPolicy: NoneCSIDrivernever apply
fsGroupPolicy: FileCSIDriveralways, even RWX
VOLUME_MOUNT_GROUPdriverdriver does it
readOnly: truePodskipped

3.SELinux : a mount option

  • PV, seLinuxOptions.level, driver seLinuxMount: true : mount -o context. Else (secret, configMap ...) a full relabel.
  • RWOP and change-policy gates locked v1.36. SELinuxMount GA v1.37, unlocked : =false in v1.37 only.
MountOptionRecursive
Who labelskubelet, -o contextruntime walk
Costconstantgrows with files
One nodesame label onlyany labels
Needslabel, opt-innothing

4.Label conflicts from v1.37

  • Two labels (or privileged + not) on one volume and node : the 2nd Pod hangs. Fix : Recursive or one label.
  • Audit first : selinux-warning-controller (off, Controllers). Its metric names Pods : admin-only. Kubelet : volume_manager_selinux_*.
Pod A label s0:c1 mounted on node-1 -o context=s0:c1 Pod B, same node label s0:c2 ContainerCreating conflicting SELinux labels both Pods set policy Recursive both run runtime relabels, slower start selinux-warning-controller flags the pair before they meet

5.Access modes, enforced where

  • See Volumes, Topology. RWO is one node at attach unless the PV lists RWX, ROX or nothing.
  • recursiveReadOnly: IfPossible may fall back silently. Use Enabled.
ModeSchedulerA/D ctrlKubelet
RWOno checkone nodeothers may write
RWOPpreemptsone noderefuses a 2nd Pod
RWX, ROXnonemany nodesnone

6.Commands

CommandShows
kubectl get csidriver <d> -o jsonpath='{.spec}'fsGroupPolicy, seLinuxMount
kubectl patch csidriver <d> --type=merge -p '{"spec":{"fsGroupPolicy":"None","seLinuxMount":false}}'Mutable. attachRequired is not
kubectl get events -A --field-selector reason=SELinuxLabelConflictPotential conflicts (Normal)
kubectl get --raw /metrics | grep kubernetes_feature_enabledGate states (SELinuxMount 1)
kubectl get pod <p> -o jsonpath='{range .status.containerStatuses[*].volumeMounts[*]}{.name}{" rro="}{.recursiveReadOnly}{"\n"}{end}'Recursive read-only took effect ?
In short : OnRootMismatch, fsGroupPolicy and -o context skip the walks. v1.37 : one label per volume per node.

1.Attach and detach, the normal path

  • A second node is refused unless the PV lists RWX or ROX : Waiting for detach (was Multi-Attach error before v1.37.0, v1.36.4, v1.35.9). Reason is still FailedAttachVolume. In a rollout it is usually a race.
  • The 6m clock (hard-coded) starts once the Pod is gone or terminated. Call order : CSI Drivers.
Pod gone or terminated attach-detach ctrl not desired, 6m clock yes : wait in volumesInUse ? kubelet still mounted no VA deleted ControllerUnpublish, then finalizer off not Ready, over 6m force detach (timeout) out-of-service detach now red : skip volumesInUse, NodeUnpublish and NodeUnstage

2.A node dies under RWO

  • Terminating Pods look running : no clock. Deleting the Node fails too.
  • Node off : out-of-service NoExecute taint (StatefulSets). Instant detach, no health check : never on a rebooting node. Untaint after.
  • Or force-delete the Pod : detach after 6m unless --disable-force-detach-on-timeout.
PathTriggerSkipsRisk
Normalunmountednothingnone
Timeout6m, NotReadyin-use checkcorruption
Tainttaint key+ no waitsame
VA finalizerby handUnpublishstill attached

3.New signals

  • Unused : Bound PVCs, Pending Pods count. Nothing acts.
  • Health : redesigned, off, writes dropped.
  • Stats need GET_VOLUME_STATS (not KSM).
SignalByv1.37
Unused conditionkcmbeta, on
3 health fieldssidecar, kubeletalpha, off
kubelet_volume_stats_*kubelet, 1malpha
..._forced_detacheskcmno _total

4.Symptom, cause, fix

FailedAttachVolume = attach side. FailedMount (FailedMapVolume for Block) = node side. A plain timeout posts no summary event since v1.28 : read the kubelet log. FileSystemResizeFailed is read-only mounts only.

Event reasonMessage startsCauseFix
FailedAttachVolumeWaiting for detachRWO in use elsewherewait, Recreate, or taint a dead node
FailedAttachVolumeAttachVolume.Attach failedControllerPublish errorVA attachError, attacher logs
FailedMountMountVolume.MountDevice / SetUp failedNodeStage / NodePublishnode plugin logs
FailedMount... failed while expanding volumenode expansionnode logs, lower if infeasible
FailedMount... conflicting SELinux labelstwo labels, one nodeRecursive, or one label
ExternalExpandingnothing after itno resizer for the driverdeploy csi-resizer
VolumeResizeFailedonly supports offline / OutOfRangeoffline driver / infeasiblestop Pods / lower request
VolumeModifyFailedInvalidArgumentmodify Infeasibleset previous class

Missing attacher or driver : CSI Drivers. VolumeAttachmentLimitExceeded : Volume Topology. Restore as the way back : Snapshots.

5.Commands

CommandShows
kubectl get volumeattachment -o custom-columns=PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,ATTACHED:.status.attached,DEL:.metadata.deletionTimestamp,FIN:.metadata.finalizers,ERR:.status.detachError.messageStuck detaches : fix the driver, keep the finalizer
kubectl get node <n> -o jsonpath='{.status.volumesInUse}{"\n"}{.status.volumesAttached}{"\n"}'Kubelet view, controller view
kubectl taint nodes <n> node.kubernetes.io/out-of-service=nodeshutdown:NoExecuteNode is off. Add - to remove
kubectl get events -A --field-selector reason=FailedAttachVolumeAny wording
kubectl get pvc <p> -o jsonpath='{.status.conditions[?(@.type=="Unused")]}'Unused since
kubectl get --raw "/api/v1/nodes/<n>/proxy/metrics" | grep '^kubelet_volume_stats_available_bytes'Fill per PVC (nodes/proxy)
In short : A volume leaves when the kubelet lets go or the node is proven gone. Alert on reasons.