Basics : Volumes and Storage. Gate : PersistentVolumeClaimResizeadmission (on). Its only dynamically provisioned pvc can be resized misleads : static PVs on an expandable class grow.
CSI : external-resizer does the controller step and status, the kubelet the node step.
Offline-only driver : FAILED_PRECONDITION while published. The resizer waits for zero Pods.
An unused PVC never finishes : Waiting for user to (re-)start a pod.
3.Status values
Exactly five are accepted. The API reference still names ControllerResizeFailed and NodeResizeFailed : both are rejected.
Value
By
Means
ControllerResizeInProgress
resizer
started
ControllerResizeInfeasible
resizer
driver refused
NodeResizePending
resizer
grow the FS
NodeResizeInProgress
kubelet
node call running
NodeResizeInfeasible
kubelet
node refused
Infeasible : InvalidArgument, OutOfRange, NotFound. Timeouts : still running.
4.Undo a typo (GA v1.34)
Lower to anything strictly abovestatus.capacity. Quota (max of spec and allocated) drops with it.
No help while ControllerResizeInProgress holds, or once only the node step failed.
Editing a PV's capacity cancels the resize. Raising the PVC to match then fakes a grown backend.
$ kubectl patch pvc data ... 1000Gi
1000Gi ControllerResizeInfeasible 10Gi
$ kubectl patch pvc data ... 100Gi # ok$ ... 10Gi # Forbidden: can not be less
5.VolumeAttributesClass
Edit IOPS of a Bound PVC. GA v1.34, v1beta1 fails on v1.37.
Immutable classes, no default, vac-protection on each.
Cancel = old class back. get pvc : desired, get pv : applied.
A VAC quota checks create and spec. Status can pass hard.
Expansion
VAC modify
Spec field
requests.storage
VAC name
Allowed on
expandable class
MODIFY_VOLUME
Progress
allocatedResource*
modifyVolumeStatus
Controller
ControllerExpand
ControllerModify
Node step
NodeExpandVolume
none
Infeasible
3 codes
INVALID_ARGUMENT
Undo
lower, > capacity
previous class
6.Commands
Command
Shows
kubectl get pvc data -o jsonpath='{.spec.resources.requests.storage}{" alloc="}{.status.allocatedResources.storage}{" state="}{.status.allocatedResourceStatuses.storage}{" cap="}{.status.capacity.storage}{"\n"}'
State in one line. Read it first
kubectl events --for pvc/data
Resize and VolumeModify events
kubectl get pvc data -o jsonpath='{.status.currentVolumeAttributesClassName}{" "}{.status.modifyVolumeStatus}{"\n"}'
Applied class, change in flight
kubectl -n <ns> get resourcequota <q> -o jsonpath='{.status.used}{" / "}{.status.hard}{"\n"}'
VAC quota over hard
In short : external-resizer runs both changes. Infeasible = driver refused : change the spec back.
1.Two hidden walks
A mount can walk the whole volume twice : kubelet fsGroup chown, runtime SELinux relabel. Both grow with file count.
VOLUME_MOUNT_GROUP driver : gets volume_mount_group, no walk, both policy fields ignored.
2.What the fsGroup walk does
Lchown to the GID, OR in 0660 (0440 ro), setgid on dirs. Never removes bits.
OnRootMismatch skips if the root has the GID, 0770 (0550 ro) and setgid.
30s : VolumePermissionChangeInProgress, then every 60s (CSI only).
RWX skipped by default : NFS needs File.
Lever
Where
Effect
OnRootMismatch
Pod
skip if root matches
Always, or unset
Pod
walk every mount
fsGroupPolicy: None
CSIDriver
never apply
fsGroupPolicy: File
CSIDriver
always, even RWX
VOLUME_MOUNT_GROUP
driver
driver does it
readOnly: true
Pod
skipped
3.SELinux : a mount option
PV, seLinuxOptions.level, driver seLinuxMount: true : mount -o context. Else (secret, configMap ...) a full relabel.
RWOP and change-policy gates locked v1.36. SELinuxMount GA v1.37, unlocked : =false in v1.37 only.
MountOption
Recursive
Who labels
kubelet, -o context
runtime walk
Cost
constant
grows with files
One node
same label only
any labels
Needs
label, opt-in
nothing
4.Label conflicts from v1.37
Two labels (or privileged + not) on one volume and node : the 2nd Pod hangs. Fix : Recursive or one label.
Audit first : selinux-warning-controller (off, Controllers). Its metric names Pods : admin-only. Kubelet : volume_manager_selinux_*.
5.Access modes, enforced where
See Volumes, Topology. RWO is one node at attach unless the PV lists RWX, ROX or nothing.
recursiveReadOnly: IfPossible may fall back silently. Use Enabled.
kubectl get events -A --field-selector reason=SELinuxLabelConflict
Potential conflicts (Normal)
kubectl get --raw /metrics | grep kubernetes_feature_enabled
Gate states (SELinuxMount 1)
kubectl get pod <p> -o jsonpath='{range .status.containerStatuses[*].volumeMounts[*]}{.name}{" rro="}{.recursiveReadOnly}{"\n"}{end}'
Recursive read-only took effect ?
In short : OnRootMismatch, fsGroupPolicy and -o context skip the walks. v1.37 : one label per volume per node.
1.Attach and detach, the normal path
A second node is refused unless the PV lists RWX or ROX : Waiting for detach (was Multi-Attach error before v1.37.0, v1.36.4, v1.35.9). Reason is still FailedAttachVolume. In a rollout it is usually a race.
The 6m clock (hard-coded) starts once the Pod is gone or terminated. Call order : CSI Drivers.
2.A node dies under RWO
Terminating Pods look running : no clock. Deleting the Node fails too.
Node off : out-of-serviceNoExecute taint (StatefulSets). Instant detach, no health check : never on a rebooting node. Untaint after.
Or force-delete the Pod : detach after 6m unless --disable-force-detach-on-timeout.
FailedAttachVolume = attach side. FailedMount (FailedMapVolume for Block) = node side. A plain timeout posts no summary event since v1.28 : read the kubelet log. FileSystemResizeFailed is read-only mounts only.
kubectl get volumeattachment -o custom-columns=PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,ATTACHED:.status.attached,DEL:.metadata.deletionTimestamp,FIN:.metadata.finalizers,ERR:.status.detachError.message
Stuck detaches : fix the driver, keep the finalizer
kubectl get node <n> -o jsonpath='{.status.volumesInUse}{"\n"}{.status.volumesAttached}{"\n"}'