Storage internals

CSI Drivers and the Volume Lifecycle

1.What a CSI driver is

  • A gRPC server on a UNIX socket (CSI_ENDPOINT, ends in .sock).
  • Identity (always), Controller (provision, attach, expand, snapshot), Node (stage, publish, stats). Also GroupController and alpha SnapshotMetadata.
  • Capabilities switch steps off, down to NodePublish and NodeUnpublish only. No capability says "block".
CapabilityWithout it
CREATE_DELETE_VOLUMEprovisioner refuses
PUBLISH_UNPUBLISH_VOLUME"trivial handler"
STAGE_UNSTAGE_VOLUMEkubelet skips NodeStage
VOLUME_MOUNT_GROUPkubelet applies fsGroup
EXPAND, MODIFY, CLONE ...no resize, VAC, copies

Plus plugin CONTROLLER_SERVICE to provision. v1.37 vendors a commit just before spec v1.13.0.

2.Two halves, two callers

  • Controller plugin : Deployment or StatefulSet, leader election. More replicas add availability, not throughput. Sidecars share the socket in an emptyDir.
  • Node plugin : privileged DaemonSet, Bidirectional propagation on /var/lib/kubelet/pods and .../plugins (staging). Privileged Pod Security level.

3.The sidecars

SidecarCallsNewest
external-provisionerCreate / Deletev6.3.0
external-attacherPublish / Unpublishv4.13.0
external-resizerExpand / Modifyv2.3.0
csi-snapshotterCreateSnapshotv8.6.0
external-health-monitorVolumeHealthv0.19.0
node-driver-registrarGetPluginInfo onlyv2.18.0
livenessprobeProbev2.20.0
PVC VolumeAttachment PVC size / VAC SnapshotContent watch : provisionerCreateVolume attacherControllerPublish resizerControllerExpand snapshotterCreateSnapshot csi.sock in a shared emptyDir livenessprobe driver (controller) storage API kubelet registrar-reg.sock GetInfo NodeStageNodePublish csi.sock (hostPath) driver (node) : mounts node DaemonSet Pod

Each controller sidecar is a reconcile loop with leader election. Checked 2026-10-07 : provisioner --timeout is 10s in code (README : 15s), set it for slow backends. Resizer image tops out at v2.2.1. The health monitor exits unless the driver advertises health.

4.Registration on a node

  • Registrar : GetPluginInfo only, then a socket in plugins_registry/. Kubelet (inotify) : GetInfo, NodeGetInfo (2m), then the csi.volume.kubernetes.io/nodeid annotation.
  • --kubelet-registration-path = host path of the driver socket. Wrong : not found in the list of registered....
  • Same watcher as DRA, no maxSurge upgrades (all send "1.0.0").
registrar plugins_registry/ <driver>-reg.sock 2. creates 1. GetPluginInfo kubelet 3. GetInfo 6. Notify status driver (node) 4. NodeGetInfo 5. writes CSINode Node labels nodeid annot. lab : hostpath.csi.k8s.io key topology.hostpath.csi/node

5.Three API objects

  • CSIDriver : 11 fields in v1.37, two immutable (path misspelled spec.attachedRequired). No object is not neutral : attach stays on, no Pod info, tokens or inline.
  • CSINode : by the kubelet, never the driver. nodeID, topologyKeys (keys only), allocatable.count.
  • VolumeAttachment : csi-<sha256(handle+driver+node)>. spec immutable, status only by the attacher.
CSIDriver fieldDefault
attachRequiredtrue, immutable
volumeLifecycleModes[Persistent], immutable
podInfoOnMountfalse
tokenRequestsnone
requiresRepublishfalse
serviceAccountTokenInSecretsunset

The other five : Volume Topology, Volume Operations.

In short : Sidecars turn PVCs, VolumeAttachments and snapshots into calls on the controller half; the kubelet calls the node half itself once the registrar announces its socket.

1.Provision : PVC to PV

  • PV controller : ExternalProvisioning. WaitForFirstConsumer waits for selected-node.
  • external-provisioner : CreateVolume pvc-<UID>, minus reserved keys; writes the PV, finalizer.
  • No secret : provisioning fails, deletion does not.
Reserved keyBecomesRead by
fstypePV fsTypekubelet
provisioner-secret-*Create, Deleteprovisioner
*-publish-, node-stage-*PV *SecretRefattacher, kubelet
*-expand-, *-modify-*refs, annotationresizer, kubelet

Prefix csi.storage.k8s.io/. Secrets : -name / -namespace, templates like ${pvc.namespace}.

2.Attach : the VA handshake

  • The attach-detach controller (ADC) creates VA csi-<sha256>, waits 2m for status.attached.
  • external-attacher : finalizer, node ID from CSINode, ControllerPublishVolume, attachmentMetadata.
  • Immutable. v1.36+ : flipping to false cleans old VAs.
attachRequiredtrue (default)false
VolumeAttachmentper volume per nodenever
driver can publishControllerPublishskipped (wrong)
driver cannottrivial handler, no RPCnothing
attacher missingtimed out after 2mfine
publish_contextVA attachmentMetadatanil

3.The whole lifecycle

control plane driver PV controller external-provisioner writes PV {driver, volumeHandle} attach-detach controller 2. VA csi-<sha256> external-attacher writes status.attached; ADC : volumesAttached controller CreateVolume, Publish 1. Create Publish driver node volumesAttached : ADC volumesInUse : kubelet container runtime 5. bind mount kubelet node Stage, Publish 3. NodeStage 4. NodePublish stage once per node, publish per Pod teardown : NodeUnpublish, NodeUnstage, volumesInUse cleared, VA deleted, ControllerUnpublish, DeleteVolume

4.Stage and publish on the node

  • MountDevice : globalmount + vol_data.json, NodeStage. No staging : directory only.
  • SetUp : Pod directory + vol_data.json, driver makes mount, then fsGroup.
  • Canceled, DeadlineExceeded, Unavailable, ResourceExhausted, Aborted : uncertain, kept. Others : removed.
Path (/var/lib/kubelet/...)Written by
plugins_registry/<driver>-reg.sockregistrar
plugins/<driver>/csi.sockdriver
plugins/kubernetes.io/csi/<driver>/<sha256(handle)>/globalmountkubelet
pods/<podUID>/volumes/kubernetes.io~csi/<pv>/mountkubelet, driver

Reconcile every 100ms. Only the first access mode is sent (RWO as 7 if advertised). Before v1.24 : .../csi/pv/<pv>/globalmount.

5.Teardown and kubelet restart

  • Unpublish, Unstage (last Pod on the node), volumesInUse cleared (0 to 10s), VA deleted, ControllerUnpublish (observed : 34.327, 34.428, 35.118). Why : Volume Operations.
  • DeleteVolume is retried : idempotent. A VA left : VolumeFailedDelete ... still attached. Reconstruction (GA v1.30) : a restarted kubelet rebuilds from vol_data.json and re-issues Stage and Publish.

6.Commands

CommandShows
VA=csi-$(printf '%s%s%s' "$HANDLE" "$DRIVER" "$NODE" | sha256sum | cut -d' ' -f1)VA name (a hash)
printf '%s' "$HANDLE" | sha256sumStaging dir (a hash)
kubectl get volumeattachment <va> -o jsonpath='{.status.attachError}{"\n"}{.metadata.finalizers}'Error, finalizers
findmnt -o TARGET,SOURCE,FSTYPE,PROPAGATION | grep kubernetes.ioReal mounts (hostpath : no globalmount)
journalctl -u kubelet | grep -E 'MountVolume.(WaitForAttach|MountDevice|SetUp)|UnmountDevice'Node steps
In short : Provision makes the PV, attach is a VolumeAttachment the attacher answers, stage mounts once per node, publish once per Pod, and teardown runs backwards, gated by volumesInUse.

1.Volumes that skip steps

Raw block : volumeMode: Block, devicePath, no fsType; paths under .../csi/volumeDevices/, held by a loop device.

CSI persistentCSI inlineGeneric ephemeralImage volumeIn-tree, migrated
API fieldPV spec.csivolumes[].csivolumes[].ephemeralvolumes[].imageold PV fields
ParametersadminPod authoradminPod authoradmin
CSI callsallPublish onlyallnoneall, translated
VAif attachRequiredneverif attachRequiredneverif needed
Capacity-awareyesnoyesn/ayes

2.Inline, generic and image volumes

  • Inline : CSIDriver lists Ephemeral (Volumes). volume_id = csi-<sha256(podUID+name)>. podInfoOnMount : five keys (pod name, namespace, uid, serviceAccount.name, ephemeral) to NodePublish only.
  • Pod authors set volumeAttributes; restricted Pod Security allows csi. Guard with admission.
  • Generic : PVC <pod>-<volume>, no PVC create right needed. Names collide : pod is not owner.
csi kubelet NodePublish only driver makes, deletes ephemeral ephemeral ctrl PVC <pod>-<vol> owner Pod, full cycle image kubelet, runtime read-only, no CSI PullImage via CRI

3.Credentials into drivers

  • tokenRequests : Pod-bound token per audience, 600s minimum. Refresh : RBAC; vs projected.
  • Node audience restriction (beta, on) : only audiences used on that node.
  • requiresRepublish : republish on pod syncs (1m to 1.5m, labs 17s to 80s), not 0.1s. Contents only.
Tokens involume_contextsecrets
Whendefaultfield true
Logsprintedstripped
API warningyes, if unsetno
Real leaksCVE-2023-2878, CVE-2024-3744

serviceAccountTokenInSecrets (GA v1.36) without tokenRequests, even false : rejected. Real drivers : SPIFFE CSI, Secrets Store CSI (ESO).

4.Migration and what comes next

  • Ten in-tree plugins were removed (1.26 to 1.36). Seven types need their CSI driver (Volumes). CephFS, RBD, GlusterFS have no translator : move by hand first. FlexVolume : deprecated, still there.
  • Horizon : all-in-one sidecars (KEP-4958, alpha target v1.38); COSI v1alpha2 only on main (tag v0.2.2).

5.Debugging and commands

Attach and mount symptoms : Volume Operations. -v=5 driver logs are sanitized (not hostpath's gRPCCall).

SymptomLook atCause and fix
PVC Pending, only ExternalProvisioningprovisioner, leaderprovisioner down, or another driver named
does not support snapshot restoredataSourceno CREATE_DELETE_SNAPSHOT (clones: CLONE_VOLUME), no RPC
timed out waiting for external-attacherVAs, attacher podsdeploy one, or recreate with attachRequired false
not found in the list of registeredregistrar logsno node plugin, wrong path
VA CSINode ... does not contain driverkubectl get csinodenot registered on that node
volume mode "Ephemeral" not supportedvolumeLifecycleModesinline volume, Persistent-only driver
CommandShows
kubectl logs <controller-pod> -c csi-attacherTrivial sync started
kubectl logs <node-pod> -c node-driver-registrarGetInfo, NotifyRegistrationStatus
ls -l /var/lib/kubelet/plugins_registry/ /var/lib/kubelet/plugins/Both sockets
find /var/lib/kubelet -name vol_data.json -exec sh -c 'echo {}; cat {}' \;Kubelet bookkeeping
kubectl get --raw /api/v1/nodes/<n>/proxy/metrics | grep -E '^(csi_operations|reconstruct_volume)'Kubelet CSI metrics
kubectl delete csidriver <d> && kubectl apply -f csidriver.yamlOnly way to change immutables
In short : Variants skip or swap lifecycle steps, tokens belong in the secrets field, and a stuck volume's message names the part to read : provisioner, attacher or registrar.