Storage internals

Snapshots, Clones and Backup

1.Not core : CRDs plus two controllers

  • snapshot.storage.k8s.io/v1 CRDs (GA v1.20), CSI only : VolumeSnapshot (vs, namespaced), VolumeSnapshotContent (vsc), VolumeSnapshotClass (vsclass).
  • The distro installs them and one snapshot-controller; each driver runs a csi-snapshotter (v8.6.0, CSI Drivers). CEL rules in the CRDs replaced the webhook (gone in v8.2.0); VolumeSnapshotClass has none, so edits pass.
kubectl or Velero create VolumeSnapshot new PVC dataSourceRef -> snapshot kube-apiserver CEL rules in the CRDs watch external-provisioner checks binding, driver, ready, size, volume mode watch snapshot-controller one per cluster : default class, finalizers, binding CreateVolume CSI driver creates CreateSnapshot VolumeSnapshotContent snapcontent-<VS uid> csi-snapshotter watches its driver's contents handle, ready, size into content status

2.One dynamic snapshot

  • No class : the driver's one class with is-default-class: "true". Zero or two fail (unlike StorageClass).
  • Driver returns at the cut; readyToUse may lag hours.
FieldVolumeSnapshotContent
linkboundVolumeSnapshot­ContentNamevolumeSnapshotRef
handlenonesnapshotHandle
creationTimedate-timeint64 ns
restoreSizequantityint64 bytes

3.Pre-provisioned, deletionPolicy

  • Import : a content with snapshotHandle, deletionPolicy, volumeSnapshotRef, ideally sourceVolumeMode.
  • Trust it if both point at each other. Delete the VS, not the content (finalizers).
WhenDeleteRetain
VS deletedstorage snapshot goesboth stay
namespace gonestorage snapshot goescontent orphaned
importedset by handset by hand
Veleroonce backup expireswhile backup exists

4.Restore : always a new PVC

  • dataSourceRef names the VolumeSnapshot. No revert in place.
  • Mode guard (GA v1.30) : Block <-> Filesystem needs allow-volume-mode-change: "true" on the content, so content write access is powerful (RBAC).
  • Errors : ProvisioningFailed on the PVC.
Check, in orderError says
CREATE_DELETE_SNAPSHOTcapability is not reported
snapshot not deletingis being deleted
bound, content points backbound to a different snapshot
content driver = provisionernot handled by CSI driver of ...
readyToUse and a handleis not Ready
size >= restoreSizeis less than the size
volume mode guarddoes not have permission

5.Clone a PVC

  • Source PVC in dataSource : Bound, storageClassName on both, size >= source, same driver (any class, v3.2.0+), same volumeMode, CLONE_VOLUME.
  • Same namespace (cross-namespace is alpha). Errors say not bound or invalid : see the provisioner log.

6.Commands

CommandShows
kubectl get vs -n <ns>RESTORESIZE 1Gi
kubectl get vscin bytes
kubectl get vsclass -o custom-columns='NAME:.metadata.name,DEFAULT:.metadata.annotations.snapshot\.storage\.kubernetes\.io/is-default-class'Two defaults ?
kubectl patch vsc <c> --type=merge -p '{"spec":{"deletionPolicy": "Retain"}}'Keep the data
In short : A snapshot is a namespaced request plus a cluster-scoped content holding the storage handle, and they must point at each other. Restores and clones always make a new PVC.

1.dataSource vs dataSourceRef

  • A custom kind in dataSource is dropped on create, no warning : an empty volume. Use dataSourceRef.
  • AnyVolumeDataSource : GA v1.33, gate removed in v1.37. Passing it stops the component.
dataSourcedataSourceRef
kindsPVC, snapshotany non-core kind, PVC
other kinddropped silentlyrejected
synccopied to the Refcopied back if no namespace
both setmust match; both immutable
namespacemust stay emptyalpha gate only

2.Volume populators

  • Unknown kind : the provisioner assumes a populator, stops.
  • lib-volume-populator v3.4.0 : waits for selected-node (WFFC), fills prime-<uid>, moves the PV over.
  • A VolumePopulator (populator.storage.k8s.io/v1beta1) registers each kind. Populators rebind PVs : control-plane code (Operators).
PVC, Pending dataSourceRef -> Hello + target-protection provisioner : assumes a populator, stops populator namespace PVC' prime-<uid> PV-x binds Pod populate-<uid> writes claimRef patched PVC' Lost, Pod and PVC' deleted PVC Bound to PV-x finalizer removed

3.Across namespaces (alpha, stalled)

  • CrossNamespaceVolumeDataSource : alpha, off since v1.26, KEP-3294 closed "not planned". Gate on kube-apiserver, controller-manager and csi-provisioner, plus the ReferenceGrant CRD and RBAC.
  • The grant sits in the source namespace; an empty to.name opens every object of that kind. Gate off : a namespaced ref is dropped silently.

4.Group snapshots : one cut

  • GA v1.36. Snapshotter v8.6.0 serves v1, stores v1beta2, gate Beta and off. Driver needs GroupController.
  • Label the PVCs. Members are owned by the group : none deleted alone, one driver.
  • Use v1 : conversion is None, so v1beta1 readers lose members and its writes skip CEL (CRDs).
SingleGroup
storedv1v1beta2 (serves v1)
turn onCRDs, controller+ CSIVolumeGroupSnapshot gate
picksone PVC namelabel selector, resolved once
consistencyone volumecrash-consistent across all
restorePVC from itone PVC per member
CSI callCreateSnapshotCreateVolumeGroupSnapshot

5.Changed block tracking

  • KEP-3314, beta v1.36, out of tree : the driver advertises SNAPSHOT_METADATA_SERVICE and runs external-snapshot-metadata v1.1.0. Its SnapshotMetadataService gives address, audience, CA.
  • GetMetadataDelta (base = an older snapshotHandle) returns block ranges, not data. FAILED_PRECONDITION : full backup. Velero 1.18 skips it.
backup app audience-scoped token 5. restore to a Block PVC, read those ranges kube-apiserver metadata sidecar TLS, per driver CSI driver 1. TokenRequest 2. gRPC + token block ranges 3. TokenReview, SAR 4. GetMetadataDelta

6.Commands

CommandShows
kubectl get volumepopulatorsRegistered populator kinds
kubectl get crd volumegroupsnapshots.groupsnapshot.storage.k8s.io -o jsonpath='{range .spec.versions[*]}{.name} served={.served} storage={.storage}{"\n"}{end}'v1 served, v1beta2 stored
kubectl get vs -n <ns> -o jsonpath='{range .items[?(@.metadata.ownerReferences[0].name=="<vgs>")]}{.metadata.name}{"\n"}{end}'Members of a group
kubectl get smsCBT drivers (cbt.storage.k8s.io/v1beta1)

Group snapshots ignored ? In v8.6.0 support is a flag on both snapshot components : check their args before the driver.

In short : Populators fill a PVC from any kind, group snapshots cut many volumes at one crash-consistent point, and changed block tracking makes backups incremental.

1.Why a snapshot is not a backup

  • It sits on the same storage as the volume and goes with its namespace under Delete. Record handles.
FailureCSI snapshotVelero, no data moveVelero data mover, fs-backup
namespace deletedgone too under Deletekept (Retain), YAML in bucketdata in the bucket
storage pool lostlost if kept beside the volumelost, only the YAML survivessurvives if the bucket is elsewhere
whole cluster lostre-import by snapshotHandlere-imported, same driver nameany cluster with CSI and the bucket
ransomware, bad writeonly if nobody deleted itsameseparate bucket; guard the key

2.Consistency and hooks

  • One snapshotcrash-consistent
  • Groupcrash-consistent, all volumes
  • + fsfreeze hooksapp-consistent (frozen FS)
  • fs-backupnot point in time

KEP-3476 : quiesce weekly, group snapshot nightly.

3.Velero v1.18 : three ways

  • v1.18.4 (now velero-io/velero) needs --features=EnableCSI, though CSI is built in since v1.14. Hooks : no shell; first container, Fail, 30s.
  • Class pick : PVC annotation, Backup _<driver> annotation, class label velero.io/csi-volumesnapshot-class, the default, the only class.
CSI snapshotCSI snapshot data movementfs-backup (kopia)
turn on--features=EnableCSI+ --use-node-agent, --snapshot-move-data--use-node-agent + pod annotation
data goes tostays on the storagekopia repo in the BackupStorageLocationkopia repo
works forCSI volumesCSI volumesmounted volumes, not hostPath
runs asVelero serverdata-mover Pods, root, BestEffortnode-agent, kubelet hostPath
one volumesnapshot or fs-backup, never both. Restic restores stop working in Velero 1.19

4.One backup with data movement

  • csiSnapshotTimeout (10m) waits for the handle (the cut); itemOperationTimeout (4h) cancels uploads.
  • All repos share one static key : set repository-password first. Groups : velero.io/volume-group label (1.18.x talks v1beta2).
velero backup --snapshot-move-data Velero server VS per PVC, hooks snapshot-controller + csi-snapshotter DataUpload one per snapshot handle (cut) node-agent PVC from snapshot data-mover Pod kopia uploader bucket (BSL) .../kopia/<ns> snapshot deleted after the upload

5.Restore in order, and its pitfalls

  • Storage before workloads, like Flux.
  • A snapshot returns as an imported content (Retain) plus a VS; the PVC loses selected-node, so WFFC picks again. Same driver name.
  • Map classes with resource modifiers. ignoreDelayBinding risks the wrong zone (Topology).
1. CRDs, then refresh discovery 2. namespaces, StorageClasses 3. VS classes, contents, VS, DataUploads 4. PVs, PVCs 5. RBAC, SAs, Secrets, ConfigMaps 6. Pods, ReplicaSets, Services 7. the rest, A to Z (Deployments ...) low priority : Cluster API, Carvel

6.Commands

CommandShows
velero install --features=EnableCSI --use-node-agent --plugins=<object-store-plugin> ...Velero with CSI and the node-agent
velero backup create NAME --snapshot-move-dataSnapshot, then copy the data off
kubectl -n velero get datauploads -o wide -l velero.io/backup-name=NAME -wWatch these, not the backup phase
velero restore create --from-backup NAME --resource-modifier-configmap <cm>Restore with patches (new class). Postgres : CNPG
In short : Snapshots guard against bad writes, not against losing the storage, namespace or cluster. Velero copies data to object storage and restores storage before workloads.