Supply chain security

Images, Registries and Pulls

1.A reference, part by part

  • Shape : [host[:port]/]path[:tag][@digest]. Tag and digest both given : only the digest is pulled.
  • First part is a host only with a . or :, as localhost, or in uppercase. Else docker.io (+ library/ for one word).
  • The kubelet only adds :latest. containerd maps short names to Docker Hub. CRI-O uses its search list and can refuse an ambiguous name.
In the Podcontainerd pullsDefault policy
busyboxdocker.io/library/busybox:latestAlways
myregistry/app:1docker.io/myregistry/app:1 (no . or :, so Docker Hub)IfNotPresent
registry:5000/app:1registry:5000/app:1 (: makes it a host)IfNotPresent
nginx@sha256:<d>docker.io/library/nginx@sha256:<d>IfNotPresent
nginx:latest@sha256:<d>docker.io/library/nginx@sha256:<d> (tag dropped)Always

2.Behind one tag

  • A multi-arch tag points at an index : one manifest per platform.
  • The runtime picks the node's platform. None fits : "no match for platform in manifest".
  • A manifest names one config and its layers, all by digest.
  • Clients re-hash what they fetch and reject a mismatch ("unexpected commit digest").
  • The config digest is the ImageID (crictl IMAGE ID).
tag 3.10.2 pause registry resolves index sha256:f548e0e8... runtime picks platform linux/arm64 a4332146... +6 more linux/amd64 manifest sha256:412c4a72... fetch by digest config 4a83b15d... = ImageID layer 81ede362... gzip, as stored
signature SBOM provenance subject image manifest sha256:A client GET /referrers/sha256:A index of 3 pause:3.10.2 on 2026-10-07 : empty (legacy .sig tags)

3.Things that point at an image

  • OCI 1.1 : an artifact names the image digest in subject, its kind in artifactType or the config type.
  • GET /v2/<name>/referrers/<digest> lists them. A 404 : fall back to the tag sha256-<hex>.
  • The kubelet and the CRI never read referrers. Checks live in admission, tools or runtime policy : Image Signing, Attestations, Image Policy.

4.imagePullPolicy

  • Defaulted once, at create : Always when the parsed tag is latest (no tag, :latest, even :latest@sha256:, not :Latest), else IfNotPresent.
  • Fixed on a Pod. Moving a template to :latest later keeps the old default.
  • Always : a registry round trip per container start, not a re-download. Needs the registry up.
  • IfNotPresent keeps the old bytes of a moved tag.

5.What actually ran : imageID

  • imageID copies the CRI image_ref (Architecture). On containerd : the index digest.
  • It can name another repo with the same bytes. Seen : docker.io/rancher/mirrored-pause@sha256:f548e0e8...
  • Compare only the digest part, with crane digest without --platform.
CommandShows
kubectl get pod web -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.imageID}{"\n"}{end}'The digest that ran, per container
crane digest <image> / crane digest --platform linux/amd64 <image>Index digest / one platform's manifest
crane manifest <image> | jq -r '.mediaType, (.manifests | length)'Index or not, and how many platforms
oras discover <ref>@sha256:<d>Referrers tree. Finds nothing for legacy .sig tags
In short : A tag is a question and a digest is the answer. Pin digests, remember that pull policy is fixed when the Pod is created, and read imageID to see what really ran.

1.One container start, step by step

  • Always skips the presence check. IfNotPresent and Never ask CRI ImageStatus.
  • Never and no image : ErrImageNeverPull. Present : pull records decide if this Pod may use it.
  • Else : credentials, per-Pod back-off, a Pulling event, the queue.
  • The Pulled event shows two times : "in 209ms (209ms including waiting)". The gap is time in the queue.
Pod spec policy set at create kubelet image manager default tag present ? pull records back-off credentials PullImage (one per credential) CRI ImageService (containerd, CRI-O) normalize mirrors index : platform fetch, re-hash image_ref pulled record status.imageID

2.Queue, rate and back-off

  • Default : 1 worker, queue of 10, 1 pull at a time.
  • maxParallelImagePulls >= 2 (stable v1.35) flips an unset serializeImagePulls to false. An explicit true stops the kubelet.
  • Keep it at 31 or less before v1.37.0 (panic).
  • registryPullQPS 5, burst 10 : a request rate. Over it : pull QPS exceeded.
  • Back-off per Pod UID + image, 10s doubling to 300s. A new Pod starts fresh.

3.Where pull credentials come from

  • Order : Pod imagePullSecrets, node config.json, then plugins.
  • The ServiceAccount's secrets are copied only at create, only if the Pod lists none.
  • Only dockerconfigjson and dockercfg Secrets count : kubectl create secret docker-registry.
  • One PullImage per matching credential, first success wins. No anonymous retry.
MethodScopeRotationStatic Pods
Pod imagePullSecretsThe PodUpdate the Secretno
ServiceAccount imagePullSecretsNew Pods of the SA, listing noneUpdate the Secretno
Node config.jsonEvery Pod on the nodeRe-read within 5 minutesyes
Exec credential providerPods whose image matchesPlugin's cache durationyes
SA-token credential providerThe Pod's ServiceAccountShort-lived, no Secretonly without a token

4.Credential provider plugins

  • CredentialProviderConfig (kubelet.config.k8s.io/v1) : name = binary, matchImages, defaultCacheDuration.
  • Flags --image-credential-provider-config (a directory since v1.34) and -bin-dir.
  • stdin in, stdout out, 1-minute timeout, cached in memory. A bad config stops the kubelet.

5.Pulling with the Pod's own identity

  • tokenAttributes (beta, on since v1.34) : the plugin trades a Pod-bound token for short-lived credentials.
  • cacheType required since v1.34. ServiceAccount : one SA, shared credentials.
  • NodeRestriction : grant the kubelet request-serviceaccounts-token-audience (RBAC).
  • Required SA annotation missing : skipped, plain 401.
matchImages rules (and config.json keys)
  • *.k8s.io matches x.k8s.io, *.io does not : one label per *. A path is a plain prefix : /images also matches /images2.
  • * in a path never matches : the v1.34 blog's *.myregistry.io/* matches nothing. Write *.myregistry.io.

6.Commands

CommandShows
kubectl apply --dry-run=server -f pod.yaml -o jsonpath='{range .spec.containers[*]}{.image}{"\t"}{.imagePullPolicy}{"\n"}{end}'Defaulted pull policy, nothing created
kubectl get --raw /api/v1/nodes/<node>/proxy/configz | jq '.kubeletconfig | {maxParallelImagePulls, imagePullCredentialsVerificationPolicy}'The kubelet's effective image settings
kubectl events --for pod/<p>Pulling, Pulled (queue time), BackOff
crictl images --digests / crictl inspecti <image>Node images : DIGEST (index) vs IMAGE ID (config)
In short : The kubelet pulls one image at a time unless told otherwise, tries each matching credential in a fixed order, and can pull as the Pod's ServiceAccount with no stored Secret.

1.The cached-image leak

  • IfNotPresent or Never : any Pod can run a private image another tenant pulled there (#18787).
  • Old fix : AlwaysPullImages (admission, off by default). Skips subresources : kubectl debug escapes it.
  • New fix : KubeletEnsureSecretPulledImages, beta and on since v1.35.
Pod A (team-a) has a pull Secret Pod B (team-b) no Secret node cache private image pulls present : runs ! with verification (gate on) : kubelet checks pull record B pulls with its own creds 401

2.Pull records

  • Files in /var/lib/kubelet/image_manager/, keyed by the image name as the Pod wrote it.
  • A Secret hash matches from any namespace. An SA matches by UID.
  • Pulled anonymously, by node config.json or a non-token plugin : nodePodsAccessible, open to any Pod.
  • No record under Never : ErrImageNeverPull, same text as a missing image.
PolicyBy kubeletPreloaded
NeverVerifyno checkno check
NeverVerifyPreloadedImagescheckedno check
NeverVerifyAllowlistedImagesif not listedif not listed
AlwaysVerifycheckedchecked

Red = the default. The allowlist ignores how the image arrived.

3.Turning it on safely

  • Preloaded : on the node before the gate, from crictl pull --creds USER:PASS <img> or a node image. Clean private images off first.
  • Spell it NeverVerifyAllowlistedImages : the docs' AllowListed stops the kubelet.
  • Allowlist : no tag, only a trailing /*, spelled like the Pods (busybox is not docker.io/library/busybox).
  • Never delete image_manager/. Watch ..._mustpull_checks_total{result="mustAuthenticate"}.
Stop the leakAlwaysPullImagesKubelet checkImage source policy
Runs inadmissionkubeletadmission
Registry needed on every startyesnew credentials onlyno
kubectl debug containersnot coveredcoveredper policy
Beaten by a credential-less mirroryesyes (inference)no
registry.k8s.io/pause:3.10.2 certs.d/registry.k8s.io/hosts.toml ?ns=registry.k8s.io mirror 1 pull, resolve mirror 2 pull only server upstream, last timeout : "trying next host"

4.Mirrors and caches

  • registry.k8s.io : no uptime SLA, backends move (blobs from cdn.registry.k8s.io on 2026-10-07). Mirror what you run.
  • containerd hosts.toml, no restart. Untrusted mirror : capabilities = ["pull"]. No line = full trust.
  • CRI-O : pull-from-mirror = "digest-only", SIGHUP.
  • Digest pulls are safe via any mirror. A tag trusts its resolver.
  • A cache with no auth challenge (Spegel) serves anyone, past both checks above.

5.Image volumes take the same path

  • volumes[].image (GA v1.36, Volumes) : same pull path, credentials, back-off and verification. Mounted read-only.
  • A failed volume pull holds back only the containers that mount it. The Pod stays Pending (the docs say Failed).
  • ImageVolumeWithDigest : alpha, first shipped in v1.36, empty on v1.37.1. Pin the reference by digest instead.
In short : Verification stops one tenant running another's cached private image, mirrors live in the runtime and are safe only for digests, and image volumes share the pull path.