Supply chain security
Images, Registries and Pulls
1.A reference, part by part
- Shape :
[host[:port]/]path[:tag][@digest]. Tag and digest both given : only the digest is pulled. - First part is a host only with a
.or:, aslocalhost, or in uppercase. Elsedocker.io(+library/for one word). - The kubelet only adds
:latest. containerd maps short names to Docker Hub. CRI-O uses its search list and can refuse an ambiguous name.
| In the Pod | containerd pulls | Default policy |
|---|---|---|
| busybox | docker.io/ | Always |
| myregistry/ | docker.io/. or :, so Docker Hub) | IfNotPresent |
| registry:5000/ | registry:5000/: makes it a host) | IfNotPresent |
| nginx | docker.io/ | IfNotPresent |
| nginx:latest | docker.io/ | Always |
2.Behind one tag
- A multi-arch tag points at an index : one manifest per platform.
- The runtime picks the node's platform. None fits : "no match for platform in manifest".
- A manifest names one config and its layers, all by digest.
- Clients re-hash what they fetch and reject a mismatch ("unexpected commit digest").
- The config digest is the ImageID (
crictlIMAGE ID).
3.Things that point at an image
- OCI 1.1 : an artifact names the image digest in
subject, its kind inartifactTypeor the config type. GET /lists them. A 404 : fall back to the tagv2/<name>/ referrers/<digest> sha256-<hex>.- The kubelet and the CRI never read referrers. Checks live in admission, tools or runtime policy : Image Signing, Attestations, Image Policy.
4.imagePullPolicy
- Defaulted once, at create : Always when the parsed tag is
latest(no tag,:latest, even:latest@sha256:, not:Latest), else IfNotPresent. - Fixed on a Pod. Moving a template to
:latestlater keeps the old default. - Always : a registry round trip per container start, not a re-download. Needs the registry up.
- IfNotPresent keeps the old bytes of a moved tag.
5.What actually ran : imageID
imageIDcopies the CRIimage_ref(Architecture). On containerd : the index digest.- It can name another repo with the same bytes. Seen :
docker.io/ rancher/ mirrored-pause@sha256:f548e0e8... - Compare only the digest part, with
crane digestwithout--platform.
| Command | Shows |
|---|---|
kubectl get pod web -o jsonpath='{range . | The digest that ran, per container |
crane digest <image> / crane digest --platform linux/amd64 <image> | Index digest / one platform's manifest |
crane manifest <image> | jq -r '.mediaType, (.manifests | length)' | Index or not, and how many platforms |
oras discover <ref>@sha256:<d> | Referrers tree. Finds nothing for legacy .sig tags |
In short : A tag is a question and a digest is the answer. Pin digests, remember that pull policy is fixed when the Pod is created, and read
imageID to see what really ran.1.One container start, step by step
- Always skips the presence check. IfNotPresent and Never ask CRI
ImageStatus. - Never and no image :
ErrImageNeverPull. Present : pull records decide if this Pod may use it. - Else : credentials, per-Pod back-off, a
Pullingevent, the queue. - The
Pulledevent shows two times : "in 209ms (209ms including waiting)". The gap is time in the queue.
2.Queue, rate and back-off
- Default : 1 worker, queue of 10, 1 pull at a time.
maxParallelImagePulls>= 2 (stable v1.35) flips an unsetserializeImagePullsto false. An explicittruestops the kubelet.- Keep it at 31 or less before v1.37.0 (panic).
registryPullQPS5, burst 10 : a request rate. Over it :pull QPS exceeded.- Back-off per Pod UID + image, 10s doubling to 300s. A new Pod starts fresh.
3.Where pull credentials come from
- Order : Pod
imagePullSecrets, nodeconfig.json, then plugins. - The ServiceAccount's secrets are copied only at create, only if the Pod lists none.
- Only
dockerconfigjsonanddockercfgSecrets count :kubectl create secret docker-registry. - One
PullImageper matching credential, first success wins. No anonymous retry.
| Method | Scope | Rotation | Static Pods |
|---|---|---|---|
Pod imagePullSecrets | The Pod | Update the Secret | no |
ServiceAccount imagePullSecrets | New Pods of the SA, listing none | Update the Secret | no |
Node config.json | Every Pod on the node | Re-read within 5 minutes | yes |
| Exec credential provider | Pods whose image matches | Plugin's cache duration | yes |
| SA-token credential provider | The Pod's ServiceAccount | Short-lived, no Secret | only without a token |
4.Credential provider plugins
CredentialProviderConfig(kubelet.) :config. k8s. io/ v1 name= binary,matchImages,defaultCacheDuration.- Flags
--image-credential-provider-config(a directory since v1.34) and-bin-dir. - stdin in, stdout out, 1-minute timeout, cached in memory. A bad config stops the kubelet.
5.Pulling with the Pod's own identity
tokenAttributes(beta, on since v1.34) : the plugin trades a Pod-bound token for short-lived credentials.cacheTyperequired since v1.34.ServiceAccount: one SA, shared credentials.- NodeRestriction : grant the kubelet
request-serviceaccounts-token-audience(RBAC). - Required SA annotation missing : skipped, plain 401.
matchImages rules (and config.json keys)
*.k8s.iomatchesx.k8s.io,*.iodoes not : one label per*. A path is a plain prefix :/imagesalso matches/images2.*in a path never matches : the v1.34 blog's*.myregistry.io/*matches nothing. Write*.myregistry.io.
6.Commands
| Command | Shows |
|---|---|
kubectl apply --dry-run=server -f pod.yaml -o jsonpath='{range . | Defaulted pull policy, nothing created |
kubectl get --raw / | The kubelet's effective image settings |
kubectl events --for pod/<p> | Pulling, Pulled (queue time), BackOff |
crictl images --digests / crictl inspecti <image> | Node images : DIGEST (index) vs IMAGE ID (config) |
In short : The kubelet pulls one image at a time unless told otherwise, tries each matching credential in a fixed order, and can pull as the Pod's ServiceAccount with no stored Secret.
1.The cached-image leak
- IfNotPresent or Never : any Pod can run a private image another tenant pulled there (#18787).
- Old fix :
AlwaysPullImages(admission, off by default). Skips subresources :kubectl debugescapes it. - New fix :
KubeletEnsureSecretPulledImages, beta and on since v1.35.
2.Pull records
- Files in
/, keyed by the image name as the Pod wrote it.var/ lib/ kubelet/ image_ manager/ - A Secret hash matches from any namespace. An SA matches by UID.
- Pulled anonymously, by node
config.jsonor a non-token plugin :nodePodsAccessible, open to any Pod. - No record under Never :
ErrImageNeverPull, same text as a missing image.
| Policy | By kubelet | Preloaded |
|---|---|---|
| NeverVerify | no check | no check |
| NeverVerify | checked | no check |
| NeverVerify | if not listed | if not listed |
| AlwaysVerify | checked | checked |
Red = the default. The allowlist ignores how the image arrived.
3.Turning it on safely
- Preloaded : on the node before the gate, from
crictl pull --creds USER:PASS <img>or a node image. Clean private images off first. - Spell it
NeverVerifyAllowlistedImages: the docs'AllowListedstops the kubelet. - Allowlist : no tag, only a trailing
/*, spelled like the Pods (busyboxis notdocker.).io/ library/ busybox - Never delete
image_manager/. Watch..._.mustpull_ checks_ total{result="mustAuthenticate"}
| Stop the leak | Always | Kubelet check | Image source policy |
|---|---|---|---|
| Runs in | admission | kubelet | admission |
| Registry needed on every start | yes | new credentials only | no |
kubectl debug containers | not covered | covered | per policy |
| Beaten by a credential-less mirror | yes | yes (inference) | no |
4.Mirrors and caches
- registry.k8s.io : no uptime SLA, backends move (blobs from
cdn.on 2026-10-07). Mirror what you run.registry. k8s. io - containerd
hosts.toml, no restart. Untrusted mirror :capabilities = ["pull"]. No line = full trust. - CRI-O :
pull-from-mirror = "digest-only", SIGHUP. - Digest pulls are safe via any mirror. A tag trusts its resolver.
- A cache with no auth challenge (Spegel) serves anyone, past both checks above.
5.Image volumes take the same path
volumes[].image(GA v1.36, Volumes) : same pull path, credentials, back-off and verification. Mounted read-only.- A failed volume pull holds back only the containers that mount it. The Pod stays Pending (the docs say Failed).
ImageVolumeWithDigest: alpha, first shipped in v1.36, empty on v1.37.1. Pin the reference by digest instead.
In short : Verification stops one tenant running another's cached private image, mirrors live in the runtime and are safe only for digests, and image volumes share the pull path.