CI and builds
Building Images in the Cluster
1.A RUN step is a container
- A builder unpacks the base image. Each
RUNruns in a new container over the layer stack and is saved as a layer. Then the image is pushed. - For that it calls
unshareorclonewithCLONE_,NEWUSER NEWNSandNEWPID, mounts overlay layers and a fresh/proc, and writes files as many UIDs. - No Docker socket on containerd nodes (Images and Pulls).
Two traps
- No seccomp profile means
Unconfined, unless the kubelet setsseccompDefault: true(upstream defaultfalse). A builder can break the day a node flips it. - In dind, containers get Docker's default profile, so Buildah fails with
unshare(CLONE_: not permitted, unless the job container is privileged or unconfined.NEWUSER)
2.The four fences
3.The options
| Option | Pod needs | Lowest level |
|---|---|---|
| Docker-in-Docker | privileged: true | Privileged |
| Rootless dind | still privileged: true | Privileged |
| BuildKit rootless | Unconfined profiles, UID 1000 | Privileged (explicit Unconfined) |
| BuildKit userns | hostUsers: false plus privileged: true | Privileged |
| Buildah, userns Pod | hostUsers: false, UID 1000, no seccomp field | Baseline, while seccomp is effectively Unconfined |
| kaniko forks | root, no extra capabilities | Baseline |
Nothing that runs RUN steps with real namespaces is shown to fit Restricted. Levels : Pod Security.
4.Docker-in-Docker
- The official image says
--privilegedis required, and the rootless variant needs it too : "rootless" is the daemon's UID, not the Pod's power. - A privileged Pod is node root. Any workflow author on that runner can own the node, and the namespace must allow
privileged. - ARC's
dindmode and Gitea's dind sidecar are this : ARC, Gitea runners. Best as a native sidecar. - TLS off (
DOCKER_) and arguments empty or starting with a flag : the stock entrypoint listens onTLS_ CERTDIR="" tcp://0.0.0.0:2375, no auth. Keep TLS, or start the arguments with the worddockerd.
5.kaniko, archived
- Google archived it in June 2025 (last release v1.24.0). Forks : osscontainertools publishes images, Chainguard source tags only.
- It is the exception : commands run in its own root and are snapshotted in userspace, so it needs root but no new namespaces and fits Baseline. Its README says it does not make untrusted builds safe, and it must run in its own image.
- No multi-arch index of its own : use
manifest-tool.
Seen in practicePitfall : a dind container beside the job with no startup probe, so the first
docker call fails and every workflow grows its own until docker info loop. Do better : run dind as a native sidecar with a docker info startup probe, as ARC's dind mode does, so the job starts only once the daemon answers.In short : A build needs namespaces and mounts. dind and BuildKit get them by being privileged or unconfined, Buildah can get them from a user-namespace Pod, kaniko fakes them in userspace. Pick the smallest hole you can live with.
1.Three shapes, and who gives you which
| Runner | Daemon in the job ? Build with |
|---|---|
ARC dind | yes, privileged sidecar : docker build, Buildx |
ARC kubernetes | no : daemonless, remote BuildKit, or a hook template that adds dind |
| Gitea with dind | yes, job containers privileged only with container. : docker build, Buildah in a privileged job |
Gitea kubernetes:// | no : daemonless or remote BuildKit |
2.BuildKit on Kubernetes
- ExamplesUpstream
examples/: Pod, Deployment plus Service with mTLS, StatefulSet (warm cache), Job withkubernetes buildctl-. Each privileged, rootless (recommended) or userns. All usedaemonless.sh mastertags : pin one. - RootlessStill seccomp and AppArmor
Unconfined, UID 1000, and--oci-: leftover step processes are not killed and steps can signal the daemon.worker- no- process- sandbox - Buildx driver
docker buildx create --driver kubernetesmakes the Pods, privileged by default.rootless=trueswaps in Unconfined profiles. Apersistent-option gives a StatefulSet with a PVC (v0.34.0+).volume- claim - Trap
requests.sets a limit, not a request : scratch use above it evicts the builder.ephemeral- storage - Client
buildctl --addr kube-pod://<pod>needspods/exec(RBAC).
3.Buildah in a user-namespace Pod
spec: # abridged
hostUsers: false
containers:
- name: buildah
image: quay.io/buildah/stable:v1.43.4-immutable
securityContext: {runAsUser: 1000} # else root
volumes: # storage.conf points overlay here
- {name: storage, emptyDir: {sizeLimit: 10Gi}}
- No
privileged, capabilities,/dev/fuseor seccomp field : Baseline on the field rules, but it works only while the node runs PodsUnconfined. - Rootless BuildKit cannot do this (rootlesskit needs its own mappings) : BuildKit in a user namespace runs the rootful image with
privileged: true.
4.User namespaces and Pod Security
| Rule | With hostUsers: false |
|---|---|
Restricted runAsNonRoot, runAsUser; Baseline procMount | skipped; Unmasked allowed only with it |
| Policy version | only v1.35 and later : a namespace pinned to enforce- gets none |
| Still checked | privileged, capabilities, seccomp, AppArmor, host namespaces |
privileged: true | allowed, still the Privileged level |
The field, node needs and UID mapping : Pod Security.
The Buildah Pod is one published setup (CERN, 2025), not upstream advice.
5.Storage and scratch
vfsworks anywhere but copies every layer : slow and disk-hungry. Overlay on an overlay root fails (a mount_) : use an emptyDir or PVC, or fuse-overlayfs (needsprogram is required /dev/fuse).- Image-declared
VOLUMEs (dind's/var/lib/docker, Buildah's/var/lib/containers) with no mount become a fresh directory in containerd's state on the node for every container start : the data is lost on each restart. - Mount an emptyDir with a
sizeLimitand setlimits., or a big build fills the node or is evicted (Requests and Limits).ephemeral- storage
In short : A builder runs as a privileged daemon beside the runner, as a shared BuildKit service, or daemonless in the job Pod. Buildah in a user-namespace Pod is the one shape here that runs
RUN steps in real namespaces without privileged, and only while seccomp stays unconfined.1.Multi-arch
- dockerd's own builder (
docker:dindhas no emulators) and Buildah need binfmt handlers : a registration from a privileged container changes every Pod on that node until reboot. Pin the binfmt image : Buildxqemu.imageandsetup-qemu-actionboth default to a floatinglatest. - Emulated steps can crash (
qemu: uncaught target signal 11) : keepRUNout of the final stage. - Native builders : pick nodes by
kubernetes.io/arch(Scheduling).
| Strategy | Needs | Catch |
|---|---|---|
| QEMU | bundled in BuildKit images, else binfmt handlers | much slower, node-wide |
| Native nodes | one builder per arch (nodeselector=) | needs those nodes |
| Cross-compile | $BUILDPLATFORM stage, ARG TARGETOS TARGETARCH | toolchain must cross-compile |
| Per-arch, then merge | buildx imagetools create, buildah manifest, manifest-tool (kaniko) | extra merge step |
2.Reproducible output
SOURCE_. BuildKit stamps metadata from it; layer file times needDATE_ EPOCH=$(git log -1 --pretty=%ct) rewrite-, and a failed rewrite only warns : compare two digests.timestamp=true - Buildah 1.41+ :
--source-,date- epoch --rewrite-. kaniko :timestamp --reproducible. - Pin bases by digest and
compatibility-(30). Provenance records the build.version
3.Caching
- A per-job builder starts cold. A registry cache survives :
--cache-to type=registry,(ref=..., mode=max min, the default, skips earlier stages). RUN --mount=type=cacheis not exported : a warm builder needs a PVC-backed StatefulSet. Buildah needs--layersfor--cache-to, kaniko--cache-repo.- Never share a BuildKit cache between trusted and untrusted builds (a poisoning bug, fixed in v0.33.1).
4.Pushing
- A
dockerconfigjsonSecret asconfig.json(BuildKit$DOCKER_, kanikoCONFIG /kaniko/.docker/, Buildah--authfile). Record the digest. - Pull request builds never push. Only protected branches do, with a narrow write-only token (Gitea's job token cannot publish packages).
- Anonymous Docker Hub pulls : 100 per 6 hours per IPv4 address or IPv6 /64, shared by one NAT. Log in or mirror.
- Then sign and gate it, and let GitOps pick up the tag (Flux, Image Updater).
Seen in practicePitfall : an arm64 leg that compiles native code under QEMU : every build takes many minutes and a cache miss repeats them. Do better : cross-compile in a
$BUILDPLATFORM stage where the toolchain allows it, move the heavy compile into a rarely rebuilt base image, and keep a mode=max registry cache.5.Commands
| Command | Does |
|---|---|
docker buildx create --bootstrap --name kube --driver kubernetes --driver-opt namespace=buildkit,rootless=true | BuildKit Pods as a Buildx builder |
docker buildx build --platform linux/amd64,linux/arm64 --push --cache-to type=registry,ref=<repo>:buildcache,mode=max --cache-from type=registry,ref=<repo>:buildcache -t <repo>:<tag> . | Multi-arch with a registry cache |
buildah build --platform linux/amd64,linux/arm64 --manifest <list> --source-date-epoch "$(git log -1 --pretty=%ct)" --rewrite-timestamp . | Multi-arch, reproducible |
buildah manifest push --all --digestfile digest.txt <list> docker://<repo>:<tag> | Push all, keep the digest |
kubectl label ns <builds> pod-security.kubernetes.io/enforce=privileged | Allow privileged here |
In short : Cross-compile instead of emulating, set
SOURCE_DATE_EPOCH and the timestamp rewrite, push a mode=max registry cache, and give push credentials only to protected-branch builds.