CI and builds

Building Images in the Cluster

1.A RUN step is a container

  • A builder unpacks the base image. Each RUN runs in a new container over the layer stack and is saved as a layer. Then the image is pushed.
  • For that it calls unshare or clone with CLONE_NEWUSER, NEWNS and NEWPID, mounts overlay layers and a fresh /proc, and writes files as many UIDs.
  • No Docker socket on containerd nodes (Images and Pulls).
Two traps
  • No seccomp profile means Unconfined, unless the kubelet sets seccompDefault: true (upstream default false). A builder can break the day a node flips it.
  • In dind, containers get Docker's default profile, so Buildah fails with unshare(CLONE_NEWUSER) : not permitted, unless the job container is privileged or unconfined.

2.The four fences

CI job builder dockerd, buildkitd, buildah, kaniko each RUN needs unshare, mount, chown as root fences seccomp RuntimeDefault : no unshare, no mount AppArmor default : deny mount masked /proc : no fresh /proc Baseline : no privileged, no Unconfined ways past privileged : every fence down Unconfined profiles : rootless BuildKit user namespace Pod : root is not host root userspace snapshots : kaniko

3.The options

OptionPod needsLowest level
Docker-in-Dockerprivileged: truePrivileged
Rootless dindstill privileged: truePrivileged
BuildKit rootlessUnconfined profiles, UID 1000Privileged (explicit Unconfined)
BuildKit usernshostUsers: false plus privileged: truePrivileged
Buildah, userns PodhostUsers: false, UID 1000, no seccomp fieldBaseline, while seccomp is effectively Unconfined
kaniko forksroot, no extra capabilitiesBaseline

Nothing that runs RUN steps with real namespaces is shown to fit Restricted. Levels : Pod Security.

4.Docker-in-Docker

  • The official image says --privileged is required, and the rootless variant needs it too : "rootless" is the daemon's UID, not the Pod's power.
  • A privileged Pod is node root. Any workflow author on that runner can own the node, and the namespace must allow privileged.
  • ARC's dind mode and Gitea's dind sidecar are this : ARC, Gitea runners. Best as a native sidecar.
  • TLS off (DOCKER_TLS_CERTDIR="") and arguments empty or starting with a flag : the stock entrypoint listens on tcp://0.0.0.0:2375, no auth. Keep TLS, or start the arguments with the word dockerd.

5.kaniko, archived

  • Google archived it in June 2025 (last release v1.24.0). Forks : osscontainertools publishes images, Chainguard source tags only.
  • It is the exception : commands run in its own root and are snapshotted in userspace, so it needs root but no new namespaces and fits Baseline. Its README says it does not make untrusted builds safe, and it must run in its own image.
  • No multi-arch index of its own : use manifest-tool.
Seen in practicePitfall : a dind container beside the job with no startup probe, so the first docker call fails and every workflow grows its own until docker info loop. Do better : run dind as a native sidecar with a docker info startup probe, as ARC's dind mode does, so the job starts only once the daemon answers.
In short : A build needs namespaces and mounts. dind and BuildKit get them by being privileged or unconfined, Buildah can get them from a user-namespace Pod, kaniko fakes them in userspace. Pick the smallest hole you can live with.

1.Three shapes, and who gives you which

a. daemon beside the runner runner socket dockerd privileged RUN containers b. builder service CI job buildctl mTLS buildkitd Pods push registry c. daemonless, in the job Pod job Pod runs the builder buildah, kaniko or buildctl-daemonless.sh push registry no daemon to keep running
RunnerDaemon in the job ? Build with
ARC dindyes, privileged sidecar : docker build, Buildx
ARC kubernetesno : daemonless, remote BuildKit, or a hook template that adds dind
Gitea with dindyes, job containers privileged only with container.privileged: true : docker build, Buildah in a privileged job
Gitea kubernetes://no : daemonless or remote BuildKit

2.BuildKit on Kubernetes

  • ExamplesUpstream examples/kubernetes : Pod, Deployment plus Service with mTLS, StatefulSet (warm cache), Job with buildctl-daemonless.sh. Each privileged, rootless (recommended) or userns. All use master tags : pin one.
  • RootlessStill seccomp and AppArmor Unconfined, UID 1000, and --oci-worker-no-process-sandbox : leftover step processes are not killed and steps can signal the daemon.
  • Buildx driverdocker buildx create --driver kubernetes makes the Pods, privileged by default. rootless=true swaps in Unconfined profiles. A persistent-volume-claim option gives a StatefulSet with a PVC (v0.34.0+).
  • Traprequests.ephemeral-storage sets a limit, not a request : scratch use above it evicts the builder.
  • Clientbuildctl --addr kube-pod://<pod> needs pods/exec (RBAC).

3.Buildah in a user-namespace Pod

spec:   # abridged
  hostUsers: false
  containers:
  - name: buildah
    image: quay.io/buildah/stable:v1.43.4-immutable
    securityContext: {runAsUser: 1000}   # else root
  volumes:      # storage.conf points overlay here
  - {name: storage, emptyDir: {sizeLimit: 10Gi}}
  • No privileged, capabilities, /dev/fuse or seccomp field : Baseline on the field rules, but it works only while the node runs Pods Unconfined.
  • Rootless BuildKit cannot do this (rootlesskit needs its own mappings) : BuildKit in a user namespace runs the rootful image with privileged: true.

4.User namespaces and Pod Security

RuleWith hostUsers: false
Restricted runAsNonRoot, runAsUser; Baseline procMountskipped; Unmasked allowed only with it
Policy versiononly v1.35 and later : a namespace pinned to enforce-version: v1.34 gets none
Still checkedprivileged, capabilities, seccomp, AppArmor, host namespaces
privileged: trueallowed, still the Privileged level

The field, node needs and UID mapping : Pod Security.

The Buildah Pod is one published setup (CERN, 2025), not upstream advice.

5.Storage and scratch

  • vfs works anywhere but copies every layer : slow and disk-hungry. Overlay on an overlay root fails (a mount_program is required) : use an emptyDir or PVC, or fuse-overlayfs (needs /dev/fuse).
  • Image-declared VOLUMEs (dind's /var/lib/docker, Buildah's /var/lib/containers) with no mount become a fresh directory in containerd's state on the node for every container start : the data is lost on each restart.
  • Mount an emptyDir with a sizeLimit and set limits.ephemeral-storage, or a big build fills the node or is evicted (Requests and Limits).
In short : A builder runs as a privileged daemon beside the runner, as a shared BuildKit service, or daemonless in the job Pod. Buildah in a user-namespace Pod is the one shape here that runs RUN steps in real namespaces without privileged, and only while seccomp stays unconfined.

1.Multi-arch

source build stage, once per target FROM --platform=$BUILDPLATFORM (native) GOOS=$TARGETOS GOARCH=$TARGETARCH final stage per TARGETPLATFORM COPY --from=build no RUN, so no QEMU index : linux/amd64 + linux/arm64 push to registry other ways : QEMU, one builder per arch
  • dockerd's own builder (docker:dind has no emulators) and Buildah need binfmt handlers : a registration from a privileged container changes every Pod on that node until reboot. Pin the binfmt image : Buildx qemu.image and setup-qemu-action both default to a floating latest.
  • Emulated steps can crash (qemu: uncaught target signal 11) : keep RUN out of the final stage.
  • Native builders : pick nodes by kubernetes.io/arch (Scheduling).
StrategyNeedsCatch
QEMUbundled in BuildKit images, else binfmt handlersmuch slower, node-wide
Native nodesone builder per arch (nodeselector=kubernetes.io/arch=...)needs those nodes
Cross-compile$BUILDPLATFORM stage, ARG TARGETOS TARGETARCHtoolchain must cross-compile
Per-arch, then mergebuildx imagetools create, buildah manifest, manifest-tool (kaniko)extra merge step

2.Reproducible output

  • SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct). BuildKit stamps metadata from it; layer file times need rewrite-timestamp=true, and a failed rewrite only warns : compare two digests.
  • Buildah 1.41+ : --source-date-epoch, --rewrite-timestamp. kaniko : --reproducible.
  • Pin bases by digest and compatibility-version (30). Provenance records the build.

3.Caching

  • A per-job builder starts cold. A registry cache survives : --cache-to type=registry,ref=...,mode=max (min, the default, skips earlier stages).
  • RUN --mount=type=cache is not exported : a warm builder needs a PVC-backed StatefulSet. Buildah needs --layers for --cache-to, kaniko --cache-repo.
  • Never share a BuildKit cache between trusted and untrusted builds (a poisoning bug, fixed in v0.33.1).

4.Pushing

  • A dockerconfigjson Secret as config.json (BuildKit $DOCKER_CONFIG, kaniko /kaniko/.docker/, Buildah --authfile). Record the digest.
  • Pull request builds never push. Only protected branches do, with a narrow write-only token (Gitea's job token cannot publish packages).
  • Anonymous Docker Hub pulls : 100 per 6 hours per IPv4 address or IPv6 /64, shared by one NAT. Log in or mirror.
  • Then sign and gate it, and let GitOps pick up the tag (Flux, Image Updater).
Seen in practicePitfall : an arm64 leg that compiles native code under QEMU : every build takes many minutes and a cache miss repeats them. Do better : cross-compile in a $BUILDPLATFORM stage where the toolchain allows it, move the heavy compile into a rarely rebuilt base image, and keep a mode=max registry cache.

5.Commands

CommandDoes
docker buildx create --bootstrap --name kube --driver kubernetes --driver-opt namespace=buildkit,rootless=trueBuildKit Pods as a Buildx builder
docker buildx build --platform linux/amd64,linux/arm64 --push --cache-to type=registry,ref=<repo>:buildcache,mode=max --cache-from type=registry,ref=<repo>:buildcache -t <repo>:<tag> .Multi-arch with a registry cache
buildah build --platform linux/amd64,linux/arm64 --manifest <list> --source-date-epoch "$(git log -1 --pretty=%ct)" --rewrite-timestamp .Multi-arch, reproducible
buildah manifest push --all --digestfile digest.txt <list> docker://<repo>:<tag>Push all, keep the digest
kubectl label ns <builds> pod-security.kubernetes.io/enforce=privilegedAllow privileged here
In short : Cross-compile instead of emulating, set SOURCE_DATE_EPOCH and the timestamp rewrite, push a mode=max registry cache, and give push credentials only to protected-branch builds.