Networking internals

Pod Networking and CNI

1.What Kubernetes asks for

  • Kubernetes states the rules (one IP per Pod, Pod to Pod with no NAT). A plugin builds them.
  • Observed on k3s v1.37.1 : Pod IPs crossed the tunnel intact (10.42.1.2 > 10.42.0.5).
Three address owners
  • Pod IPsthe network plugin (CNI)
  • Service IPskube-apiserver
  • Node IPskubelet or cloud controller

2.Who calls CNI

  • The kubelet only calls CRI. containerd makes the netns (none for hostNetwork) and runs CNI.
  • The kubelet sorts the IPs into status.podIPs, then sets PodReadyToStartContainers (GA v1.37).
  • Errors : kubectl get events -A --field-selector reason=FailedCreatePodSandBox
kubelet containerd (CRI) netns cni-<uuid> RunPodSandbox create status.podIPs sorted by node family libcni first conflist only CNI ADD : env + stdin stdout : ips, routes flannel main plugin bridge cni0 + veth exec host-local IPAM exec portmap hostPort then bandwidth tc shaping CNI agent DaemonSet flanneld, calico-node, cilium-agent hostNetwork, runs before any Pod writes the conflist + binaries, programs routes between nodes

3.The CNI contract

  • A plugin is a binary : CNI_* env in, JSON config on stdin, a result (ips, routes) or an error code on stdout.
  • Spec 1.1.0 added GC and STATUS. containerd sends STATUS (1.1.0 configs only), never GC.
  • DEL runs the chain in reverse, must be repeatable, and follows every ADD, even a failed one.
VariableADDDELGCSTATUS
CNI_COMMANDreqreqreqsent
CNI_CONTAINERIDsandboxreq--
CNI_NETNSreqopt--
CNI_IFNAMEeth0req--
CNI_ARGSK8S_POD_*opt--
CNI_PATHoptoptreqopt

4.A conflist, chained

  • containerd loads only the lexically first file (max_conf_num = 1) : a runtime rule, not the spec.
  • k3s uses /var/lib/rancher/k3s/agent/etc/cni/net.d.
  • Later plugins get prevResult; the main one runs its own IPAM plugin.
  • bandwidth : Pod egress is shaped on an ifb named bwp<hash>.
k3s 10-flannel.conflist
{ "name": "cbr0", "cniVersion": "1.0.0", "plugins": [
  { "type": "flannel", "delegate": { "hairpinMode": true,
      "forceAddress": true, "isDefaultGateway": true } },
  { "type": "portmap", "capabilities": { "portMappings": true } },
  { "type": "bandwidth", "capabilities": { "bandwidth": true } } ] }

5.A node with no network

  • Empty conf dir : cni plugin not initialized, Node Ready=False, not-ready taints (evict after 300s).
  • Only hostNetwork Pods start, so CNI agents are hostNetwork DaemonSets.
  • A conflist mv'd in is ignored (create event). Check crictl info | jq '.lastCNILoadStatus'.
  • NetworkUnavailable is a separate signal with its own taint.
WriterNetworkUnavailable reason
Route controllerRouteCreated, NoRouteCreated
Upstream flannelFlannelIsUp
calico-nodeCalicoIsUp, CalicoIsDown
Cilium operatorCiliumIsUp
kubelet, k3s flannelnever
In short : Kubernetes only states the rules. The runtime makes a netns and runs the first conflist's plugin chain for the Pod's IPs; with no conflist, only hostNetwork Pods run.

1.Inside the Pod, on the host

  • Pod : eth0@if5, 10.42.1.2/24, default via 10.42.1.1 (the cni0 address).
  • Host : veth71d39cee@if2 ... master cni0. if5 and index 5: are one veth pair.
  • cni0 appears with the first non-hostNetwork Pod. Pod MTU 1450 from FLANNEL_MTU.

Observed on k3s v1.37.1+k3s1, flannel VXLAN.

Pod A netns eth0 10.42.1.2 Pod B netns eth0 10.42.1.3 veth veth cni0 bridge 10.42.1.1 same node : L2 over cni0 node routes 10.42.0.0/24 flannel.1 VTEP eth0 uplink other node

2.Bridge or routed

  • flannel : Pods share an L2 on cni0, so a Pod with NET_RAW can ARP-spoof. k3s says use restricted.
  • Calico : a cali* host side with proxy ARP, one /32 route per local Pod.
  • Calico v3.33 has Felix, not BIRD, program IPIP routes by default. Its docs page still says BGP.
flannel + bridgeCalico routed
Host sideport on cni0cali*, /32 route
Pod gatewaycni0 (.1)169.254.1.1
Neighbourslocal Pods, one L2MAC ee:ee:ee:ee:ee:ee
ARP spoofingyes, with NET_RAWno shared L2

3.One packet across nodes (flannel VXLAN, observed)

  • Per peer : one route, one neighbour, one FDB entry (ip route, ip neigh, bridge fdb), from Node annotations.
  • tcpdump -nni <uplink> udp port 8472 : VXLAN ... vni 1 around IP 10.42.1.2 > 10.42.0.5.
  • 50 bytes each : ping -M do -s 1422 passes, 1423 fails. A missed MTU : pings work, big transfers hang.
Node 2 : 172.31.250.11, holds 10.42.1.0/24 Pod B 10.42.1.2 cni0 gateway .1 route via flannel.1 neighbour to VtepMAC FDB MAC to .250.10 flannel.1 wrap, UDP 8472 outer 172.31.250.11 > .10, vni 1, inner Pod IPs untouched Node 1 : 172.31.250.10, holds 10.42.0.0/24 flannel.1 unwrap cni0 bridge Pod A 10.42.0.5

4.Overlay, routed or VPC-native

  • Overlay : only UDP between nodes. Routed : one L2 (host-gw) or BGP. VPC-native : real VPC IPs.
  • Cilium defaults to routingMode: tunnel (VXLAN).
  • 8472 (kernel) vs 4789 (IANA) : a firewall for one breaks the other. Never expose 8472.
On a 1500 linkPortCostPod MTU
flannel vxlanUDP 8472501450
flannel host-gwnone01500
wireguard-nativeUDP 51820801420
Calico VXLANUDP 4789501450
Cilium VXLAN, Geneve8472, 6081501450

5.k3s defaults and multi-NIC traps

  • Pods 10.42.0.0/16, Services 10.43.0.0/16, DNS 10.43.0.10.
  • Backends : vxlan, host-gw, wireguard-native, none. Others fail.
  • flannel binds the default-route NIC, not the node-ip one : nodes Ready, 100% Pod loss.
  • flannel.1 dies with its uplink; k3s v1.36.0 to v1.37.1 (IPv4-only) then panic and restart.
/etc/rancher/k3s/config.yaml
node-ip: 172.31.254.11
flannel-iface: fabric0  # on every node
  • ip -d link show flannel.1 : local <ip> dev <nic>
  • flannel.alpha.coreos.com/public-ip must equal the InternalIP.
  • Alert on a missing flannel.1, k3s-agent restarts.
In short : Pods hang off veths on their node. flannel wraps cross-node packets in UDP 8472 (MTU 1450), routed designs skip the tunnel, and multi-NIC k3s needs flannel-iface.

1.Node Pod ranges

  • KCM --allocate-node-cidrs (off by default) carves /24 and /64 blocks.
  • spec.podCIDRs is set once : no change, not even a second family.
  • A /16 of /24s holds 256 nodes; node 257 gets CIDRNotAvailable, no range.
  • IPv6 : /64 per node needs a /48 or longer cluster CIDR.
  • kubectl get nodes -o custom-columns=N:.metadata.name,CIDR:.spec.podCIDRs
--cluster-cidr 10.42.0.0/16 node IPAM controller RangeAllocator in KCM Node A 10.42.0.0/24 Node B 10.42.1.0/24 node 257 CIDRNotAvailable podCIDRs flanneld writes subnet.env host-local .2 to .254 = 253 Pod IPs per Pod ADD

2.When the CNI owns IPAM

  • In-tree ClusterCIDR was removed in v1.29. node-ipam-controller v0.3.0 brings it back as a CRD.
  • Calico and Cilium ignore podCIDRs; block sizes are permanent.
  • Cilium's default 10.0.0.0/8 can hit node networks.
IPAM modelPicked byPer node
RangeAllocatorKCM/24, 253 IPs
ClusterCIDRnode-ipam-ctrlperNodeHostBits
calico-ipamCalico/26 blocks
cluster-poolcilium-operatormask 24

3.Service ranges are objects

  • ServiceCIDR and IPAddress : GA v1.33, v1beta1 gone in v1.37. See kubectl get servicecidr, ipaddress.
  • Random picks use the upper band : hand-picked IPs (DNS .10) go low.
  • Full : failed to allocate a serviceIP ... range is full.
  • v1.37 : no IPv4 size limit. IPv6 shorter than /64 shows Ready, allocates nothing.
Grow it live
apiVersion: networking.k8s.io/v1
kind: ServiceCIDR
metadata:
  name: newcidr1
spec:
  cidrs: ["10.96.0.0/24"]

The failed Service then got 10.96.0.173. Guard it : Admission.

4.Dual-stack end to end

  • Both families on every component : apiserver, KCM, kubelet --node-ip (first = primary), CNI.
  • status.podIPs follows the node's primary family, not the CNI (10.42.1.2, then fd00:42:0:1::2). k3s : dual-stack only at creation.
  • ipFamilyPolicy : SingleStack (default), PreferDualStack, RequireDualStack. Primary never changes.
  • get svc hides the second IP : kubectl describe svc <s> | grep -E '^IP'.

5.Symptoms, and second networks

  • portmap : skip KUBE-MARK-MASQ (absent in nftables mode); its nftables backend before v1.9.0 has CVE-2025-67499.
  • Multus sorts first (00-multus.conf) and adds net1. DRA reports networkData (GA v1.37).
SymptomCheckCause
no IP addresses availablefiles in /var/lib/cni/networks/<net>leaks (no CNI GC) or too many Pods
cni0 already has an IPdid the node subnet change ?forceAddress: false
subnet doesn't contain podCIDRflannel Network vs KCMkubeadm range not 10.244.0.0/16
hostPort refused, no eventsconflist portMappings ?no portmap in the chain
incompatible CNI versionsCNI_COMMAND=VERSION on the binaryplugins older than the conflist
In short : Plan addresses once : node ranges never change, ServiceCIDRs grow live, and dual-stack needs both families on every component from day one.