Networking internals
Pod Networking and CNI
1.What Kubernetes asks for
- Kubernetes states the rules (one IP per Pod, Pod to Pod with no NAT). A plugin builds them.
- Observed on k3s v1.37.1 : Pod IPs crossed the tunnel intact (
10.42.1.2 > 10.42.0.5).
Three address owners
- Pod IPsthe network plugin (CNI)
- Service IPskube-apiserver
- Node IPskubelet or cloud controller
2.Who calls CNI
- The kubelet only calls CRI. containerd makes the netns (none for hostNetwork) and runs CNI.
- The kubelet sorts the IPs into
status.podIPs, then setsPodReadyToStartContainers(GA v1.37). - Errors :
kubectl get events -A --field-selector reason=FailedCreatePodSandBox
3.The CNI contract
- A plugin is a binary :
CNI_*env in, JSON config on stdin, a result (ips,routes) or an error code on stdout. - Spec 1.1.0 added
GCandSTATUS. containerd sends STATUS (1.1.0 configs only), never GC. - DEL runs the chain in reverse, must be repeatable, and follows every ADD, even a failed one.
| Variable | ADD | DEL | GC | STATUS |
|---|---|---|---|---|
CNI_COMMAND | req | req | req | sent |
CNI_CONTAINERID | sandbox | req | - | - |
CNI_NETNS | req | opt | - | - |
CNI_IFNAME | eth0 | req | - | - |
CNI_ARGS | K8S_POD_* | opt | - | - |
CNI_PATH | opt | opt | req | opt |
4.A conflist, chained
- containerd loads only the lexically first file (
max_conf_num = 1) : a runtime rule, not the spec. - k3s uses
/.var/ lib/ rancher/ k3s/ agent/ etc/ cni/ net. d - Later plugins get
prevResult; the main one runs its own IPAM plugin. bandwidth: Pod egress is shaped on anifbnamedbwp<hash>.
k3s 10-flannel.conflist
{ "name": "cbr0", "cniVersion": "1.0.0", "plugins": [
{ "type": "flannel", "delegate": { "hairpinMode": true,
"forceAddress": true, "isDefaultGateway": true } },
{ "type": "portmap", "capabilities": { "portMappings": true } },
{ "type": "bandwidth", "capabilities": { "bandwidth": true } } ] }
5.A node with no network
- Empty conf dir :
cni plugin not initialized, NodeReady=False,not-readytaints (evict after 300s). - Only hostNetwork Pods start, so CNI agents are hostNetwork DaemonSets.
- A conflist
mv'd in is ignored (create event). Checkcrictl info | jq '..lastCNILoadStatus' NetworkUnavailableis a separate signal with its own taint.
| Writer | NetworkUnavailable reason |
|---|---|
| Route controller | RouteCreated, NoRouteCreated |
| Upstream flannel | FlannelIsUp |
| calico-node | CalicoIsUp, CalicoIsDown |
| Cilium operator | CiliumIsUp |
| kubelet, k3s flannel | never |
In short : Kubernetes only states the rules. The runtime makes a netns and runs the first conflist's plugin chain for the Pod's IPs; with no conflist, only hostNetwork Pods run.
1.Inside the Pod, on the host
- Pod :
eth0@if5,10.42.1.2/24,default via 10.42.1.1(thecni0address). - Host :
veth71d39cee@if2 ... master cni0.if5and index5:are one veth pair. cni0appears with the first non-hostNetwork Pod. Pod MTU 1450 fromFLANNEL_MTU.
Observed on k3s v1.37.1+k3s1, flannel VXLAN.
2.Bridge or routed
- flannel : Pods share an L2 on
cni0, so a Pod withNET_RAWcan ARP-spoof. k3s says use restricted. - Calico : a
cali*host side with proxy ARP, one /32 route per local Pod. - Calico v3.33 has Felix, not BIRD, program IPIP routes by default. Its docs page still says BGP.
| flannel + bridge | Calico routed | |
|---|---|---|
| Host side | port on cni0 | cali*, /32 route |
| Pod gateway | cni0 (.1) | 169.254.1.1 |
| Neighbours | local Pods, one L2 | MAC ee:ee:ee:ee:ee:ee |
| ARP spoofing | yes, with NET_RAW | no shared L2 |
3.One packet across nodes (flannel VXLAN, observed)
- Per peer : one route, one neighbour, one FDB entry (
ip route,ip neigh,bridge fdb), from Node annotations. tcpdump -nni <uplink> udp port 8472:VXLAN ... vni 1aroundIP 10.42.1.2 > 10.42.0.5.- 50 bytes each :
ping -M do -s 1422passes, 1423 fails. A missed MTU : pings work, big transfers hang.
4.Overlay, routed or VPC-native
- Overlay : only UDP between nodes. Routed : one L2 (
host-gw) or BGP. VPC-native : real VPC IPs. - Cilium defaults to
routingMode: tunnel(VXLAN). - 8472 (kernel) vs 4789 (IANA) : a firewall for one breaks the other. Never expose 8472.
| On a 1500 link | Port | Cost | Pod MTU |
|---|---|---|---|
| flannel vxlan | UDP 8472 | 50 | 1450 |
| flannel host-gw | none | 0 | 1500 |
| wireguard-native | UDP 51820 | 80 | 1420 |
| Calico VXLAN | UDP 4789 | 50 | 1450 |
| Cilium VXLAN, Geneve | 8472, 6081 | 50 | 1450 |
5.k3s defaults and multi-NIC traps
- Pods
10.42.0.0/16, Services10.43.0.0/16, DNS10.43.0.10. - Backends :
vxlan,host-gw,wireguard-native,none. Others fail. - flannel binds the default-route NIC, not the node-ip one : nodes Ready, 100% Pod loss.
flannel.1dies with its uplink; k3s v1.36.0 to v1.37.1 (IPv4-only) then panic and restart.
/etc/rancher/k3s/config.yaml
node-ip: 172.31.254.11
flannel-iface: fabric0 # on every node
ip -d link show flannel.1:local <ip> dev <nic>flannel.must equal the InternalIP.alpha. coreos. com/ public-ip - Alert on a missing
flannel.1, k3s-agent restarts.
In short : Pods hang off veths on their node. flannel wraps cross-node packets in UDP 8472 (MTU 1450), routed designs skip the tunnel, and multi-NIC k3s needs
flannel-iface.1.Node Pod ranges
- KCM
--allocate-node-cidrs(off by default) carves /24 and /64 blocks. spec.podCIDRsis set once : no change, not even a second family.- A /16 of /24s holds 256 nodes; node 257 gets
CIDRNotAvailable, no range. - IPv6 : /64 per node needs a /48 or longer cluster CIDR.
kubectl get nodes -o custom-columns=N:.metadata. name,CIDR:. spec. podCIDRs
2.When the CNI owns IPAM
- In-tree ClusterCIDR was removed in v1.29.
node-ipam-controllerv0.3.0 brings it back as a CRD. - Calico and Cilium ignore
podCIDRs; block sizes are permanent. - Cilium's default
10.0.0.0/8can hit node networks.
| IPAM model | Picked by | Per node |
|---|---|---|
| RangeAllocator | KCM | /24, 253 IPs |
| ClusterCIDR | node-ipam-ctrl | perNodeHostBits |
| calico-ipam | Calico | /26 blocks |
| cluster-pool | cilium-operator | mask 24 |
3.Service ranges are objects
- ServiceCIDR and IPAddress : GA v1.33,
v1beta1gone in v1.37. Seekubectl get servicecidr,ipaddress. - Random picks use the upper band : hand-picked IPs (DNS
.10) go low. - Full :
failed to allocate a serviceIP ... range is full. - v1.37 : no IPv4 size limit. IPv6 shorter than /64 shows Ready, allocates nothing.
Grow it live
apiVersion: networking.k8s.io/v1 kind: ServiceCIDR metadata: name: newcidr1 spec: cidrs: ["10.96.0.0/24"]
The failed Service then got 10.96.0.173. Guard it : Admission.
4.Dual-stack end to end
- Both families on every component : apiserver, KCM, kubelet
--node-ip(first = primary), CNI. status.podIPsfollows the node's primary family, not the CNI (10.42.1.2, thenfd00:42:0:1::2). k3s : dual-stack only at creation.ipFamilyPolicy:SingleStack(default),PreferDualStack,RequireDualStack. Primary never changes.get svchides the second IP :kubectl describe svc <s> | grep -E '^IP'.
5.Symptoms, and second networks
- portmap : skip
KUBE-MARK-MASQ(absent in nftables mode); its nftables backend before v1.9.0 has CVE-2025-67499. - Multus sorts first (
00-multus.conf) and addsnet1. DRA reportsnetworkData(GA v1.37).
| Symptom | Check | Cause |
|---|---|---|
| no IP addresses available | files in / | leaks (no CNI GC) or too many Pods |
| cni0 already has an IP | did the node subnet change ? | forceAddress: false |
| subnet doesn't contain podCIDR | flannel Network vs KCM | kubeadm range not 10.244.0.0/16 |
| hostPort refused, no events | conflist portMappings ? | no portmap in the chain |
| incompatible CNI versions | CNI_ on the binary | plugins older than the conflist |
In short : Plan addresses once : node ranges never change, ServiceCIDRs grow live, and dual-stack needs both families on every component from day one.