Networking internals
The Service Dataplane
1.A controller, not a proxy
A Service is kernel NAT rules, not a process. This picks up where the watch loops in Services stop (v1.37).
- Informers : Services (never headless ones), EndpointSlices, ServiceCIDRs and its own Node.
- Runner : syncs at least 1s apart, retries within 30s, full sync at least hourly.
- Partial sync : changed Services only. Over 1000 endpoints, iptables skips the hourly full sync.
2.Ports, health and metrics
:10256/healthz: 200 unless an update waited over 2 xsyncPeriod(60s) or the Node is being deleted./livezignores the Node.127.0.0.1:10249(loopback) :/metrics,/configz,/proxyMode. Trycurl -s 127..0. 0. 1:10249/ proxyMode - k3s binds healthz to 127.0.0.1, so LB probes on
<nodeIP>:10256fail.
| kubeproxy_... | Tells you |
|---|---|
| sync_full_ / sync_partial_ | Sync cost |
| network_programming_ | Slice change to rules |
| sync_proxy_rules_ | Staleness |
| ..._iptables_restore_ | Failed syncs |
3.The config that matters
- Unset
mode= iptables plus an error log line. TheKubeProxyDefaultModeevent never arrives : grep the logs forNo kube-proxy mode specified. - Sync settings come from the section matching
mode. Flags lose to--config, and an unknown field only logs. Lint with--write-config-to.
Pin the mode (validated, v1.37.1)
apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: nftables # never leave it unset clusterCIDR: 10.244.0.0/16 nftables: # not iptables: syncPeriod: 30s # retry, not resync
4.iptables mode, chain by chain
KUBE-SERVICES: one rule per Service port, walked in order. O(Services) per new connection.KUBE-SVC: rule i jumps with--probability 1/(N-i), the last always. Each endpoint gets 1/N.KUBE-SEPDNATs toPodIP:port. Marked packets (0x4000) are masqueraded on the way out.
| Hook | Jumps to (verdict) |
|---|---|
| nat PREROUTING, OUTPUT | KUBE-SERVICES (DNAT) |
| nat POSTROUTING | KUBE-POSTROUTING (SNAT) |
| filter FORWARD, OUTPUT | KUBE-SERVICES (REJECT) |
| filter INPUT, FORWARD | KUBE-EXTERNAL-SERVICES (DROP) |
Real chain names from iptables-save on a kind node (v1.37.1, observed 2026-10-07).
5.When rules vanish
- Only the mangle
KUBE-PROXY-CANARYis polled. If it goes, a full resync follows (observed : 18s). - Flushing only nat or filter goes unnoticed (observed : broken after 80s). Restart kube-proxy.
iptables-save is not the input
The mark prints as --set-xmark 0x4000/0x4000, probabilities as 0.50000000000. Grep names : iptables-save -t nat | grep -E 'KUBE-(
In short : kube-proxy turns Services and EndpointSlices into node rules. In iptables mode a new connection walks
KUBE-SERVICES, takes a random jump and is DNATed.1.nftables mode
- Owns
table ip kube-proxyandtable ip6 kube-proxy(noinettable). Base chains :nat-prerouting,nat-output,nat-postrouting, filter chains. servicesmarks off-cluster ClusterIP traffic, then does one lookup :nft list map ip kube-proxy service-ips.
- A Service chain picks the endpoint in one
numgen random mod Nrule (since v1.36). One endpoint : a plaindnat to. Endpoint chains remain only for ClientIP affinity. - The
no-endpoint-*maps reject (no endpoints) or drop (none local). Never edit this table or the mark.
nft list table ip kube-proxy (kind, v1.37.1, observed)
chain services {
ip daddr @cluster-ips ip saddr != 10.244.0.0/16 meta mark set meta mark | 0x00004000 ...
ip daddr . meta l4proto . th dport vmap @service-ips
ip daddr @nodeport-ips meta l4proto . th dport vmap @service-nodeports }
chain service-SUSTUHLJ-default/web/tcp/http {
meta l4proto tcp dnat ip to numgen random mod 2 map { 0 : 10.244.0.5 . 8080, 1 : 10.244.1.2 . 8080 } }
2.Why it scales, and the switch
- iptables : O(Services) per new connection and update, global lock. nftables : one lookup, updates O(changes).
- Changes : NodePorts on the primary IP only, none on
localhost(v1.37 alpha gate). Open the NodePort range yourself. Kernel 5.13+. - Switch : check the
kubeproxy_counters, setiptables_* mode: nftables. Old rules go away; mixed nodes are fine.
| Release | iptables | nftables | ipvs |
|---|---|---|---|
| v1.37 | default | GA | deprecated, gated |
| v1.39 | kubeadm rejects 5.4 and 5.10 kernels | ||
| v1.40 | not default | default | off |
| v1.43 | ok | ok | removed |
From v1.39 on : plans (KEP-5343 is provisional).
3.ipvs, while it lasts
- Service IPs sit as /32s on the dummy
kube-ipvs0. One IPVS virtual server per IP and port, schedulerrr, NAT. - Masquerade, NodePorts and filtering still use iptables, matched against 17 ipsets.
- Leaving ipvs for iptables keeps
kube-ipvs0: stop kube-proxy, then runkube-proxy --cleanup.
4.conntrack, the hidden half
- Later packets skip the nat table, so only new connections re-balance. gRPC and HTTP/2 stay put. Pins :
conntrack -L -d <clusterIP>. - UDP : stale entries are deleted after a sync (v1.37 : also with zero endpoints). TCP : never.
- Max = 32768 x CPUs, capped at 1,048,576 (v1.36), at least 131072. Check
conntrack -S. - Read-only sysctl (LXC) : v1.36 and v1.37.1 can fail to start. Set
maxPerCore: 0. - TCP established 24h, close-wait 1h. k3s leaves both to the kernel.
5.SNAT and the client IP
- SNAT :
masqueradeAll, off-cluster or host sources to a ClusterIP, hairpin, and all external traffic under eTP=Cluster. - MASQUERADE uses the egress address. "Local Pod" comes from
detectLocalMode(defaultClusterCIDR; noclusterCIDR= no-op).
| Path | Backend sees (observed) |
|---|---|
| Pod to ClusterIP | the client Pod IP |
| Pod to its own Service | SNAT, 10.244.0.1 |
| host to ClusterIP | node IP if remote, 10.244.0.1 if local |
| external, eTP=Cluster | a node address, even if local |
Test : wget -qO- http://<svc>/ (agnhost). eTP=Local : Services.
In short : nftables swaps the rule walk for one map lookup (planned default in v1.40), ipvs is leaving, conntrack makes the first pick stick, and SNAT decides who the backend sees.
1.What kube-proxy reads
servingmirrors Pod Ready,terminatingcomes with deletion,ready= serving and not terminating.- Local =
nodeNameequals kube-proxy's node name, never by IP. A wronghostnameOverridemakes all endpoints remote. - 100 endpoints per slice by default (Controllers).
| Field or label | Effect |
|---|---|
kubernetes. | Required link |
addresses[0] | The only IP used |
conditions | ready, serving, terminating |
nodeName | Locality |
zone, hints | Topology |
2.Which endpoints a connection can reach
- Per policy : Ready endpoints, else serving and terminating ones. This holds for Cluster policy too (docs say Local only).
- Observed : a Pod with a 120s
preStopsleep, scaled to 0, kept taking new connections. - kube-proxy uses hints only if every Ready endpoint has one. Local policies win over hints.
PreferSameZonehas no overload guard : spread replicas (Scheduling). Values : Services.
3.Health checks for Local policies
- Past the eTP panel in Services : one listener per Service on
healthCheckNodePort, 200 with local Ready endpoints, else 503, plus headerX-Load-Balancing-Endpoint-Weight. - Probe :
curl -si http://<nodeIP>:<port>/. Terminating local Pods fail it, so the LB drains the node.healthz - iTP=Local with no local endpoint DROPs (timeout), not the REJECT the docs imply.
4.eBPF : Services without netfilter
- Verified programs run at hooks (XDP, tc, cgroup
connect4) and keep state in maps. - Cilium
kubeProxyReplacement: true: socket LB atconnect(), NodePorts per packet (snat,dsr,hybrid).
- Calico eBPF : connect-time LB for TCP only. NodePorts keep the client IP, even with eTP=Cluster.
- Traps : socket LB breaks Istio sidecars, KubeVirt, Kata, gVisor. tcpdump misses XDP.
| iptables | nftables | ipvs | Cilium KPR | Calico eBPF | |
|---|---|---|---|---|---|
| v1.37 | default | GA | deprecated | opt-in | opt-in |
| Dispatch | rule walk | one map | IPVS table | BPF map | BPF map |
| State | conntrack | conntrack | + IPVS | BPF maps | BPF maps |
| Kernel | - | 5.13 | - | 5.10 | 5.10 |
5.Removing kube-proxy
- kubeadm :
--skip-phases=addon/. Live cluster : delete the DaemonSet and ConfigMap, runkube-proxy kube-proxy --cleanup, setk8sServiceHost. - Cilium breaks open Service connections in the switch. Calico keeps TCP flows, drops VXLAN ones, wants ipvs gone first. k3s :
--disable-kube-proxy, or--kube-proxy-arg=proxy-mode=nftables. - Check :
cilium-dbg status --verbose | grep -A12 'KubeProxyReplacement Details'. CNI side : Pod Networking.
In short : Slice conditions and hints pick the reachable Pods, health checks steer LBs to nodes with one, and eBPF can do it all at
connect().