Networking internals

The Service Dataplane

1.A controller, not a proxy

A Service is kernel NAT rules, not a process. This picks up where the watch loops in Services stop (v1.37).

  • Informers : Services (never headless ones), EndpointSlices, ServiceCIDRs and its own Node.
  • Runner : syncs at least 1s apart, retries within 30s, full sync at least hourly.
  • Partial sync : changed Services only. Over 1000 endpoints, iptables skips the hourly full sync.
informers Services, Slices, Node, CIDRs change trackers old + new state sync runner 1s min, 30s retry, 1h max iptables-restore or one nft transaction client socket Pod or host process 1st packet kernel rules DNAT, SNAT mark records conntrack entry keeps the pick later : replayed

2.Ports, health and metrics

  • :10256/healthz : 200 unless an update waited over 2 x syncPeriod (60s) or the Node is being deleted. /livez ignores the Node.
  • 127.0.0.1:10249 (loopback) : /metrics, /configz, /proxyMode. Try curl -s 127.0.0.1:10249/proxyMode.
  • k3s binds healthz to 127.0.0.1, so LB probes on <nodeIP>:10256 fail.
kubeproxy_...Tells you
sync_full_ / sync_partial_proxy_rules_duration_secondsSync cost
network_programming_duration_secondsSlice change to rules
sync_proxy_rules_last_timestamp_secondsStaleness
..._iptables_restore_failures_total, ..._nftables_sync_failures_totalFailed syncs

3.The config that matters

  • Unset mode = iptables plus an error log line. The KubeProxyDefaultMode event never arrives : grep the logs for No kube-proxy mode specified.
  • Sync settings come from the section matching mode. Flags lose to --config, and an unknown field only logs. Lint with --write-config-to.
Pin the mode (validated, v1.37.1)
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: nftables        # never leave it unset
clusterCIDR: 10.244.0.0/16
nftables:             # not iptables:
  syncPeriod: 30s     # retry, not resync

4.iptables mode, chain by chain

  • KUBE-SERVICES : one rule per Service port, walked in order. O(Services) per new connection.
  • KUBE-SVC : rule i jumps with --probability 1/(N-i), the last always. Each endpoint gets 1/N.
  • KUBE-SEP DNATs to PodIP:port. Marked packets (0x4000) are masqueraded on the way out.
HookJumps to (verdict)
nat PREROUTING, OUTPUTKUBE-SERVICES (DNAT)
nat POSTROUTINGKUBE-POSTROUTING (SNAT)
filter FORWARD, OUTPUTKUBE-SERVICES (REJECT)
filter INPUT, FORWARDKUBE-EXTERNAL-SERVICES (DROP)
-d 10.96.144.12 --dport 80 nat PREROUTING or OUTPUT (host) KUBE-SERVICES 1 rule per port KUBE-SVC-CDGG... off-cluster : mark p=0.5 else KUBE-SEP-7U6M... DNAT 10.244.0.5:8080 KUBE-SEP-UIQK... DNAT 10.244.1.2:8080 last rule (local dst) KUBE-NODEPORTS by --dport KUBE-EXT-<hash> external dst rules KUBE-SVC or SVL SVL : local endpoints KUBE-MARK-MASQ set mark 0x4000 nat POSTROUTING : KUBE-POSTROUTING marked : MASQUERADE --random-fully SVC, SEP and EXT jump to the mark nat PREROUTING or OUTPUT (host) KUBE-SERVICES 1 rule per port -d 10.96.144.12:80 KUBE-SVC-CDGG... off-cluster : mark p=0.5 else KUBE-SEP-7U6M... DNAT 10.244.0.5:8080 KUBE-SEP-UIQK... DNAT 10.244.1.2:8080 KUBE-NODEPORTS last SERVICES rule KUBE-EXT-<hash> then SVC or SVL KUBE-MARK-MASQ set mark 0x4000 KUBE-POSTROUTING MASQUERADE

Real chain names from iptables-save on a kind node (v1.37.1, observed 2026-10-07).

5.When rules vanish

  • Only the mangle KUBE-PROXY-CANARY is polled. If it goes, a full resync follows (observed : 18s).
  • Flushing only nat or filter goes unnoticed (observed : broken after 80s). Restart kube-proxy.
iptables-save is not the input

The mark prints as --set-xmark 0x4000/0x4000, probabilities as 0.50000000000. Grep names : iptables-save -t nat | grep -E 'KUBE-(SERVICES|SVC|SVL|EXT|SEP)'

In short : kube-proxy turns Services and EndpointSlices into node rules. In iptables mode a new connection walks KUBE-SERVICES, takes a random jump and is DNATed.

1.nftables mode

  • Owns table ip kube-proxy and table ip6 kube-proxy (no inet table). Base chains : nat-prerouting, nat-output, nat-postrouting, filter chains.
  • services marks off-cluster ClusterIP traffic, then does one lookup : nft list map ip kube-proxy service-ips.
  • A Service chain picks the endpoint in one numgen random mod N rule (since v1.36). One endpoint : a plain dnat to. Endpoint chains remain only for ClientIP affinity.
  • The no-endpoint-* maps reject (no endpoints) or drop (none local). Never edit this table or the mark.
nft list table ip kube-proxy (kind, v1.37.1, observed)
chain services {
  ip daddr @cluster-ips ip saddr != 10.244.0.0/16 meta mark set meta mark | 0x00004000 ...
  ip daddr . meta l4proto . th dport vmap @service-ips
  ip daddr @nodeport-ips meta l4proto . th dport vmap @service-nodeports }
chain service-SUSTUHLJ-default/web/tcp/http {
  meta l4proto tcp dnat ip to numgen random mod 2 map { 0 : 10.244.0.5 . 8080, 1 : 10.244.1.2 . 8080 } }

2.Why it scales, and the switch

  • iptables : O(Services) per new connection and update, global lock. nftables : one lookup, updates O(changes).
  • Changes : NodePorts on the primary IP only, none on localhost (v1.37 alpha gate). Open the NodePort range yourself. Kernel 5.13+.
  • Switch : check the kubeproxy_iptables_* counters, set mode: nftables. Old rules go away; mixed nodes are fine.
Releaseiptablesnftablesipvs
v1.37defaultGAdeprecated, gated
v1.39kubeadm rejects 5.4 and 5.10 kernels
v1.40not defaultdefaultoff
v1.43okokremoved

From v1.39 on : plans (KEP-5343 is provisional).

3.ipvs, while it lasts

  • Service IPs sit as /32s on the dummy kube-ipvs0. One IPVS virtual server per IP and port, scheduler rr, NAT.
  • Masquerade, NodePorts and filtering still use iptables, matched against 17 ipsets.
  • Leaving ipvs for iptables keeps kube-ipvs0 : stop kube-proxy, then run kube-proxy --cleanup.

4.conntrack, the hidden half

  • Later packets skip the nat table, so only new connections re-balance. gRPC and HTTP/2 stay put. Pins : conntrack -L -d <clusterIP>.
  • UDP : stale entries are deleted after a sync (v1.37 : also with zero endpoints). TCP : never.
  • Max = 32768 x CPUs, capped at 1,048,576 (v1.36), at least 131072. Check conntrack -S.
  • Read-only sysctl (LXC) : v1.36 and v1.37.1 can fail to start. Set maxPerCore: 0.
  • TCP established 24h, close-wait 1h. k3s leaves both to the kernel.
Pod 10.244.0.6:48986 SYN to 10.96.144.12:80 first packet nat rules pick 10.244.1.2:8080, DNAT conntrack entry orig : dst 10.96.144.12:80 reply : src 10.244.1.2:8080 later packets and replies : no rule walk UDP endpoints changed in a sync kube-proxy deletes stale entries

5.SNAT and the client IP

  • SNAT : masqueradeAll, off-cluster or host sources to a ClusterIP, hairpin, and all external traffic under eTP=Cluster.
  • MASQUERADE uses the egress address. "Local Pod" comes from detectLocalMode (default ClusterCIDR; no clusterCIDR = no-op).
PathBackend sees (observed)
Pod to ClusterIPthe client Pod IP
Pod to its own ServiceSNAT, 10.244.0.1
host to ClusterIPnode IP if remote, 10.244.0.1 if local
external, eTP=Clustera node address, even if local

Test : wget -qO- http://<svc>/clientip (agnhost). eTP=Local : Services.

In short : nftables swaps the rule walk for one map lookup (planned default in v1.40), ipvs is leaving, conntrack makes the first pick stick, and SNAT decides who the backend sees.

1.What kube-proxy reads

  • serving mirrors Pod Ready, terminating comes with deletion, ready = serving and not terminating.
  • Local = nodeName equals kube-proxy's node name, never by IP. A wrong hostnameOverride makes all endpoints remote.
  • 100 endpoints per slice by default (Controllers).
Field or labelEffect
kubernetes.io/service-nameRequired link
addresses[0]The only IP used
conditionsready, serving, terminating
nodeNameLocality
zone, hintsTopology

2.Which endpoints a connection can reach

  • Per policy : Ready endpoints, else serving and terminating ones. This holds for Cluster policy too (docs say Local only).
  • Observed : a Pod with a 120s preStop sleep, scaled to 0, kept taking new connections.
  • kube-proxy uses hints only if every Ready endpoint has one. Local policies win over hints.
  • PreferSameZone has no overload guard : spread replicas (Scheduling). Values : Services.
Local traffic policy? yes no (Cluster) local Ready this node's endpoints Ready node hint, zone hint, or all none none local serving + terminating serving + terminating none none DROP client times out REJECT connection refused

3.Health checks for Local policies

  • Past the eTP panel in Services : one listener per Service on healthCheckNodePort, 200 with local Ready endpoints, else 503, plus header X-Load-Balancing-Endpoint-Weight.
  • Probe : curl -si http://<nodeIP>:<port>/healthz. Terminating local Pods fail it, so the LB drains the node.
  • iTP=Local with no local endpoint DROPs (timeout), not the REJECT the docs imply.

4.eBPF : Services without netfilter

  • Verified programs run at hooks (XDP, tc, cgroup connect4) and keep state in maps.
  • Cilium kubeProxyReplacement: true : socket LB at connect(), NodePorts per packet (snat, dsr, hybrid).
  • Calico eBPF : connect-time LB for TCP only. NodePorts keep the client IP, even with eTP=Cluster.
  • Traps : socket LB breaks Istio sidecars, KubeVirt, Kata, gVisor. tcpdump misses XDP.
in-cluster : socket LB, once per connection connect() to 10.96.0.50:80 cgroup hook connect4 service map pick a backend socket to 10.244.1.7:8080 on the wire no DNAT north-south : per packet external client NodePort or LB IP XDP or tc at the NIC Maglev pick consistent hash SNAT or DSR loadBalancer.mode backend node DSR : replies direct in-cluster north-south connect() to 10.96.0.50:80 cgroup hook connect4 service map pick a backend socket to 10.244.1.7:8080 on the wire no DNAT external client NodePort or LB IP XDP or tc at the NIC Maglev pick consistent hash SNAT or DSR loadBalancer.mode backend node DSR : replies direct
iptablesnftablesipvsCilium KPRCalico eBPF
v1.37defaultGAdeprecatedopt-inopt-in
Dispatchrule walkone mapIPVS tableBPF mapBPF map
Stateconntrackconntrack+ IPVSBPF mapsBPF maps
Kernel-5.13-5.105.10

5.Removing kube-proxy

  • kubeadm : --skip-phases=addon/kube-proxy. Live cluster : delete the DaemonSet and ConfigMap, run kube-proxy --cleanup, set k8sServiceHost.
  • Cilium breaks open Service connections in the switch. Calico keeps TCP flows, drops VXLAN ones, wants ipvs gone first. k3s : --disable-kube-proxy, or --kube-proxy-arg=proxy-mode=nftables.
  • Check : cilium-dbg status --verbose | grep -A12 'KubeProxyReplacement Details'. CNI side : Pod Networking.
In short : Slice conditions and hints pick the reachable Pods, health checks steer LBs to nodes with one, and eBPF can do it all at connect().