Networking internals

Cluster DNS in Depth

1.Two halves

The kubelet writes each Pod's resolver file once, at sandbox creation (GetPodDNS), from clusterDNS, clusterDomain and resolvConf. CoreDNS (Deployment coredns, label k8s-app=kube-dns) answers from API watches behind the Service still named kube-dns : the 10th Service IP on kubeadm (10.96.0.10), 10.43.0.10 on k3s.

Pod (ns shop) getaddrinfo() /etc/resolv.conf nameserver 10.96.0.10, options ndots:5 search shop.svc.cluster.local svc.cluster.local cluster.local lan A, AAAA : one query per suffix 10.96.0.10:53 Service kube-dns kube-proxy DNAT CoreDNS Pod Deployment coredns kubernetes plugin cluster.local, *.arpa cluster names, answered from informers forward . /etc/resolv.conf everything else node's upstream resolvers CoreDNS runs dnsPolicy Default

2.What the kubelet writes

  • ClusterFirst options are exactly ndots:5. Node edns0, timeout, attempts are dropped.
  • hostNetwork : silently Default (kind), use ClusterFirstWithHostNet. No clusterDNS : Default + MissingClusterDNS.
  • dnsConfig is appended, options merged by name (Pod wins). None : dnsConfig alone.
dnsPolicyserversearchopts
ClusterFirstclusterDNS3 cluster + nodendots:5
+ hostNetworknode filenode filenode
...WithHostNetclusterDNS3 cluster + nodendots:5
DefaultresolvConfsame filefile

Over 3 servers, 32 searches or 2048 chars : the API rejects, the kubelet trims the merged file (DNSConfigForming). See it : kubectl exec <pod> -- cat /etc/resolv.conf. Basics : Namespaces.

3.The ndots:5 walk

  • Under 5 dots : all suffixes, then as typed. A 4-dot FQDN too.
  • musl always sends A and AAAA.
  • glibc drops AAAA only with no IPv6 but ::1. Link-local fe80:: counts : 8 queries for 3 suffixes on kind.
  • Plan for 2 x (suffixes + 1) queries.
api.stripe.com.shop.svc.cluster.local NXDOMAIN, A+AAAA api.stripe.com.svc.cluster.local NXDOMAIN, A+AAAA api.stripe.com.cluster.local NXDOMAIN, A+AAAA api.stripe.com.lan NXDOMAIN, A+AAAA api.stripe.com. NOERROR, A+AAAA 10 queries, 8 NXDOMAIN (example names, lab counts)

4.glibc vs musl

Behaviour (seen in the lab)glibcmusl (Alpine)
name with dots >= ndotsas typed, then searchas typed only
NODATA or SERVFAIL on a suffixkeeps walkingstops and fails
. in the search listas typed at that spotskipped
search line over about 255 charsusedwhole line ignored
TCP fallback on truncationyesmusl 1.2.4+ (Alpine 3.18+)

The docs' dnsutils Pod runs musl 1.1.24 (no TCP fallback). dig and nslookup use their own resolver, and dig skips the search list without +search. Test in the app's image : kubectl exec <pod> -- getent ahosts <name>.

5.Cutting the cost

  • Trailing dot : one name, any libc.
  • ndots: "2" : names with 2+ dots go upstream first (seen on kind). musl never searches them.
  • autopath : 1 round trip. Needs pods verified, Pod IP.
  • Searches may hold _ and a lone . (GA v1.34).
Per-Pod tuning (ran on kind)
dnsConfig:
  options:
  - { name: ndots, value: "2" }
  - name: single-request-reopen
  - { name: timeout, value: "2" }
# options timeout:2 ndots:2 single-request-reopen
In short : The kubelet writes a resolv.conf once per Pod, and ndots:5 walks every suffix, often twice per name. glibc and musl differ : test with the app's libc, and cut the walk with trailing dots or lower ndots.

1.Records beyond the basics

Query (kind)Answer
TXT dns-version.<zone>"1.1.0", the spec version
AAAA, IPv4-only ServiceNOERROR, no answer
dig -x 10.96.0.10kube-dns.kube-system.svc.cluster.local.
headless, none readyNXDOMAIN
ExternalNameCNAME + the target's A records

Basic records : Services, StatefulSets.

Per-Pod names
  • hostname + subdomain = headless Service, same ns.
  • Ready Pods only. publishNotReadyAddresses adds unready and terminating ones.
  • setHostnameAsFQDN : 64 chars max. hostnameOverride (GA v1.37) : hostname only, not DNS.

2.pods modes, and a captured name

  • disabled (default) : a-b-c-d.<ns>.pod NXDOMAIN.
  • insecure (kubeadm, k3s) : no Pod check, 1-2-3-4.default.pod gave 1.2.3.4. No *.pod certs.
  • verified : Pod must exist. Watches all Pods (~4x memory, old data).
  • Capture : Service github in namespace com answers github.com (derived). Guard namespaces.

3.A query through CoreDNS

  • Order is compiled (plugin.cfg), not the Corefile's.
  • ttl (plugin 5, kubeadm 30) is also the negative TTL.
  • k3s caches cluster names : new headless names NXDOMAIN up to 30s.
  • Over max_concurrent : REFUSED. SERVFAIL retried only with failover.
query server block .:53 cache kubeadm skips cluster.local hosts k3s NodeHosts kubernetes cluster.local, in-addr, ip6.arpa answer or NXDOMAIN in zone fallthrough, other zone forward . /etc/resolv.conf random policy, max 15 upstreams upstream node's resolvers loadbalance shuffles A / AAAA on the way out. The order is compiled, so moving lines does nothing.

4.The default Corefile

.:53 {                   # kubeadm v1.37.1
  health {               # k3s: plain health
    lameduck 5s
  }
  kubernetes cluster.local in-addr.arpa ip6.arpa {
    pods insecure
    fallthrough in-addr.arpa ip6.arpa
    ttl 30               # k3s: no ttl (5)
  }
  forward . /etc/resolv.conf { max_concurrent 1000 }
  cache 30 {             # k3s: plain cache 30
    disable success cluster.local
    disable denial cluster.local
  }
}  # + errors ready prometheus :9153 loop reload loadbalance
Lifecycle plugins
  • health:8080. lameduck : stays 200, exit delayed 5s
  • ready:8181. 503 until plugins sync, Shutting down in lameduck
  • loopHINFO self-probe for 30s
  • reloadSHA512 check every 30s +/- 15s

The debug page's Corefile fails : 1.14.7 rejects upstream. k3s adds hosts NodeHosts and import *.override. kubeadm : 2 replicas, CoreDNS v1.14.6. k3s v1.37.1+k3s1 : 1 replica, 1.14.7.

5.Changing it safely

  • kubeadm : edit ConfigMap coredns, allow 2 minutes (kind : 59s and 84s).
  • A bad config keeps the old one, but restarts crash. Check : coredns -conf ./Corefile.
  • Now : rollout restart deploy/coredns (kube-system).
  • Stub domain : own block, or forward consul.local first.
k3s : coredns-custom
  • coredns is rewritten every start. *.server adds blocks.
  • A 2nd hosts, cache, errors, prometheus fails. Lameduck health is fine.
kind: ConfigMap  # coredns-custom, kube-system
data:
  example-com.override: |  # inside .:53
    forward example.com 10.0.0.1
In short : CoreDNS answers from API watches in a fixed plugin order. kubeadm and k3s differ on cache, TTL, lameduck and replicas : change the right ConfigMap and allow two minutes.

1.Enough CoreDNS

  • cluster-proportional-autoscaler (not in kubeadm) : max(ceil(cores/256), ceil(nodes/16)) with the docs' values. 40 nodes x 8 cores : 3.
  • Re-reads its ConfigMap every 10s. Tag :v1.11.0, not :1.11.0.
  • Rollouts : 1 replica without lameduck lost 1 of 450 queries per rollout (kind). With lameduck 5s : none.
  • k3s ships 1 replica, no lameduck, no PodDisruptionBudget.

2.NodeLocal DNSCache

  • hostNetwork DaemonSet node-local-dns. Its dummy nodelocaldns holds 169.254.20.10 and the kube-dns IP.
  • 12 rules per IP : 8 raw NOTRACK (6 DNS, 2 health port), 4 filter ACCEPT.
  • External names skip CoreDNS : no stub domains, rewrites or k3s overrides.
  • Create kube-dns-upstream first. An OOM kill leaves the rules : no DNS there.
  • Moved to kubernetes-sigs/node-local-dns. v1.37.1 pins 1.26.8.
Pod UDP to 10.96.0.10:53 raw PREROUTING : NOTRACK no conntrack, so no DNAT node-cache (hostNetwork) nodelocaldns : 169.254.20.10 + 10.96.0.10 kube-dns-upstream cluster.local : cache, then TCP node's resolvers "." : CoreDNS skipped kube-proxy DNAT CoreDNS

3.NodeLocal by dataplane, and DNS egress

ModeHow Pods reach itKubelet changeIf the cache dies
kube-proxy iptablesNOTRACK on the bound kube-dns IPnoneclean exit falls back
ipvs (deprecated)Pods use 169.254.20.10clusterDNS: [169.254.20.10]no fallback
nftablesundocumented; iptables install worked (1 kind node)nonefell back on clean exit
CiliumCiliumLocalRedirectPolicy on kube-dnsnonekube-dns backends
Default-deny egress : packets keep the kube-dns ClusterIP, so allowing CoreDNS Pods (or 169.254.20.10/32) stopped matching on kindnet. ipBlock: 10.96.0.10/32 fixed it. Test your CNI. See Network Policy.

4.Two failures up close

  • 5s race : parallel A and AAAA both pass DNAT unconfirmed, one is dropped at confirm.
  • Fixed by kernels v4.19 to v5.9. IPVS is hit too. See conntrack -S insert_failed. Basics : Service Dataplane.
  • Loop : node 127.0.0.53 makes CoreDNS forward to itself. kubeadm fixes resolvConf only if unset (leaked into kubelet-config before v1.37.1).
node resolv.conf nameserver 127.0.0.53 kubelet resolvConf copied into Default Pods CoreDNS dnsPolicy Default forward . /etc/resolv.conf 127.0.0.53 inside CoreDNS's netns loop plugin : exit CrashLoopBackOff

5.Other failures, and the playbook

SymptomCauseFix
External SERVFAIL, or REFUSED under loadupstreams down, or max_concurrentfix upstreams; scale
New headless name NXDOMAIN for ~30scluster domain cacheddisable success/denial
Cluster names SERVFAILno EndpointSlice RBAC, or first 5sdescribe clusterrole system:coredns
One node has no DNS after a NodeLocal OOMrules left behindraise its memory limit

Playbook : Pod resolv.conf, events, getent ahosts, CoreDNS Pods, EndpointSlice and logs, system:coredns, log on (2 min), metrics, conntrack -S.

Metrics : coredns_dns_responses_total{rcode}, coredns_proxy_request_duration_seconds{proxy_name="forward"} (old coredns_forward_request* gone since 1.11), hit ratio = cache_hits_total / cache_requests_total.

In short : Scale CoreDNS with replicas and lameduck, add NodeLocal DNSCache to skip conntrack (and fix DNS egress policies), and debug from the Pod outward with the app's libc.