Networking internals
Cluster DNS in Depth
1.Two halves
The kubelet writes each Pod's resolver file once, at sandbox creation (GetPodDNS), from clusterDNS, clusterDomain and resolvConf. CoreDNS (Deployment coredns, label k8s-app=kube-dns) answers from API watches behind the Service still named kube-dns : the 10th Service IP on kubeadm (10.96.0.10), 10.43.0.10 on k3s.
2.What the kubelet writes
- ClusterFirst options are exactly
ndots:5. Nodeedns0,timeout,attemptsare dropped. - hostNetwork : silently Default (kind), use ClusterFirstWithHostNet. No
clusterDNS: Default +MissingClusterDNS. dnsConfigis appended, options merged by name (Pod wins).None:dnsConfigalone.
| dnsPolicy | server | search | opts |
|---|---|---|---|
| ClusterFirst | clusterDNS | 3 cluster + node | ndots:5 |
| + hostNetwork | node file | node file | node |
| ...WithHostNet | clusterDNS | 3 cluster + node | ndots:5 |
| Default | resolvConf | same file | file |
Over 3 servers, 32 searches or 2048 chars : the API rejects, the kubelet trims the merged file (DNSConfigForming). See it : kubectl exec <pod> -- cat /etc/resolv.conf. Basics : Namespaces.
3.The ndots:5 walk
- Under 5 dots : all suffixes, then as typed. A 4-dot FQDN too.
- musl always sends A and AAAA.
- glibc drops AAAA only with no IPv6 but
::1. Link-localfe80::counts : 8 queries for 3 suffixes on kind. - Plan for 2 x (suffixes + 1) queries.
4.glibc vs musl
| Behaviour (seen in the lab) | glibc | musl (Alpine) |
|---|---|---|
| name with dots >= ndots | as typed, then search | as typed only |
| NODATA or SERVFAIL on a suffix | keeps walking | stops and fails |
. in the search list | as typed at that spot | skipped |
| search line over about 255 chars | used | whole line ignored |
| TCP fallback on truncation | yes | musl 1.2.4+ (Alpine 3.18+) |
The docs' dnsutils Pod runs musl 1.1.24 (no TCP fallback). dig and nslookup use their own resolver, and dig skips the search list without +search. Test in the app's image : kubectl exec <pod> -- getent ahosts <name>.
5.Cutting the cost
- Trailing dot : one name, any libc.
- ndots: "2" : names with 2+ dots go upstream first (seen on kind). musl never searches them.
- autopath : 1 round trip. Needs
pods verified, Pod IP. - Searches may hold
_and a lone.(GA v1.34).
dnsConfig:
options:
- { name: ndots, value: "2" }
- name: single-request-reopen
- { name: timeout, value: "2" }
# options timeout:2 ndots:2 single-request-reopen
1.Records beyond the basics
| Query (kind) | Answer |
|---|---|
TXT dns-version.<zone> | "1.1.0", the spec version |
| AAAA, IPv4-only Service | NOERROR, no answer |
dig -x 10.96.0.10 | kube-dns. |
| headless, none ready | NXDOMAIN |
| ExternalName | CNAME + the target's A records |
Basic records : Services, StatefulSets.
hostname+subdomain= headless Service, same ns.- Ready Pods only.
publishNotReadyAddressesadds unready and terminating ones. setHostnameAsFQDN: 64 chars max.hostnameOverride(GA v1.37) : hostname only, not DNS.
2.pods modes, and a captured name
disabled(default) :a-b-c-d.<ns>.NXDOMAIN.pod insecure(kubeadm, k3s) : no Pod check,1-2-3-4.gave 1.2.3.4. Nodefault. pod *.podcerts.
verified: Pod must exist. Watches all Pods (~4x memory, old data).- Capture : Service
githubin namespacecomanswersgithub.com(derived). Guard namespaces.
3.A query through CoreDNS
- Order is compiled (
plugin.cfg), not the Corefile's. ttl(plugin 5, kubeadm 30) is also the negative TTL.- k3s caches cluster names : new headless names NXDOMAIN up to 30s.
- Over
max_concurrent: REFUSED. SERVFAIL retried only withfailover.
4.The default Corefile
.:53 { # kubeadm v1.37.1
health { # k3s: plain health
lameduck 5s
}
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30 # k3s: no ttl (5)
}
forward . /etc/resolv.conf { max_concurrent 1000 }
cache 30 { # k3s: plain cache 30
disable success cluster.local
disable denial cluster.local
}
} # + errors ready prometheus :9153 loop reload loadbalance
- health:8080. lameduck : stays 200, exit delayed 5s
- ready:8181. 503 until plugins sync,
Shutting downin lameduck - loopHINFO self-probe for 30s
- reloadSHA512 check every 30s +/- 15s
The debug page's Corefile fails : 1.14.7 rejects upstream. k3s adds hosts NodeHosts and import *.override. kubeadm : 2 replicas, CoreDNS v1.14.6. k3s v1.37.1+k3s1 : 1 replica, 1.14.7.
5.Changing it safely
- kubeadm : edit ConfigMap
coredns, allow 2 minutes (kind : 59s and 84s). - A bad config keeps the old one, but restarts crash. Check :
coredns -conf ./Corefile. - Now :
rollout restart deploy/coredns(kube-system). - Stub domain : own block, or
forward consul.localfirst.
corednsis rewritten every start.*.serveradds blocks.- A 2nd
hosts,cache,errors,prometheusfails. Lameduckhealthis fine.
kind: ConfigMap # coredns-custom, kube-system data: example-com.override: | # inside .:53 forward example.com 10.0.0.1
1.Enough CoreDNS
- cluster-proportional-autoscaler (not in kubeadm) :
max(with the docs' values. 40 nodes x 8 cores : 3.ceil( cores/ 256), ceil(nodes/16)) - Re-reads its ConfigMap every 10s. Tag
:v1.11.0, not:1.11.0.
- Rollouts : 1 replica without lameduck lost 1 of 450 queries per rollout (kind). With
lameduck 5s: none. - k3s ships 1 replica, no lameduck, no PodDisruptionBudget.
2.NodeLocal DNSCache
- hostNetwork DaemonSet
node-local-dns. Its dummynodelocaldnsholds 169.254.20.10 and the kube-dns IP. - 12 rules per IP : 8 raw NOTRACK (6 DNS, 2 health port), 4 filter ACCEPT.
- External names skip CoreDNS : no stub domains, rewrites or k3s overrides.
- Create
kube-dns-upstreamfirst. An OOM kill leaves the rules : no DNS there. - Moved to
kubernetes-sigs/. v1.37.1 pins 1.26.8.node-local-dns
3.NodeLocal by dataplane, and DNS egress
| Mode | How Pods reach it | Kubelet change | If the cache dies |
|---|---|---|---|
| kube-proxy iptables | NOTRACK on the bound kube-dns IP | none | clean exit falls back |
| ipvs (deprecated) | Pods use 169.254.20.10 | clusterDNS: [169.254.20.10] | no fallback |
| nftables | undocumented; iptables install worked (1 kind node) | none | fell back on clean exit |
| Cilium | CiliumLocalRedirectPolicy on kube-dns | none | kube-dns backends |
169.254.20.10/32) stopped matching on kindnet. ipBlock: 10.96.0.10/32 fixed it. Test your CNI. See Network Policy.4.Two failures up close
- 5s race : parallel A and AAAA both pass DNAT unconfirmed, one is dropped at confirm.
- Fixed by kernels v4.19 to v5.9. IPVS is hit too. See
conntrack -Sinsert_failed. Basics : Service Dataplane. - Loop : node
127.0.0.53makes CoreDNS forward to itself. kubeadm fixesresolvConfonly if unset (leaked intokubelet-configbefore v1.37.1).
5.Other failures, and the playbook
| Symptom | Cause | Fix |
|---|---|---|
| External SERVFAIL, or REFUSED under load | upstreams down, or max_concurrent | fix upstreams; scale |
| New headless name NXDOMAIN for ~30s | cluster domain cached | disable success/denial |
| Cluster names SERVFAIL | no EndpointSlice RBAC, or first 5s | describe clusterrole system:coredns |
| One node has no DNS after a NodeLocal OOM | rules left behind | raise its memory limit |
Playbook : Pod resolv.conf, events, getent ahosts, CoreDNS Pods, EndpointSlice and logs, system:coredns, log on (2 min), metrics, conntrack -S.
Metrics : coredns_, coredns_ (old coredns_ gone since 1.11), hit ratio = cache_hits_total / cache_.