Core Kubernetes · 12

Nodes

1.What is a Node ?

A Node is a cluster-scoped v1 object that stands for one machine. You rarely write it : the kubelet creates its own Node when it starts, then keeps its status fresh.

  • On the machine : the kubelet, a container runtime and kube-proxy (see Architecture).
  • The name is unique in the cluster. Delete a rebuilt machine's old Node before it joins again.
  • The object is a record, not the machine : deleting it stops nothing.
  • The kubelet signs in as system:node:<name>, group system:nodes. Ready stays False until the runtime answers and the network plugin is up.
machine node-3 kubelet registerNode true by default API server Node node-3 labels, capacity Lease node-3 kube-node-lease status Ready True 1. creates it 2. renews, 10s 3. posts, updates Scheduler can place Pods here Nodes, by watch

Cloud : the cloud-controller-manager adds providerID, zone, region and instance type, and deletes the Node once the instance is gone. Karpenter and Cluster API make nodes on demand.

2.What the object holds

FieldHolds
spec.podCIDRthis node's Pod address range, if the cluster gives one per node
spec.taints, unschedulablewho may land here
status.addressesHostname, InternalIP, ExternalIP, InternalDNS, ExternalDNS
status.conditionsReady and the pressure flags
status.nodeInfokernel, OS image, runtime and kubelet versions
status.imagesup to 50 images on the node

3.Capacity vs allocatable

capacity : all the machine has kubeReserved systemReserved eviction allocatable both empty by default mem 100Mi for Pod requests
  • Stock kubelet : allocatable memory is capacity minus 100Mi, CPU is all of it. Managed services reserve more.
  • The scheduler adds up requests against allocatable, never usage.
  • pods is 110 by default (maxPods). Cloud CNIs often change it.

4.Labels and roles

LabelSet by
kubernetes.io/hostname, /os, /archkubelet
topology.kubernetes.io/zone, /regioncloud, or you
node.kubernetes.io/instance-typecloud
node-role.kubernetes.io/<role>kubeadm, or you

Old guides say node-role.kubernetes.io/master : it is control-plane now.

ROLES is only labels
  • The column lists the <role> part of each node-role label, or <none>. The value is ignored.
  • kubeadm sets control-plane. Any other is yours : kubectl label node n3 node-role.kubernetes.io/worker=
  • A kubelet cannot give itself a role : NodeRestriction forbids it.

5.Reading kubectl describe node

Part, in print orderWhat to read there
Roles, Labels, Taints, Unschedulablewhich Pods may land here, and whether it is cordoned
Lease, Conditionsis it alive : each condition has a reason and a message. A healthy Ready : kubelet is posting ready status
Addresses, Capacity, Allocatable, System Infowhere it is, how big, which kernel, runtime and kubelet
Non-terminated Pods, Allocated resourcesevery Pod here, then the sum of their requests and limits. Not usage : a node can be 95% allocated and idle (for usage, kubectl top node needs metrics-server)
Eventsrecent history, such as RegisteredNode and NodeReady
In short : A Node is the kubelet's own record of its machine. Pods are scheduled against allocatable, which is capacity minus reservations and the eviction threshold.

1.Two heartbeats

HeartbeatHow often
Lease renewTimeevery 10s
Node statuson a change, else every 5 min
  • One Lease per node in kube-node-lease, so a healthy node need not rewrite a large status every 10s.
  • No change for 50s (40s before v1.32) : the node lifecycle controller calls the node late.

2.NotReady means two things

Ready FalseReady Unknown
Written bythe kubeletthe controller
Meanshere, but cannot run Pods (runtime down, no CNI)nobody has heard from the kubelet
Taintnot-readyunreachable
get nodesNotReadyNotReady

Other conditions : MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable. Reason and message : describe node.

3.A node dies : the timeline

0s : last Lease renewal then the kubelet stops : crash, power, network cut 50s grace period : node lifecycle controller about 50s (lab : 52s) : Ready Unknown taints unreachable:NoSchedule and :NoExecute Pods Ready False, yet still listed 1/1 Running each Pod's own 300s toleration : taint-eviction controller about 350s (lab : 353s) : Pods deleted 1/1 Terminating : no kubelet to finish the job a Deployment makes the replacement at once
  • No controller setting moves the Pods. The 300s is tolerationSeconds: 300, added at admission to each Pod without its own. It starts when the taint lands.
  • The containers may still run on the lost machine. Kubelet back : Ready turns True, both taints go by themselves, deleted Pods' containers are killed.
  • Faster failover : give the Pods their own unreachable and not-ready tolerations with a smaller tolerationSeconds. Lab, with 20s : deleted 73s after the last heartbeat.
At 350sDeploymentStatefulSetDaemonSet
its Pod isreplaced at once : a Pod marked for deletion no longer countsstuck Terminating, not replaced : its name is still takennever evicted : it tolerates both taints with no time limit

4.The node protects itself

  • Node-pressure eviction : past a threshold the kubelet sets a condition, the controller adds a NoSchedule taint, and the kubelet evicts Pods. At once, no PodDisruptionBudget asked.
  • Memory : Pods using more than they requested go first, then by priority. No QoS class is safe, only critical Pods are spared.
  • The evicted Pod stays listed : phase Failed, reason Evicted, STATUS often Error.
SignalEvict underCondition
memory.available100MiMemoryPressure
nodefs.available10%DiskPressure
nodefs.inodesFree5%
imagefs.available15%
imagefs.inodesFree5%
pid.availableno defaultPIDPressure

Hard defaults on Linux. Setting one key in evictionHard drops the others. The condition stays True 5 min after the pressure ends.

API eviction (drain)Taint evictionNode-pressureOOM kill
Whoyou, through the API servertaint-eviction controllerkubeletLinux kernel
Budgets (PDB)honouredignoredignorednot involved
Pod afterwardsdeleteddead node : Terminating till it returnsFailed, Evicted, stays listedsame Pod, container restarted

5.Shutdown

  • Graceful node shutdown is on and still does nothing : shutdownGracePeriod and shutdownGracePeriodCriticalPods are both 0. Set them (say 30s and 10s) and the kubelet holds the systemd shutdown to stop ordinary Pods, then critical ones.
Power lost : the out-of-service taint kubectl taint nodes n3 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute gets its Pods deleted, stuck and DaemonSet ones too, and volumes detached at once. Only when the node is really off : if not, old containers run beside the new. Remove the taint yourself.
In short : A node is alive while its Lease is renewed. After 50s of silence it is tainted, and each Pod's own 300s toleration decides when it is deleted. Deployments replace, StatefulSet Pods wait.

1.Cordon, drain, uncordon

  • cordon sets spec.unschedulable. Running Pods stay, DaemonSet Pods still land.
  • drain cordons, then evicts each Pod through the Eviction API, which asks its PodDisruptionBudget. A refusal is retried every 5s, for ever unless --timeout is set.
  • uncordon clears the flag, no Pod moves back. Drain never stops the kubelet or deletes the Node.
1. cordon no new Pods 2. list the Pods static ones skipped 3. Eviction one per Pod refused : nothing is evicted DaemonSet, emptyDir or bare Pod budget allows ? yes Pod deleted own grace period no HTTP 429 try again every 5s

2.What drain refuses

Pod on the nodedrain saysFlag, and what it costs
DaemonSet Podcannot delete DaemonSet-managed Pods--ignore-daemonsets : leaves them running
uses an emptyDircannot delete Pods with local storage--delete-emptydir-data : that data is lost
no controllercannot delete Pods that declare no controller--force : the Pod is gone for good
budget would breakCannot evict pod as it would violate the pod's disruption budget.fix the budget or add replicas (--disable-eviction ignores budgets)

The first three stop the drain before anything is evicted. One replica with minAvailable: 1 blocks it for ever.

3.Labels, nodeSelector, a taint

$ kubectl label node n3 pool=gpu
$ kubectl taint nodes n3 dedicated=gpu:NoSchedule
nodeSelector: {pool: gpu}   # Pod template
tolerations:
- {key: dedicated, value: gpu, effect: NoSchedule}

Label : Pods in. Taint : others out (Advanced Scheduling).

4.Removing a node

  • Drain, kubectl delete node, then kubeadm reset on the machine.
  • delete node removes only the object. Its Pods go about a minute later, no budget asked.
  • A running kubelet does not register again. After a restart it is back in seconds, without hand-added labels, taints or the cordon.

5.Version skew and upgrades

  • A kubelet may be up to three minors older than the API server, never newer. Control plane first.
  • Then node by node : new kubeadm, kubeadm upgrade node, drain, new kubelet and restart, uncordon.
What a kubelet may do Only its own Node and Pods : NodeRestriction and the Node authorizer see to it. But get on nodes/proxy reaches the kubelet API, exec included : treat it as admin (RBAC).

6.Why is it NotReady ?

describe node showsCauseLook at, then fix
Unknown, "Kubelet stopped posting node status."kubelet, machine or network downjournalctl -u kubelet : start it
False, "cni plugin not initialized"no CNI plugin yetls /etc/cni/net.d : install the CNI
False, "container runtime is down"containerd stoppedcrictl info : restart the runtime
False, "PLEG is not healthy"runtime too slow to list containersnode load, disk I/O : restart the runtime

7.Commands

CommandWhat it shows or does
kubectl get lease -n kube-node-leaseOne Lease per node : the heartbeat
kubectl get pods -A -o wide --field-selector spec.nodeName=n3Every Pod on one node
kubectl drain n3 --ignore-daemonsets --delete-emptydir-dataCordon and evict. Then kubectl uncordon n3
kubectl debug node/n3 -it --image=busyboxA Pod with the host root at /host. Delete it after
In short : Cordon blocks new Pods, drain evicts through budgets, and deleting the Node removes only the record. Upgrade the control plane first, then one drained node at a time.