Scheduling
Advanced Scheduling
1.What does kube-scheduler do ?
It gives a node to each Pod with no spec.nodeName.
- Filter : drop each node that breaks a hard rule.
- Score : rank the rest on soft rules. The highest total wins, ties at random.
- Bind : write
spec.nodeNamethrough the API server. - 0 nodes fit : try preemption, else Pending. The events name the failed rule.
Requests, not usage
- A Pod fits if its request fits the node's allocatable minus what is already requested.
- Effective request : the larger of the summed app + sidecar containers and the biggest init container, plus RuntimeClass overhead.
- Usage and limits are ignored. No requests = fits anywhere. QoS : Quick Revision.
2.One Pod through the scheduler
The scheduling framework (stable v1.19) chains plugin hooks.
- Scheduling cycle : serial, one Pod at a time, from PreFilter to Permit.
- PostFilter runs only when 0 nodes fit. Preemption lives there.
- Binding cycle : PreBind, Bind, PostBind. May overlap the next Pod.
- A failure at Reserve or later runs Unreserve and sends the Pod back to a queue.
- backoffQ : 1 s first, 10 s max. QueueingHints (GA v1.34) requeue only on an event that might help, like a new node.
3.Hard rules filter, soft rules score
| Control | Hard (Filter) | Soft (Score) | Plugin, weight |
|---|---|---|---|
| nodeSelector | every label must match | - | NodeAffinity, 2 |
| Node affinity | required... | preferred..., weight 1 to 100 | NodeAffinity, 2 |
| Pod (anti-)affinity | required..., with a topologyKey | preferred..., weight 1 to 100 | InterPodAffinity, 2 |
| Taints | NoSchedule, NoExecute | PreferNoSchedule | TaintToleration, 3 |
| Topology spread | DoNotSchedule (default) | ScheduleAnyway | PodTopologySpread, 2 |
| Requests | must fit free allocatable | LeastAllocated (default) | NodeResourcesFit, 1 |
| Priority | no filter : queue order, preemption | - | PrioritySort, DefaultPreemption |
| spec.nodeName | skips the scheduler | - | the kubelet admits or rejects it |
Also scoring : BalancedAllocation 1, ImageLocality 1. A soft rule only adds to a weighted sum, so it can lose.
4.Node affinity
- nodeSelector : the node must carry every listed label.
- requiredDuringScheduling... is hard. preferred... is soft, it adds its weight.
- Terms are ORed, matchExpressions in a term ANDed. With nodeSelector too, both must hold.
- Operators : In, NotIn, Exists, DoesNotExist, Gt, Lt (integer labels).
- IgnoredDuringExecution : relabeling a node never moves a running Pod.
5.Pod affinity and anti-affinity
- Matches labels of Pods already running in a domain.
topologyKeynames it and is required. kubernetes.= per node,io/ hostname topology.= per zone.kubernetes. io/ zone - Own namespace only, unless namespaces or namespaceSelector widen it.
- Costly. Not recommended above several hundred nodes.
- Trap : required anti-affinity on hostname with more replicas than nodes leaves the extras Pending.
In short : kube-scheduler filters nodes on hard rules, scores the rest on soft rules, and binds the winner. It counts requests, not usage.
1.Taints and tolerations
- A taint on a node repels. A toleration permits, it never attracts.
- NoSchedule blocks new Pods, PreferNoSchedule discourages, NoExecute also evicts.
- tolerationSeconds : time allowed under NoExecute. Unset = forever.
- operator : Equal (default) or Exists. Exists with no key matches all.
- Dedicated nodes : taint and label them. Pods get the toleration and required affinity on the label.
| Built-in taint | Effect | Set when |
|---|---|---|
| not-ready | NoExecute | Ready is False |
| unreachable | NoExecute | Ready is Unknown |
| memory-pressure | NoSchedule | MemoryPressure |
| disk-, pid-pressure | NoSchedule | DiskPressure, PIDPressure |
| network-unavailable | NoSchedule | network not set up |
| unschedulable | NoSchedule | kubectl cordon, drain |
| out-of-service | NoExecute | by hand, dead node |
| control-plane | NoSchedule | kubeadm control plane |
Keys start with node.kubernetes.io/, control-plane with node-role.kubernetes.io/. Pods tolerate not-ready and unreachable for 300 s, so they leave a dead node after about 5 min.
2.Topology spread
- maxSkew (> 0) : allowed gap to the least-loaded domain. topologyKey : the node label that makes domains.
- whenUnsatisfiable : DoNotSchedule (default) or ScheduleAnyway.
- labelSelector : which Pods count. Match the Pod's own labels.
- None set ? Soft defaults : hostname skew 3, zone skew 5. Only checked at scheduling time.
3.Priority and preemption
- PriorityClass (cluster-scoped) : yours go up to 1000000000. One may be globalDefault, else 0.
- Admission writes
spec.priority. Higher = earlier in activeQ. preemptionPolicy Never : jumps the queue, never evicts. - Preemption : delete lower-priority Pods on one node to make room, set
status.(a hint).nominatedNodeName - Best effort : PDBs may break, QoS ignored, preemption never frees DRA devices.
4.Skip, hold or swap it
- spec.nodeName : no filters, NoSchedule ignored. Pin with nodeSelector on kubernetes.io/hostname instead.
- schedulingGates (GA v1.30) : the Pod waits at SchedulingGated until every gate is gone.
- Profiles : several schedulerNames in one kube-scheduler. Pods pick with
spec..schedulerName - Descheduler : evicts Pods that now break a rule, kube-scheduler places them again.
- DRA (GA v1.34) : a ResourceClaim asks for a GPU. Device and node are picked together.
5.Why is it Pending ?
FailedScheduling : 0/N nodes are available: <reason>, ...
| Reason says | Fix |
|---|---|
| Insufficient cpu / memory | lower requests, add nodes |
| untolerated taint(s) | unnamed since v1.35 : describe node |
| node affinity/selector | label nodes or fix the rule |
| pod anti-affinity rules | preferred, or spread instead |
| topology spread constraints | raise maxSkew, add domains |
| were unschedulable | kubectl uncordon |
| unbound immediate PVCs | fix the PVC (Storage) |
| SchedulingGated (status, no event) | the gate owner removes it |
6.Commands
| Command | Shows |
|---|---|
kubectl events --for pod/<pod> --types=Warning | That Pod's warnings |
kubectl get events -A --field-selector reason=FailedScheduling | All scheduling failures |
kubectl describe node <node> | Taints, Allocated resources |
kubectl get nodes -L topology.kubernetes.io/zone | Zone as a column |
kubectl taint nodes <node> k=v:NoSchedule | Add a taint, a trailing - removes it |
kubectl get priorityclasses | VALUE, GLOBAL-DEFAULT |
In short : Taints repel, affinity attracts, spread balances, and priority decides who waits. A Pending Pod's FailedScheduling event names the rule to fix.