Scheduling

Advanced Scheduling

1.What does kube-scheduler do ?

It gives a node to each Pod with no spec.nodeName.

  • Filter : drop each node that breaks a hard rule.
  • Score : rank the rest on soft rules. The highest total wins, ties at random.
  • Bind : write spec.nodeName through the API server.
  • 0 nodes fit : try preemption, else Pending. The events name the failed rule.
Requests, not usage
  • A Pod fits if its request fits the node's allocatable minus what is already requested.
  • Effective request : the larger of the summed app + sidecar containers and the biggest init container, plus RuntimeClass overhead.
  • Usage and limits are ignored. No requests = fits anywhere. QoS : Quick Revision.

2.One Pod through the scheduler

The scheduling framework (stable v1.19) chains plugin hooks.

  • Scheduling cycle : serial, one Pod at a time, from PreFilter to Permit.
  • PostFilter runs only when 0 nodes fit. Preemption lives there.
  • Binding cycle : PreBind, Bind, PostBind. May overlap the next Pod.
  • A failure at Reserve or later runs Unreserve and sends the Pod back to a queue.
  • backoffQ : 1 s first, 10 s max. QueueingHints (GA v1.34) requeue only on an event that might help, like a new node.
New Podspec.nodeName empty PreEnqueuescheduling gates activeQhighest priority first PreFilter + Filterdrop infeasible nodes PreScore + Scoresoft rules, weighted Reserve + Permitapprove or deny PostFilterpreemption 0 fit backoffQor unschedulable pool denied hint matched or backoff over PreBind + Bindbinding cycle, async ok BoundScheduled event nodeName written

3.Hard rules filter, soft rules score

ControlHard (Filter)Soft (Score)Plugin, weight
nodeSelectorevery label must match-NodeAffinity, 2
Node affinityrequired...preferred..., weight 1 to 100NodeAffinity, 2
Pod (anti-)affinityrequired..., with a topologyKeypreferred..., weight 1 to 100InterPodAffinity, 2
TaintsNoSchedule, NoExecutePreferNoScheduleTaintToleration, 3
Topology spreadDoNotSchedule (default)ScheduleAnywayPodTopologySpread, 2
Requestsmust fit free allocatableLeastAllocated (default)NodeResourcesFit, 1
Priorityno filter : queue order, preemption-PrioritySort, DefaultPreemption
spec.nodeNameskips the scheduler-the kubelet admits or rejects it

Also scoring : BalancedAllocation 1, ImageLocality 1. A soft rule only adds to a weighted sum, so it can lose.

4.Node affinity

  • nodeSelector : the node must carry every listed label.
  • requiredDuringScheduling... is hard. preferred... is soft, it adds its weight.
  • Terms are ORed, matchExpressions in a term ANDed. With nodeSelector too, both must hold.
  • Operators : In, NotIn, Exists, DoesNotExist, Gt, Lt (integer labels).
  • IgnoredDuringExecution : relabeling a node never moves a running Pod.

5.Pod affinity and anti-affinity

  • Matches labels of Pods already running in a domain. topologyKey names it and is required.
  • kubernetes.io/hostname = per node, topology.kubernetes.io/zone = per zone.
  • Own namespace only, unless namespaces or namespaceSelector widen it.
  • Costly. Not recommended above several hundred nodes.
  • Trap : required anti-affinity on hostname with more replicas than nodes leaves the extras Pending.
In short : kube-scheduler filters nodes on hard rules, scores the rest on soft rules, and binds the winner. It counts requests, not usage.

1.Taints and tolerations

  • A taint on a node repels. A toleration permits, it never attracts.
  • NoSchedule blocks new Pods, PreferNoSchedule discourages, NoExecute also evicts.
  • tolerationSeconds : time allowed under NoExecute. Unset = forever.
  • operator : Equal (default) or Exists. Exists with no key matches all.
  • Dedicated nodes : taint and label them. Pods get the toleration and required affinity on the label.
Built-in taintEffectSet when
not-readyNoExecuteReady is False
unreachableNoExecuteReady is Unknown
memory-pressureNoScheduleMemoryPressure
disk-, pid-pressureNoScheduleDiskPressure, PIDPressure
network-unavailableNoSchedulenetwork not set up
unschedulableNoSchedulekubectl cordon, drain
out-of-serviceNoExecuteby hand, dead node
control-planeNoSchedulekubeadm control plane

Keys start with node.kubernetes.io/, control-plane with node-role.kubernetes.io/. Pods tolerate not-ready and unreachable for 300 s, so they leave a dead node after about 5 min.

2.Topology spread

  • maxSkew (> 0) : allowed gap to the least-loaded domain. topologyKey : the node label that makes domains.
  • whenUnsatisfiable : DoNotSchedule (default) or ScheduleAnyway.
  • labelSelector : which Pods count. Match the Pod's own labels.
  • None set ? Soft defaults : hostname skew 3, zone skew 5. Only checked at scheduling time.
new Pod, app=web maxSkew 1 DoNotSchedule zone a zone b zone c 3 - 1 = 2 > 1 blocked 3 - 1 = 2 > 1 blocked 2 - 2 = 0 placed here skew = Pods in the domain - fewest in any domain

3.Priority and preemption

  • PriorityClass (cluster-scoped) : yours go up to 1000000000. One may be globalDefault, else 0.
  • Admission writes spec.priority. Higher = earlier in activeQ. preemptionPolicy Never : jumps the queue, never evicts.
  • Preemption : delete lower-priority Pods on one node to make room, set status.nominatedNodeName (a hint).
  • Best effort : PDBs may break, QoS ignored, preemption never frees DRA devices.

4.Skip, hold or swap it

  • spec.nodeName : no filters, NoSchedule ignored. Pin with nodeSelector on kubernetes.io/hostname instead.
  • schedulingGates (GA v1.30) : the Pod waits at SchedulingGated until every gate is gone.
  • Profiles : several schedulerNames in one kube-scheduler. Pods pick with spec.schedulerName.
  • Descheduler : evicts Pods that now break a rule, kube-scheduler places them again.
  • DRA (GA v1.34) : a ResourceClaim asks for a GPU. Device and node are picked together.

5.Why is it Pending ?

FailedScheduling : 0/N nodes are available: <reason>, ...

Reason saysFix
Insufficient cpu / memorylower requests, add nodes
untolerated taint(s)unnamed since v1.35 : describe node
node affinity/selectorlabel nodes or fix the rule
pod anti-affinity rulespreferred, or spread instead
topology spread constraintsraise maxSkew, add domains
were unschedulablekubectl uncordon
unbound immediate PVCsfix the PVC (Storage)
SchedulingGated (status, no event)the gate owner removes it

6.Commands

CommandShows
kubectl events --for pod/<pod> --types=WarningThat Pod's warnings
kubectl get events -A --field-selector reason=FailedSchedulingAll scheduling failures
kubectl describe node <node>Taints, Allocated resources
kubectl get nodes -L topology.kubernetes.io/zoneZone as a column
kubectl taint nodes <node> k=v:NoScheduleAdd a taint, a trailing - removes it
kubectl get priorityclassesVALUE, GLOBAL-DEFAULT
In short : Taints repel, affinity attracts, spread balances, and priority decides who waits. A Pending Pod's FailedScheduling event names the rule to fix.