Core Kubernetes · 13

Quick Revision

1.Object map

dashed box = cluster-wide Ingress or Gateway HTTP(S) Service Pod IPs EndpointSlice StorageClass Deployment per template ReplicaSet keeps N Pod scheduled Node selects ConfigMap Secret PVC the claim binds PV the storage uses provisions

apiVersion : v1 for most. apps/v1, discovery.k8s.io/v1, storage.k8s.io/v1, networking.k8s.io/v1. Ingress is GA but frozen, Gateway API is the way forward. Checked for Kubernetes v1.37 (supported : 1.37, 1.36, 1.35).

Ch.ObjectDoes what
02NodeMachine that runs Pods
03PodContainers, one IP, volumes
04DeploymentRolling updates over ReplicaSets
04ReplicaSetKeeps N Pods running
05ServiceStable name + IP for its Pods
05EndpointSliceThe live Pod IPs
06ConfigMapPlain settings, 1 MiB max
06SecretSensitive values, 1 MiB max
07PVC / PVThe claim / the disk
07StorageClassHow new disks get made
08NamespaceName scope, quotas, defaults
08IngressHTTP routes in (frozen)
add-onGateway APINewer routing, as CRDs

2.Triage : STATUS, then where to look

1. get podsSTATUS, RESTARTS
2. describe podEvents, Last State
3. logs --previousthe run that crashed
4. events--for pod/<pod>
5. debugno shell in the image
STATUSWhat it meansRunUsual fixCh.
PendingFailedScheduling : requests fit no node, an untolerated taint, an unbound PVC, a hostPort clashkubectl describe pod <pod>
kubectl get events --field-selector reason=FailedScheduling
Lower requests, add nodes, tolerate, fix the PVC03, 02
ImagePullBackOffBad name or tag, image not pushed, or no imagePullSecretkubectl describe pod <pod>Fix the ref, push it, add a pull Secret03
CrashLoopBackOffKeeps exiting. Waits 10 s, 20 s, 40 s ... up to 300 s between restartskubectl logs <pod> --previousFix the app, config or probe timing03
OOMKilledOver the memory limit, Exit Code 137. RESTARTS climbskubectl describe pod <pod>
kubectl top pod <pod> --containers
Raise the limit, or fix the leak03
Read the exit code
  • 0Exited cleanly
  • 1The app reported an error
  • 137128 + 9, SIGKILL : OOM or killed
  • 143128 + 15, SIGTERM : asked to stop
STATUS is not the phase
  • STATUS shows a container reason when there is one, else status.phase.
  • ImagePullBackOff : phase Pending. CrashLoopBackOff : usually Running.

3.Commands

LookWhat it is for
kubectl get pods -o wideSTATUS, node, Pod IP. 03
kubectl describe pod <pod>Events, Last State. 03
kubectl logs <pod> --previousThe run that crashed. 03
kubectl events --for pod/<pod> --types=WarningIts warnings only, the same ones describe lists. 09
kubectl top pod <pod> --containersLive CPU and memory. Needs Metrics Server
kubectl debug -it <pod> --image=busybox:1.28 --target=<c>No shell in the image ? Add an ephemeral one. 03
ChangeWhat it is for
kubectl apply -f <file>Create or update. 01
kubectl diff -f <file>Preview an apply. 01
kubectl exec -it <pod> -- shShell in a container, via the kubelet. 02
kubectl port-forward svc/<svc> 8080:80localhost:8080 to one Pod behind the Service. 02
kubectl rollout status / undo deploy/<d>Wait for a rollout, or go back one revision. 09
kubectl scale deploy/<d> --replicas=3Set replicas by hand. An HPA overrides it. 04
In short : Each object does one job and points at the next one. When a Pod breaks, read its STATUS first, then work through get, describe, logs --previous and events.

1.Requests and limits

Set per container, for CPU and for memory.

  • requestWhat kube-scheduler reserves. The Pod lands only where the sum of requests fits, even if real use is low.
  • limitA ceiling the kernel enforces with cgroups.
  • Going above the request is fine while the node has room.
  • Limit only ? The API server copies it into the request, so a Pod with only CPU and memory limits is still Guaranteed.
  • LimitRange : namespace defaults, min and max, applied at admission. Break them, get 403 Forbidden.

See : scheduler 02, HPA % of requests 04, LimitRange and ResourceQuota 08.

CPU use above limit Memory use above limit Node pressure + use above request Throttled waits each time slice OOMKilled Exit Code 137 Evicted by the kubelet Never killed for CPU use Restart per restartPolicy Pod Failed all containers stop again ? CPU request = a weight when CPU is contended CrashLoopBackOff 10 s up to 300 s New Pod from its controller usually on another node Limits cause throttling and OOM kills. Requests (then Priority) decide who is evicted.
resources:
  requests: { cpu: "250m", memory: "128Mi" }
  limits:   { cpu: "500m", memory: "256Mi" }  # 0.5 CPU

2.QoS classes

Kubernetes derives status.qosClass from the resources. You never set it, and it is fixed once the Pod is created.

Pod spec all containers, or Pod-level resources Every container : CPU + memory, request = limit, both above 0 ? yes Guaranteed score -997 no Any CPU or memory request or limit set anywhere ? yes Burstable score 3 to 999 no BestEffort score 1000 oom_score_adj : the highest is killed first
  • Pod-level spec.resources (Beta since v1.34, on by default) count too.
ClassRuleUnder node pressure
GuaranteedEvery container sets CPU and memory, request = limitEvicted last, never over its requests
BurstableSome CPU or memory request or limit, not GuaranteedEarly if over its requests, late if under
BestEffortNo CPU or memory request or limit anywhereEarly, no request to stay under
How the kubelet picks
  1. Using more than its requests ? Goes first.
  2. Then lower Priority first.
  3. Then the furthest over its requests.
QoS only predicts this order. It does not count under DiskPressure. More on the kubelet in Architecture.

3.Labels vs annotations

LabelsAnnotationsCh.
PurposeIdentify and group objectsDescribe objects for tools and people04, 09
Selected byServices, workloads, kubectl -lNever05, 04
SelectorsA Service matches on equality only. Deployment, ReplicaSet, Job and DaemonSet also take In, NotIn, Exists, DoesNotExist. A Deployment's selector is immutable and must match its Pod template labels.05, 04
Key[prefix/]name, name 63 chars max. kubernetes.io/ and k8s.io/ are reserved. Same rules for both.04
Value63 chars max, may be emptyAny string, 256 KiB in total per object04
Exampleapp: webkubernetes.io/change-cause: "to 1.16.1"04, 09
RecommendedPrefix app.kubernetes.io/ plus nameinstanceversioncomponentpart-ofmanaged-bynone
In short : Requests place Pods and decide eviction, limits cap them (CPU is throttled, memory is OOMKilled). Labels select, annotations only describe.