Core Kubernetes · 11

Jobs and CronJobs

1.What is a Job ?

A Job runs Pods until enough of them succeed, then stops. A CronJob makes a new Job on a schedule. Both are stable batch/v1 workloads (see Deployments and Scaling).

  • job-controller in kube-controller-manager makes the Pods, counts the finished ones and writes the result.
  • restartPolicy must be Never or OnFailure. Unset means Always : rejected.
ObjectEndsLost Pod backTrigger
Jobyesyescreate
CronJobyesvia its Jobsschedule
Bare Podyesnocreate
Deploymentnoyes-
ScaledJobyesvia its Jobsevent

Use a Job even for one task : a bare Pod lost with its node is not replaced.

A Job can start the same work twice, even with parallelism 1. Make the work idempotent.

2.The retry loop

A Job is a retry loop with a failure budget. A failed Pod is replaced after a delay, until the failures pass backoffLimit.

  • backoffLimit counts failures you tolerate, not tries : the Job fails when failed > backoffLimit.
  • Default 6 : the 7th failed Pod ends the Job. 0 fails on the first.
  • Delay : 10s, doubling, capped at 10 min (the docs still say six). A success resets it.
  • activeDeadlineSeconds beats the budget. It counts from startTime, Pending and image pulls included.
Failure1234567+
wait10s20s40s80s160s320s600s
Job controller creates Pods create Pod Pod runs restartPolicy Never exit 0 fails succeeded + 1 delay back to 10s failed + 1 status.failed failed > backoffLimit ? yes Job Failed BackoffLimitExceeded no wait 10s to 10m replace the Pod enough ? Complete job done

3.Never vs OnFailure

NeverOnFailure
Retried bycontroller, new Podkubelet, same Pod
Delay cap10 min5 min (CrashLoopBackOff)
Budgetfailed Podsfailed Pods or restarts
At limitfailed Pods stayPod killed, logs may go
Default ofcreate jobcreate cronjob
Debug with NeverFailed Pods and their logs stay around to read. With OnFailure the Job also fails once container restarts in live Pods reach backoffLimit, and that count is known to be inexact (open issue).

4.Completions and parallelism

Kindcompletions / parallelismDone when
Singleunset / unsetone Pod succeeds
FixedN / PN Pods succeeded
IndexedN / Peach index 0..N-1 once
Queueunset / Pa success, none active
  • parallelism can change any time, 0 pauses. completions is frozen (Indexed : only with parallelism, same value).
Work queuePods share the queue and decide when it is empty. After the first success no new Pod starts. completions stays null (the docs say it copies parallelism; it does not).
IndexedPod i gets JOB_COMPLETION_INDEX and hostname <job>-<i>. A headless Service as subdomain lets peers find each other.
In short : A Job is a retry loop with a failure budget : failed Pods return after 10s, 20s ... up to 10 min, by default the 7th failure ends it, and completions and parallelism define done.

1.How a Job ends

Always in two steps : an interim condition first (the Pods still running are deleted), then the final one, with the same reason, once no Pod is left terminating.

Job running FailureTarget interim, Pods deleted Failed terminal SuccessCriteriaMet interim, Pods deleted Complete terminal no Pod left terminating failure checks run first

Pods deleted at the end count as failed : one exit 42 in a parallelism 3 Job showed failed: 3.

What firedReason on both
backoffLimitBackoffLimitExceeded
activeDeadlineSecondsDeadlineExceeded
podFailurePolicy FailJobPodFailurePolicy
maxFailedIndexesMaxFailedIndexesExceeded
a failed indexFailedIndexes
successPolicySuccessPolicy
completions reachedCompletionsReached

kubectl describe job shows no conditions : use get job -o yaml.

2.Pod failure policy

spec:
  podFailurePolicy:
    rules:
    - action: FailJob    # known bug
      onExitCodes:
        operator: In
        values: [42]
    - action: Ignore     # not our fault
      onPodConditions:
      - type: DisruptionTarget
  • FailJobFail the whole Job now.
  • FailIndexFail this index only (needs backoffLimitPerIndex).
  • IgnoreNot counted, but still delays the retry.
  • CountNormal. Also the result when no rule matches.

Rules run in order, first match wins (max 20). Exit 0 never matches. It forces restartPolicy: Never and podReplacementPolicy: Failed. GA v1.31.

DisruptionTarget : preemption, API eviction, taint manager, Pod GC or kubelet termination. Not your bug.

3.Per-index budget, success policy

  • backoffLimitPerIndex : each index gets its own retries. Indexed only, like the rest here (GA v1.33).
  • maxFailedIndexes : past it, the Job fails at once. Unset : every index runs, then any failed one fails the Job (not Complete, as the API comment says).
  • successPolicy : the Job succeeds early once listed indexes (or a count of them) succeed.
  • Failures still come first : the docs' success example can end BackoffLimitExceeded. Raise backoffLimit.
completions 4, backoffLimitPerIndex 1 idx 0 exit 1 exit 1 budget used idx 1 exit 42 FailIndex, no retry idx 2 exit 0 done idx 3 exit 0 done Job Failed (FailedIndexes) failedIndexes "0,1", failed 3

4.Replace, hand off, suspend

  • podReplacementPolicy : TerminatingOrFailed (default) counts a deleting Pod as failed and replaces it after the retry delay (about 10 s). Failed waits until it has failed : one Pod per index at a time. GA v1.34.
  • managedBy : hands the Job to another controller (MultiKueue). If nothing implements it : no Pods, STATUS Running forever. GA v1.35.
  • suspend : running Pods get SIGTERM, counted as neither. Resume resets startTime, so the deadline restarts. Kueue admits Jobs this way.
Edit after create ?
  • Yes : parallelism, backoffLimit, activeDeadlineSeconds, ttlSecondsAfterFinished, podReplacementPolicy, maxFailedIndexes, suspend.
  • No : completionMode, NonIndexed completions, backoffLimitPerIndex, podFailurePolicy, successPolicy, managedBy.
  • Template : scheduling and resources, only while suspended. Never the image : recreate, or use Flux's force annotation.
In short : A Job ends in two steps, failures checked before successes. podFailurePolicy, per-index budgets and successPolicy tune that, and most of the spec is frozen once the Job exists.

1.A CronJob stamps out Jobs

  • At each tick the cronjob-controller creates <cronjob>-<minutes since epoch>. The fixed name blocks a double run.
  • The Job copies jobTemplate, is owned by the CronJob and notes its slot in an annotation.
  • Edits and suspend: true only touch future runs (open issue 128155 : a schedule edit can run a passed minute). Names stop at 52 characters.
CronJob hello schedule + template tick Job hello-29854260 2026-10-06 03:00 UTC Pods history limits keep 3 ok, 1 failed create job --from=cronjob a manual run manual Job not in status.active

2.Schedule syntax

minutehourdaymonthweekday
0-590-231-311-12, jan-dec0-6, sun-sat
302**1-5
  • Above : 02:30, Monday to Friday. Also */15, 1,5, @hourly, @daily, @weekly, @monthly, @yearly.
  • Rejected : 7 as Sunday, a seconds field, @reboot, TZ=.
  • Day and weekday both set means either : 0 0 13 * 5 runs on the 13th and every Friday.
Time zones
  • No timeZone : kube-controller-manager's own zone. On kubeadm that is UTC.
  • Set an IANA name : timeZone: Etc/UTC. Local is rejected.
  • The scheduled-timestamp annotation is in that zone, UTC when unset.
  • DST (library test, New York) : 30 2 * * * skipped on 2026-03-08, 30 1 * * * twice on 2026-11-01. Etc/UTC dodges it.

3.Overlap and missed runs

PolicyLast run still active
Allow (default)a new Job starts anyway
Forbidskipped (JobAlreadyActive), may start late when the blocker ends
Replaceold Jobs deleted, then the new one

A hung run blocks Forbid forever : set activeDeadlineSeconds in the jobTemplate. Manual runs are invisible to Forbid.

Missed runs
  • Catch-up is one Job, for the latest missed slot (downtime, suspend, Forbid).
  • startingDeadlineSeconds : older slots are dropped with no Job and no event (MissSchedule never fires).
  • Over 100 missed : Warning TooManyMissedTimes, and the latest run still starts.
  • Watch lastScheduleTime or kube-state-metrics, not events : they thin out.

4.Cleanup

MechanismCoversDeletes with
ttlSecondsAfterFinishedany finished Job, clock starts at Complete or Failed, 0 = at onceForeground
history limits (3 and 1)finished Jobs the CronJob owns, 0 keeps noneBackground
kubectl delete jobthe Job and its Pods (--cascade=orphan keeps Pods)Background
raw API DELETEthe Job only, its Pods are left behindOrphan

A standalone Job without TTL stays forever. Owners and propagation : Garbage Collection.

5.Debugging, in order

CommandLook for
kubectl describe job piEvents : the reason, FailedCreate (quota, Pod Security, webhook)
kubectl get pods -l batch.kubernetes.io/job-name=pi --field-selector=status.phase=FailedOnly the failed Pods. Then describe pod : exit code, OOMKilled
kubectl logs job/pi --all-pods=trueLogs of every Pod, prefixed
kubectl create job hello-retry --from=cronjob/helloA Failed Job never restarts. Make a new one
kubectl events --for cronjob/helloJobAlreadyActive, UnexpectedJob, TooManyMissedTimes
In short : A CronJob stamps a Job per tick and catches up only the latest missed run. Clean up with TTL or history limits, and debug from the Job's reason down to the exit code.