Core Kubernetes · 11
Jobs and CronJobs
1.What is a Job ?
A Job runs Pods until enough of them succeed, then stops. A CronJob makes a new Job on a schedule. Both are stable batch/v1 workloads (see Deployments and Scaling).
- job-controller in kube-controller-manager makes the Pods, counts the finished ones and writes the result.
restartPolicymust beNeverorOnFailure. Unset means Always : rejected.
| Object | Ends | Lost Pod back | Trigger |
|---|---|---|---|
| Job | yes | yes | create |
| CronJob | yes | via its Jobs | schedule |
| Bare Pod | yes | no | create |
| Deployment | no | yes | - |
| ScaledJob | yes | via its Jobs | event |
Use a Job even for one task : a bare Pod lost with its node is not replaced.
2.The retry loop
A Job is a retry loop with a failure budget. A failed Pod is replaced after a delay, until the failures pass backoffLimit.
- backoffLimit counts failures you tolerate, not tries : the Job fails when
failed > backoffLimit. - Default 6 : the 7th failed Pod ends the Job.
0fails on the first. - Delay : 10s, doubling, capped at 10 min (the docs still say six). A success resets it.
- activeDeadlineSeconds beats the budget. It counts from
startTime, Pending and image pulls included.
| Failure | 1 | 2 | 3 | 4 | 5 | 6 | 7+ |
|---|---|---|---|---|---|---|---|
| wait | 10s | 20s | 40s | 80s | 160s | 320s | 600s |
3.Never vs OnFailure
| Never | OnFailure | |
|---|---|---|
| Retried by | controller, new Pod | kubelet, same Pod |
| Delay cap | 10 min | 5 min (CrashLoopBackOff) |
| Budget | failed Pods | failed Pods or restarts |
| At limit | failed Pods stay | Pod killed, logs may go |
| Default of | create job | create cronjob |
4.Completions and parallelism
| Kind | completions / parallelism | Done when |
|---|---|---|
| Single | unset / unset | one Pod succeeds |
| Fixed | N / P | N Pods succeeded |
| Indexed | N / P | each index 0..N-1 once |
| Queue | unset / P | a success, none active |
parallelismcan change any time,0pauses.completionsis frozen (Indexed : only with parallelism, same value).
completions stays null (the docs say it copies parallelism; it does not).JOB_COMPLETION_INDEX and hostname <job>-<i>. A headless Service as subdomain lets peers find each other.1.How a Job ends
Always in two steps : an interim condition first (the Pods still running are deleted), then the final one, with the same reason, once no Pod is left terminating.
Pods deleted at the end count as failed : one exit 42 in a parallelism 3 Job showed failed: 3.
| What fired | Reason on both |
|---|---|
backoffLimit | BackoffLimitExceeded |
activeDeadlineSeconds | DeadlineExceeded |
| podFailurePolicy FailJob | PodFailurePolicy |
maxFailedIndexes | MaxFailedIndexesExceeded |
| a failed index | FailedIndexes |
| successPolicy | SuccessPolicy |
| completions reached | CompletionsReached |
kubectl describe job shows no conditions : use get job -o yaml.
2.Pod failure policy
spec:
podFailurePolicy:
rules:
- action: FailJob # known bug
onExitCodes:
operator: In
values: [42]
- action: Ignore # not our fault
onPodConditions:
- type: DisruptionTarget
- FailJobFail the whole Job now.
- FailIndexFail this index only (needs backoffLimitPerIndex).
- IgnoreNot counted, but still delays the retry.
- CountNormal. Also the result when no rule matches.
Rules run in order, first match wins (max 20). Exit 0 never matches. It forces restartPolicy: Never and podReplacementPolicy: Failed. GA v1.31.
DisruptionTarget : preemption, API eviction, taint manager, Pod GC or kubelet termination. Not your bug.
3.Per-index budget, success policy
- backoffLimitPerIndex : each index gets its own retries. Indexed only, like the rest here (GA v1.33).
- maxFailedIndexes : past it, the Job fails at once. Unset : every index runs, then any failed one fails the Job (not Complete, as the API comment says).
- successPolicy : the Job succeeds early once listed indexes (or a count of them) succeed.
- Failures still come first : the docs' success example can end BackoffLimitExceeded. Raise backoffLimit.
4.Replace, hand off, suspend
- podReplacementPolicy :
TerminatingOrFailed(default) counts a deleting Pod as failed and replaces it after the retry delay (about 10 s).Failedwaits until it has failed : one Pod per index at a time. GA v1.34. - managedBy : hands the Job to another controller (MultiKueue). If nothing implements it : no Pods, STATUS Running forever. GA v1.35.
- suspend : running Pods get SIGTERM, counted as neither. Resume resets startTime, so the deadline restarts. Kueue admits Jobs this way.
- Yes : parallelism, backoffLimit, activeDeadlineSeconds, ttlSecondsAfterFinished, podReplacementPolicy, maxFailedIndexes, suspend.
- No : completionMode, NonIndexed completions, backoffLimitPerIndex, podFailurePolicy, successPolicy, managedBy.
- Template : scheduling and resources, only while suspended. Never the image : recreate, or use Flux's force annotation.
1.A CronJob stamps out Jobs
- At each tick the cronjob-controller creates
<cronjob>-<minutes since epoch>. The fixed name blocks a double run. - The Job copies
jobTemplate, is owned by the CronJob and notes its slot in an annotation. - Edits and
suspend: trueonly touch future runs (open issue 128155 : ascheduleedit can run a passed minute). Names stop at 52 characters.
2.Schedule syntax
| minute | hour | day | month | weekday |
|---|---|---|---|---|
| 0-59 | 0-23 | 1-31 | 1-12, jan-dec | 0-6, sun-sat |
| 30 | 2 | * | * | 1-5 |
- Above : 02:30, Monday to Friday. Also
*/15,1,5,@hourly,@daily,@weekly,@monthly,@yearly. - Rejected :
7as Sunday, a seconds field,@reboot,TZ=. - Day and weekday both set means either :
0 0 13 * 5runs on the 13th and every Friday.
- No
timeZone: kube-controller-manager's own zone. On kubeadm that is UTC. - Set an IANA name :
timeZone: Etc/UTC.Localis rejected. - The scheduled-timestamp annotation is in that zone, UTC when unset.
- DST (library test, New York) :
30 2 * * *skipped on 2026-03-08,30 1 * * *twice on 2026-11-01. Etc/UTC dodges it.
3.Overlap and missed runs
| Policy | Last run still active |
|---|---|
| Allow (default) | a new Job starts anyway |
| Forbid | skipped (JobAlreadyActive), may start late when the blocker ends |
| Replace | old Jobs deleted, then the new one |
A hung run blocks Forbid forever : set activeDeadlineSeconds in the jobTemplate. Manual runs are invisible to Forbid.
- Catch-up is one Job, for the latest missed slot (downtime, suspend, Forbid).
startingDeadlineSeconds: older slots are dropped with no Job and no event (MissSchedule never fires).- Over 100 missed : Warning TooManyMissedTimes, and the latest run still starts.
- Watch
lastScheduleTimeor kube-state-metrics, not events : they thin out.
4.Cleanup
| Mechanism | Covers | Deletes with |
|---|---|---|
ttlSecondsAfterFinished | any finished Job, clock starts at Complete or Failed, 0 = at once | Foreground |
| history limits (3 and 1) | finished Jobs the CronJob owns, 0 keeps none | Background |
kubectl delete job | the Job and its Pods (--cascade=orphan keeps Pods) | Background |
| raw API DELETE | the Job only, its Pods are left behind | Orphan |
A standalone Job without TTL stays forever. Owners and propagation : Garbage Collection.
5.Debugging, in order
| Command | Look for |
|---|---|
kubectl describe job pi | Events : the reason, FailedCreate (quota, Pod Security, webhook) |
kubectl get pods -l batch.kubernetes.io/job-name=pi --field-selector=status.phase=Failed | Only the failed Pods. Then describe pod : exit code, OOMKilled |
kubectl logs job/pi --all-pods=true | Logs of every Pod, prefixed |
kubectl create job hello-retry --from=cronjob/hello | A Failed Job never restarts. Make a new one |
kubectl events --for cronjob/hello | JobAlreadyActive, UnexpectedJob, TooManyMissedTimes |