Jobs left behind in a failed state
What does ZopNight detect here?
A Kubernetes Job that exhausted its backoffLimit stays in the cluster as a failed object, holding its pods and their logs. ZopNight surfaces these because ttlSecondsAfterFinished is unset by default, so nothing removes them; 1 broken CronJob can leave hundreds behind.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-1735 · RC-1835 · RC-1935 |
| Category | reliability |
| Severity | medium |
| Metric | none — pure configuration read |
| Source | internal/rules/k8s/job_failed.go |
Where it applies
Failed Jobs do not clean themselves up
When a Job exceeds backoffLimit, it stops retrying and is marked failed. The Job object stays.
Its pods stay too, in Error state, holding their logs.
That retention is intentional: you want the evidence. What is missing is an expiry:
ttlSecondsAfterFinished is unset by default, so nothing ever removes them.
What that costs
Not much directly, and this is worth being honest about. Terminated pods consume no CPU or memory.
They do consume:
- etcd storage and API server memory. Thousands of retained Job and pod objects measurably slow list operations across the cluster.
- Node disk, through the container logs of terminated pods, until the kubelet’s garbage collection thresholds trigger.
- Attention. A namespace with 400 failed Jobs from a broken CronJob makes the one that failed this morning impossible to find.
That last one is the real argument. Signal, not spend.
Read them before deleting them
A failed Job is a diagnostic record. The pod logs explain why it failed, and they disappear with the pod:
kubectl get jobs -A --field-selector status.successful=0kubectl logs job/<name> -n <ns> --previous--previous matters. Without it you get the last container attempt, which on a
backoffLimit-exhausted Job is often not the informative one.
Preventing the accumulation
Set ttlSecondsAfterFinished on the Job spec, and the TTL controller then removes completed
and failed Jobs after that window. A few days is usually right: long enough to investigate, short
enough that nothing piles up.
For CronJobs, successfulJobsHistoryLimit and failedJobsHistoryLimit do the same job by count
rather than time, and are the better fit for a recurring schedule.
The one to check first
A Job failing repeatedly under a CronJob produces a new failed object every schedule tick. Those are worth finding before the cleanup, because the cleanup will hide the pattern.