# Failed Job

> A Kubernetes Job that exhausted its backoffLimit stays in the cluster as a failed object, holding its pods and their logs. ZopNight surfaces these because ttlSecondsAfterFinished is unset by default, so nothing removes them; 1 broken CronJob can leave hundreds behind.

Source: https://zop.dev/integrations/kubernetes/recommendations/failed-job
Updated: 2026-08-19

---

## Failed Jobs do not clean themselves up

When a Job exceeds `backoffLimit`, it stops retrying and is marked failed. The Job object stays.
Its pods stay too, in `Error` state, holding their logs.

That retention is intentional: you want the evidence. What is missing is an expiry:
`ttlSecondsAfterFinished` is unset by default, so nothing ever removes them.

## What that costs

Not much directly, and this is worth being honest about. Terminated pods consume no CPU or
memory.

They do consume:

- **etcd storage and API server memory.** Thousands of retained Job and pod objects measurably
  slow list operations across the cluster.
- **Node disk**, through the container logs of terminated pods, until the kubelet's garbage
  collection thresholds trigger.
- **Attention.** A namespace with 400 failed Jobs from a broken CronJob makes the one that failed
  this morning impossible to find.

That last one is the real argument. Signal, not spend.

## Read them before deleting them

A failed Job is a diagnostic record. The pod logs explain why it failed, and they disappear with
the pod:

```bash
kubectl get jobs -A --field-selector status.successful=0
kubectl logs job/<name> -n <ns> --previous
```

`--previous` matters. Without it you get the last container attempt, which on a
`backoffLimit`-exhausted Job is often not the informative one.

## Preventing the accumulation

Set `ttlSecondsAfterFinished` on the Job spec, and the TTL controller then removes completed
and failed Jobs after that window. A few days is usually right: long enough to investigate, short
enough that nothing piles up.

For CronJobs, `successfulJobsHistoryLimit` and `failedJobsHistoryLimit` do the same job by count
rather than time, and are the better fit for a recurring schedule.

## The one to check first

A Job failing repeatedly under a CronJob produces a new failed object every schedule tick. Those
are worth finding before the cleanup, because the cleanup will hide the pattern.