CronJobs suspended and forgotten

Suspension is meant to be temporary

suspend: true is the pause button. It is used during incidents, migrations and deploys, and it is intended to be reverted within hours.

What makes it dangerous is that a suspended CronJob is not an error. It produces no failed runs, no alerts and no missed-schedule warnings, because from Kubernetes’ perspective nothing was scheduled to happen. The absence of a job is indistinguishable from a job that was never wanted.

What is usually behind one

Backups. Certificate renewals. Data reconciliation. Cleanup jobs that keep a volume from filling. These are exactly the workloads people suspend during an incident, and exactly the ones whose absence is invisible until the day you need the thing they were producing.

The discovery moment is typically a restore request against backups that stopped months ago.

Suspended is not the same as scheduled-and-failing

Worth separating, because the remediation differs completely:

• Suspended: nothing ran, nothing failed, no history to inspect.
• Failing: jobs ran and errored, and kubectl get jobs shows the wreckage.

A failing CronJob is loud. A suspended one is silent, which is why it needs a rule and the other one does not.

Checking it yourself

kubectl get cronjob -A -o json | jq -r '
  .items[] | select(.spec.suspend == true)
  | "\(.metadata.namespace)/\(.metadata.name)\tlast=\(.status.lastScheduleTime // "never")"'

lastScheduleTime is the useful column: it tells you how long the gap has been, and “never” means it was suspended before it ever ran.

Resuming carefully

Un-suspending a CronJob with a startingDeadlineSeconds set and a long backlog can trigger multiple immediate runs as the controller catches up on missed schedules.

For a backup job that is harmless. For anything that mutates data or sends notifications, check concurrencyPolicy first. Forbid or Replace prevents a thundering herd that Allow will happily produce.