CronJobs left on concurrencyPolicy Allow, where slow runs can overlap

What Allow does when a run is slow

A CronJob creates a Job each time its schedule fires. The CronJob concepts page lists three concurrency policies and marks Allow as the default: concurrently running Jobs are permitted. Forbid skips the new run if the previous one is still going, and Replace cancels the running Job and starts the new one in its place.

Under Allow, a job scheduled every five minutes that suddenly takes twelve leaves two or three copies running at once. If the job writes to a database, processes a queue or rotates files, the copies race each other. If it is heavy, each copy requests its own CPU and memory, so a backlog of overlapping runs can crowd other pods off the nodes. The same page also warns that scheduling is approximate and that jobs should be idempotent, because Kubernetes can create two Jobs for one slot in rare cases.

Finding CronJobs that allow overlap

Kubernetes fills in the default when the object is stored, so the field is almost always present. This lists every CronJob still set to Allow, with its schedule:

kubectl get cronjobs -A -o json | jq -r '
  .items[]
  | select((.spec.concurrencyPolicy // "Allow") == "Allow")
  | "\(.metadata.namespace)/\(.metadata.name) schedule=\(.spec.schedule)"'

Compare that list with job durations from kubectl get jobs -A to find the ones at real risk.

One field decides the finding

The check reads concurrencyPolicy as ZopNight collected it and fires only when the value is exactly Allow. There is no threshold on job duration and no history window: ZopNight does not wait to see an overlap happen before flagging the setting. When the policy is not reported at all, the CronJob is skipped.

Cases it does not try to judge

The rule cannot tell whether overlap is harmless for a given job. A job that always finishes in seconds on an hourly schedule will be flagged even though it will never overlap in practice, and some batch designs rely on parallel runs on purpose. Suspended CronJobs are not excluded; they are also reported by Suspended CronJob.

Governance finding, no dollar figure

This is a low-severity governance recommendation with no savings estimate. The cost it guards against is indirect: duplicated work, corrupted output and the extra node capacity that stacked runs consume.

Choosing Forbid or Replace

1. Decide which failure mode is acceptable. Forbid keeps the running job and skips the new slot; Replace kills the running job so the newest run wins.
2. Patch the CronJob, for example kubectl patch cronjob nightly-report -p '{"spec":{"concurrencyPolicy":"Forbid"}}'.
3. With Forbid, set startingDeadlineSeconds deliberately. Kubernetes counts skipped slots as missed, and when it finds more than 100 missed schedules it does not start the catch-up Job and logs too many missed start times. A deadline limits how far back that count looks.
4. Add activeDeadlineSeconds to the Job template so a hung run cannot hold the slot forever.