Outcome
By the end of this lesson, you will be able to suspend K8s CronJobs to skip executions, resume them on schedule, and handle concurrent jobs gracefully.
| Prerequisites | M2.6.L1 - L3 |
| Time | 9 minutes |
1. Concept
Kubernetes already has a pause button for scheduled jobs, and ZopNight simply presses it.
A CronJob carries a field called suspend. Set it to true and the job skips its next run. Set it back to false and it resumes. A namespace Stop sets it true on every CronJob in the namespace; Start sets it back to false. One exception: if a CronJob was already suspended before Stop, ZopNight records that and Start leaves it suspended.
SUSPEND PATTERN: K8s-native; non-destructive Skips next execution Preserves config + history Reversible instantlyNothing here is invented. It is the same thing a Kubernetes operator would do by hand.
Why suspend (not delete)
SUSPENDING preserves: CronJob's history (past job runs visible) Configuration (spec.schedule, concurrencyPolicy, etc.) Lineage (annotations, labels)
DELETING + RECREATING would: Lose job history Risk concurrency issues during recreation Require re-applying the YAML manifest Cause cascading impact if other things reference the CronJob name
CONCLUSION: suspend is the K8s-native way to "pause" a CronJob Delete only when permanently retiringThe pause/resume cycle is the right primitive.
What suspend does
BEFORE (active): CronJob: nightly-cleanup Spec.schedule: "0 2 * * *" (daily at 2 AM) Spec.suspend: false Result: fires every day at 2 AM; creates a Job; runs
AFTER (suspended): CronJob: nightly-cleanup Spec.schedule: "0 2 * * *" (unchanged) Spec.suspend: true Result: scheduled time passes without firing. No Job created.
AFTER RESUME: Spec.suspend: false Result: Kubernetes may start ONE catch-up Job right away for the most recent missed time, then fires normally Older missed times are not replayedThe K8s scheduler checks suspend before firing. On resume it does not replay every missed run, but it may start one Job immediately for the most recent missed time. Set startingDeadlineSeconds on the CronJob if a late run would be harmful: a missed time older than that deadline is skipped instead of run.
When to use CronJob suspend
PATTERN SUSPEND DURING─────────────────────────────────────────────────────────Maintenance window The windowCost-aggressive non-prod cleanup WeekendsTooling that runs on prod only Non-prod clusterFailed job (until investigation) Until fixedHoliday freeze The freeze periodCompliance / audit window During auditPre-launch traffic ramp During ramp (avoid interference)The use cases are broad. Suspend is the right tool for “pause this CronJob for a defined window.”
How ZopNight schedules CronJob suspend
SUSPEND CYCLE:
T+0 Namespace stop fires (cronjob/nightly-cleanup lives there)T+1s ZopNight records suspend=false, then patches cronjob/nightly-cleanup -p '{"spec":{"suspend":true}}'T+2s K8s confirms patchT+2s CronJob is suspended
T+12h Namespace start firesT+1s ZopNight: patch cronjob/nightly-cleanup -p '{"spec":{"suspend":false}}' (it was not suspended before)T+2s K8s confirms patchT+2s CronJob is active; a catch-up Job may start now for the most recent missed time (see above)
CHARACTERISTICS: Two patches per cycle Idempotent (suspend=true if already true is no-op) Fast (seconds) Audit loggedThe pattern is simple + cheap. Most CronJob scheduling is this.
Concurrency policy interaction
CronJob's concurrencyPolicy field:
Allow: Multiple Jobs can run concurrently (default) Forbid: If a Job is still running, new ones are skipped Replace: New Jobs replace currently-running one
SUSPENDING A CRONJOB: Doesn't affect Jobs currently running They complete normally per their concurrency policy Only affects NEW scheduled executions
NEXT SCHEDULED TIME AFTER RESUME: Continues per the policy At most one catch-up Job for the most recent missed time, unless startingDeadlineSeconds rules it outThe concurrencyPolicy is independent of suspend; they coexist.
What happens to already-running jobs
A SCHEDULED STOP on a CronJob does NOT affect: Jobs already in progress (they complete) Pods of running Jobs (they keep running)
ONLY new scheduled executions are affected.The next time the CronJob would fire: doesn't fire (suspend=true).
IF YOU NEED to stop in-progress jobs: More invasive action required Manually: kubectl delete job <job-name> ZopNight's K8s scheduling doesn't kill running jobs by default Conservative approach: don't interrupt running workThe conservative behavior is intentional. Killing in-progress jobs would be too aggressive.
Suspending vs scaling parent Deployment
Sometimes a workload involves both:
A LONG-RUNNING DEPLOYMENT processes a queueA CRONJOB publishes work to the queue
IF BOTH ARE IN THE SAME NON-PROD NAMESPACE, one schedule gives: Stop at 8 PM: Deployment replicas → 0, CronJob suspend → true Start at 8 AM: both restored within seconds
ORDER MATTERS: Deployment should be up before the CronJob next fires Otherwise: CronJob creates work; nothing to process
SOLUTION: Set the namespace Start before the CronJob's first run (e.g. Start 8:00 AM; CronJob's first run 8:15 AM) Different namespaces? Put them in one group with sequenced execution (T1.M1.4.L4): Deployment's namespace firstNamespace scheduling plus sequencing handles coordinated workloads cleanly.
Audit + observability
EVERY SUSPEND / RESUME logged: Timestamp CronJob name + namespace Action (suspend / resume) User (ZopNight automation) Cluster
VISIBLE IN: ZopNight: workload detail → events tab K8s: kubectl describe cronjob/<name> Cloud-provider event log
TROUBLESHOOTING: "Why didn't this job run on Sunday?" → events show suspend on Saturday "Why are jobs running off-hours?" → events show no suspend; check scheduleThe audit trail explains every skipped or run execution.
2. Demo
A team’s CronJob scheduling:
WORKLOAD: nightly-cleanup CronJob in staging cluster Schedule: 0 2 * * * (2 AM daily) Purpose: scrub old test data
STAGING CLUSTER PATTERN: Engineers don't use staging on weekends The cleanup is unnecessary off-hours
ZOPNIGHT SCHEDULE: business-hours-staging cron: 0 0 * * 6 stop (Saturday midnight) 0 8 * * 1 start (Monday 8 AM) Attached to: the staging namespace (nightly-cleanup lives there)
RESULT: Saturday 00:00: suspend=true Saturday 02:00: job skipped Sunday 02:00: job skipped (still suspended) Monday 02:00: job skipped (still suspended) Monday 08:00: suspend=false Kubernetes may start one catch-up Job now for the missed Monday 02:00 run (no startingDeadlineSeconds is set) Tuesday 02:00: next normal run
To stop the catch-up run, set startingDeadlineSeconds (for example 3600): a missed time more than an hour old is then skipped
COST SAVINGS: 2 Job executions per weekend = 104 Job runs / year At ~$0.05 per Job run (resources): ~$5.20/year per CronJob
ACROSS THE ESTATE: Many CronJobs follow same pattern Aggregate savings: meaningful (~$50-200/year typical)
PLUS: indirect savings: Not running cleanup means data not modified Avoids re-creating test data after cleanup Smoother Monday morning startupSmall per-CronJob, useful in aggregate, structurally safer than not scheduling.
3. Hands-on (5 min)
Schedule a CronJob:
□ STEP 1: Find a CronJob in your cluster Workload: __________ Schedule: __________ Suspend status: __________
□ STEP 2: Attach the CronJob's namespace to a schedule Namespace: __________ Schedule name: __________ Stop time: __________ Start time: __________
□ STEP 3: Trigger stop cron (or wait) Verify suspend=true: $ kubectl get cronjob <name> -o yaml | grep suspend Expected: suspend: true
□ STEP 4: Verify resume $ kubectl get cronjob <name> -o yaml | grep suspend Expected: suspend: false
□ STEP 5: Check for a catch-up run after resume $ kubectl get jobs --sort-by=.metadata.creationTimestamp A Job created at the resume time is the catch-up run for the most recent missed schedule $ kubectl get cronjob <name> -o jsonpath='{.spec.startingDeadlineSeconds}' Empty means a catch-up run can happen; set it if a late run would be harmfulA 10-minute exercise builds CronJob scheduling confidence.
4. Knowledge check
Q1
Suspending a CronJob:
A. Stops in-progress Jobs immediately
B. Sets spec.suspend=true
C. Deletes the CronJob
D. Pauses Pods
Show answer
Correct: B. The next scheduled time is skipped, and in-progress Jobs keep running. If you need to kill running jobs, that is a separate manual action. Suspend means skip the next run, not stop the running one.
Q2
Suspending a CronJob preserves:
A. The schedule, configuration, history, concurrency policy
B. Only the concurrency policy setting, nothing else
C. Just history
D. Just the schedule
Show answer
Correct: A. Only the skipping of scheduled times changes, and resume returns it to normal. Suspend is non-destructive.
Q3
A CronJob runs daily at 02:00 and has no startingDeadlineSeconds. It is suspended Saturday 00:00 and resumed Monday 08:00. What happens at resume:
A. Every missed run is replayed, one Job per skipped day, right after resume
B. One catch-up Job for the most recent missed time may start right at resume
C. Nothing runs until the next scheduled time, Tuesday 02:00, with no exception
D. The CronJob stays suspended until someone resumes it by hand in kubectl
Show answer
Correct: B. While suspended, the Saturday, Sunday and Monday 02:00 runs do not fire. On resume, Kubernetes does not replay them all, but it may start one Job immediately for the most recent missed time (Monday 02:00). Setting startingDeadlineSeconds makes it skip a missed time older than the deadline instead.
5. Apply
CronJob scheduling happens through the namespace: attach a schedule to the namespace on the Inventory page, and every CronJob in it is suspended on Stop. Same pattern as Deployments.
For your team: identify non-prod namespaces whose CronJobs are non-essential off-hours; schedule those namespaces.
Related lessons
- L1: Why K8s is hard
- L2: Hierarchy
- L3: Deployment to zero
- L5: StatefulSets (next)
- L6: Cross-cluster patterns
- T1.M1.4.L4: Sequenced execution