Suspending CronJobs

SUSPEND PATTERN:
  K8s-native; non-destructive
  Skips next execution
  Preserves config + history
  Reversible instantly

SUSPENDING preserves:
  CronJob's history (past job runs visible)
  Configuration (spec.schedule, concurrencyPolicy, etc.)
  Lineage (annotations, labels)

DELETING + RECREATING would:
  Lose job history
  Risk concurrency issues during recreation
  Require re-applying the YAML manifest
  Cause cascading impact if other things reference the CronJob name

CONCLUSION: suspend is the K8s-native way to "pause" a CronJob
            Delete only when permanently retiring

BEFORE (active):
  CronJob: nightly-cleanup
  Spec.schedule: "0 2 * * *" (daily at 2 AM)
  Spec.suspend: false
  Result: fires every day at 2 AM; creates a Job; runs

AFTER (suspended):
  CronJob: nightly-cleanup
  Spec.schedule: "0 2 * * *" (unchanged)
  Spec.suspend: true
  Result: scheduled time passes without firing. No Job created.

AFTER RESUME:
  Spec.suspend: false
  Result: next scheduled time fires normally
  Past skipped times do NOT backfill

PATTERN                                  SUSPEND DURING
─────────────────────────────────────────────────────────
Maintenance window                       The window
Cost-aggressive non-prod cleanup         Weekends
Tooling that runs on prod only           Non-prod cluster
Failed job (until investigation)         Until fixed
Holiday freeze                            The freeze period
Compliance / audit window                During audit
Pre-launch traffic ramp                   During ramp (avoid interference)

SUSPEND CYCLE:

T+0      Cron stop fires
T+1s     ZopNight: kubectl patch cronjob/nightly-cleanup
         -p '{"spec":{"suspend":true}}'
T+2s     K8s confirms patch
T+2s     CronJob is suspended

T+12h    Cron start fires
T+1s     ZopNight: kubectl patch cronjob/nightly-cleanup
         -p '{"spec":{"suspend":false}}'
T+2s     K8s confirms patch
T+2s     CronJob is active; next scheduled time fires normally

CHARACTERISTICS:
  Two patches per cycle
  Idempotent (suspend=true if already true is no-op)
  Fast (seconds)
  Audit logged

CronJob's concurrencyPolicy field:

  Allow:    Multiple Jobs can run concurrently (default)
  Forbid:   If a Job is still running, new ones are skipped
  Replace:  New Jobs replace currently-running one

SUSPENDING A CRONJOB:
  Doesn't affect Jobs currently running
  They complete normally per their concurrency policy
  Only affects NEW scheduled executions

NEXT SCHEDULED TIME AFTER RESUME:
  Continues per the policy
  No backfill of skipped times

A SCHEDULED STOP on a CronJob does NOT affect:
  Jobs already in progress (they complete)
  Pods of running Jobs (they keep running)

ONLY new scheduled executions are affected.
The next time the CronJob would fire: doesn't fire (suspend=true).

IF YOU NEED to stop in-progress jobs:
  More invasive action required
  Manually: kubectl delete job <job-name>
  ZopNight's K8s scheduling doesn't kill running jobs by default
  Conservative approach: don't interrupt running work

A LONG-RUNNING DEPLOYMENT processes a queue
A CRONJOB publishes work to the queue

IF BOTH ARE NON-PROD, scheduling both gives:
  Stop Deployment at 8 PM (replicas → 0)
  Stop CronJob at 8 PM (suspend → true)
  Start CronJob at 8 AM (suspend → false)
  Start Deployment at 8 AM (replicas → restored)

ORDER MATTERS:
  Deployment should start before CronJob fires
  Otherwise: CronJob creates work; nothing to process

SOLUTION:
  Sequenced execution (T1.M1.4.L4)
  Start Deployment at 7:55 AM
  Start CronJob at 8:00 AM
  Stop sequence: opposite order

EVERY SUSPEND / RESUME logged:
  Timestamp
  CronJob name + namespace
  Action (suspend / resume)
  User (ZopNight automation)
  Cluster

VISIBLE IN:
  ZopNight: workload detail → events tab
  K8s: kubectl describe cronjob/<name>
  Cloud-provider event log

TROUBLESHOOTING:
  "Why didn't this job run on Sunday?" → events show suspend on Saturday
  "Why are jobs running off-hours?" → events show no suspend; check schedule

WORKLOAD: nightly-cleanup CronJob in staging cluster
  Schedule: 0 2 * * * (2 AM daily)
  Purpose: scrub old test data

STAGING CLUSTER PATTERN:
  Engineers don't use staging on weekends
  The cleanup is unnecessary off-hours

ZOPNIGHT SCHEDULE:
  business-hours-staging cron:
    0 0 * * 6 stop (Saturday midnight)
    0 8 * * 1 start (Monday 8 AM)
  Attached to: nightly-cleanup CronJob

RESULT:
  Saturday 00:00: suspend=true
  Sunday 02:00:    job skipped (still suspended)
  Monday 08:00:    suspend=false
  Tuesday 02:00:   next normal run

COST SAVINGS:
  1 Job execution per weekend = 52 Job runs / year
  At ~$0.05 per Job run (resources): ~$2.60/year per CronJob

ACROSS THE ESTATE:
  Many CronJobs follow same pattern
  Aggregate savings: meaningful (~$50-200/year typical)

PLUS: indirect savings:
  Not running cleanup means data not modified
  Avoids re-creating test data after cleanup
  Smoother Monday morning startup

□ STEP 1: Find a CronJob in your cluster
  Workload: __________
  Schedule: __________
  Suspend status: __________

□ STEP 2: Attach to a schedule
  Schedule name: __________
  Stop time: __________
  Start time: __________

□ STEP 3: Trigger stop cron (or wait)
  Verify suspend=true:
  $ kubectl get cronjob <name> -o yaml | grep suspend
  Expected: suspend: true

□ STEP 4: Verify resume
  $ kubectl get cronjob <name> -o yaml | grep suspend
  Expected: suspend: false

□ STEP 5: Verify no missed-time backfill
  $ kubectl get cronjob <name>
  Last schedule time: should be next future time, not skipped past