Outcome
By the end of this lesson, you will be able to recognize when scheduled Databricks downtime breaks job dependencies, plan the right resolution pattern (overlap / wake-for-job / job clusters), and avoid the common pitfalls.
| Prerequisites | M2.7.L1 - L3 |
| Time | 9 minutes |
1. Concept
A Databricks job, a notebook or a pipeline on a timer, needs a cluster to run on.
Schedule the cluster off at the hour the job is due, and the job starts the cluster again: the off-hours saving is lost, and a stop that lands mid-run kills the job.
THE DEPENDENCY PROBLEM:
Cluster scheduled off 8 PM - 8 AM Job scheduled to run at 2 AM
At 2 AM: Job triggers Cluster: not running, so Databricks starts it Result: Job runs; saving lost, and a stop mid-run kills it
THIS IS A SCHEDULE CONFLICT Must be planned for Before the schedule is attachedThe conflict is predictable, but ZopNight does not check for it. You find it before you attach the schedule.
The dependency graph
JOB RUNS ON SCHEDULE─────────────────────────────────────────────────────────────────ml-train-nightly ml-cluster (12 nodes) 2 AM dailydata-pipeline-hourly data-cluster (4 nodes) hourlyreport-warehouse-refresh reporting-warehouse 6 AM dailyIf the cluster is scheduled off at 2 AM, ml-train-nightly restarts it anyway and the off-hours saving is lost; a stop that lands mid-run kills the job.
Three resolution patterns
PATTERN A: SCHEDULE CLUSTER TO BE AVAILABLE Cluster runs during the Job's time Example: cluster business-hours 8 AM - 8 PM Job moved to 9 AM (instead of 2 AM)
Result: Job runs successfully Trade-off: change Job schedule (may not match other constraints)
PATTERN B: WAKE CLUSTER FOR JOB Cluster ml-cluster schedule: 1:30 AM - 4 AM Mon-Fri (Plus regular business hours 8 AM - 8 PM)
Cluster wakes just in time Job ml-train-nightly runs at 2 AM Cluster stops at 4 AM (after Job done)
Result: Job runs; cost minimized Trade-off: precise timing required
PATTERN C: RUN JOB ON JOB CLUSTER Cluster ml-cluster: scheduled off Job configured to use Job cluster (one-time, ephemeral)
Job triggers → Job cluster spawns Job runs Job cluster terminates
Result: independent of any scheduled cluster Trade-off: each Job has cluster cold-start latencyThe patterns trade off cost, complexity, and timing precision.
Finding the dependencies yourself
ZopNight does not check Job dependencies when you attach a schedule. Attaching an off-hours schedule to a cluster that Jobs still run on is accepted without a warning, and the first sign of trouble is a cluster that keeps waking at night, or a Job run killed by the stop.
What ZopNight does give you is the inventory. Jobs are discovered as databricks-job resources alongside the clusters, so you can see what exists in each workspace before you schedule anything.
BEFORE ATTACHING AN OFF-HOURS SCHEDULE TO A CLUSTER:─────────────────────────────────────────────────────────1. List the Jobs in the workspace (discovered as databricks-job resources)2. For each Job, check in Databricks: which cluster it runs on when its trigger fires3. Mark every Job whose run time falls inside the cluster's proposed off-hours4. Pick Pattern A, B or C for each marked Job5. Only then attach the scheduleDoing this first is what keeps a 2 AM restart or a killed run from being your first signal.
Pattern A: change Job schedule
SIMPLE: change Job's schedule to match cluster availability
EXAMPLE: Cluster business-hours: 8 AM - 8 PM Mon-Fri Original Job: 2 AM daily (would wake the cluster during off-hours)
Resolution: move Job to 9 AM weekdays Or: move to off-peak time within business hours
PROS: Simplest fix Single change point No new patterns
CONS: Job timing may not match other constraints Data freshness implications Coordination with other jobs neededWhen Job timing is flexible: Pattern A is simplest.
Pattern B: overlapping cluster schedule
WAKE THE CLUSTER for the Job, then sleep again
EXAMPLE: Job ml-train-nightly: 2 AM Mon-Fri only Cluster schedule: 1:30 AM - 4 AM Mon-Fri (for nightly Job) Plus: 8 AM - 8 PM (interactive use)
T+0 Schedule cron fires at 1:30 AM TuesdayT+1s ZopNight starts ml-clusterT+3min Cluster reaches Running; available for JobsT+30min Job ml-train-nightly fires at 2 AMT+1.5h Job completesT+2.5h Schedule cron fires at 4 AM (stop cluster)T+2.5h Cluster fully stopped
TOTAL: cluster ran 2.5 hoursCOST: vs 24-hour always-on, ~90% savings on this Job's compute
PROS: Precise timing Minimal cluster runtime
CONS: Multi-schedule complexity Coordination with Job schedule changesPattern B is precise: cluster runs only when needed.
Pattern C: Job clusters
JOB CLUSTERS: Databricks's "one-time cluster" feature
CONFIGURATION: Job ml-train-nightly: Cluster setup: create new cluster, run, terminate
Each scheduled Job run: Creates new cluster (cold-start ~3 min) Runs the Job Terminates cluster
ELIMINATES dependency on long-running cluster
PROS: Most cost-efficient for sporadic Jobs No cluster scheduling complexity Independent of other cluster lifecycle
CONS: Cold-start latency per Job run Less flexibility for interactive cluster use Best when Job-only workloadFor production workloads with sporadic Jobs, this is often the cleanest pattern.
When the conflict is unavoidable
SOMETIMES A JOB needs the specific cluster: Uses local files Custom Python environment Specific cached state
CANNOT EASILY MOVE TO JOB CLUSTER
RESOLUTION: Pattern B (overlap schedule) Cluster runs for Job's window Stops outside that windowThe hard-dependency case requires Pattern B; no alternatives work cleanly.
Manual override for exceptions
SCENARIO: Tonight at 10 PM, emergency ML training run needed Cluster is scheduled off at 8 PM
OVERRIDE: force-on cluster from 9:30 PM to midnight Reason: "emergency ML training requested by data team"
EFFECT: Stop cron fires at 8 PM but is skipped (override in effect) Cluster stays running until midnight At midnight: override expires; natural scheduled state resumes
NO BACKFILL of skipped crons; simple state modelThe Override system (T1.M1.5) handles ad-hoc exceptions.
2. Demo
A team’s transition from cluster-always-on to scheduled:
BEFORE: ml-cluster: always-on, 12 nodes, ~$2,800/month
Jobs: ml-train-nightly: 2 AM daily, ~45 min runtime hourly-pipeline: every hour, ~5 min runtime weekly-report: Sun 11 PM, ~2 hours runtime
Cluster runs 24/7 to support sporadic Jobs
AUDIT before attaching the business-hours schedule: Team listed the workspace's discovered Jobs Checked each Job's cluster and trigger in Databricks All 3 Jobs run on ml-cluster during proposed off-hours
DECISION: mix of patterns ml-train-nightly: Pattern B (cluster wakes at 1:30 AM, stops at 4 AM) hourly-pipeline: Pattern C (use Job cluster: each hour spawns own) weekly-report: Pattern B (cluster wakes Sunday 10:30 PM, runs report, stops at 1 AM Monday)
WEEKLY SCHEDULE for ml-cluster: Mon-Fri: 8 AM - 8 PM: Running (interactive use) 1:30 AM - 4 AM (Mon-Fri): Running (for nightly train) Otherwise: Off Sat: Off entirely Sun: 10:30 PM - 1 AM Mon: Running (weekly report) Otherwise: Off
PER-JOB CONFIGURATION: ml-train-nightly: stays on existing cluster hourly-pipeline: switched to Job cluster (per Databricks UI) weekly-report: stays on existing cluster
ROLLOUT (gradual): Week 1: switch hourly-pipeline to Job cluster Verify hourly runs working Week 2: attach ml-cluster schedule Monitor weekday cluster lifecycle Week 3: monitor for the next weekly-report Sunday Verify cluster wakes; report runs; cluster stops
NEW COST: ~$1,400/month (50% reduction)ALL JOBS continue to workNO INCIDENTSThe mix of patterns optimized cost without breaking workflows.
3. Hands-on (5 min)
Audit your Databricks Job dependencies:
□ STEP 1: List Jobs Job 1: __________ Cluster: __________ Time: __________ Job 2: __________ Cluster: __________ Time: __________ Job 3: __________ Cluster: __________ Time: __________
□ STEP 2: Identify conflicts with proposed schedule Cluster proposed off-hours: __________ Jobs that would conflict: _____
□ STEP 3: Pick resolution per Job Job 1: □ A (change time) □ B (overlap) □ C (Job cluster) Job 2: same Job 3: same
□ STEP 4: Estimate cost impact Cluster runtime reduction: _____% Cost savings: $_____/mo
□ STEP 5: Roll out gradually First Job change: __________ Verification period: ___ week(s)A 15-minute exercise reveals the dependency map.
4. Knowledge check
Q1
A Job runs at 2 AM but the cluster is scheduled off 8 PM - 8 AM. What happens?
A. Job runs and the cluster stays off
B. The cluster starts on demand for the job
C. Discovery error
D. Job fails
Show answer
Correct: B. Databricks starts a terminated all-purpose cluster when a Job assigned to it is due, so the Job runs, but the off-hours saving is lost and a ZopNight stop that lands mid-run kills it. Use Pattern A (schedule cluster overlap), B (wake the cluster for the Job’s time) or C (use a Job cluster): cluster availability is the dependency.
Q2
Pattern C (Job clusters) is best for:
A. All workloads
B. Workloads with sporadic Job runs and no need for cluster state between runs
C. Long-running interactive analysis sessions throughout the working day
D. ML training only
Show answer
Correct: B. Each Job creates its own cluster, runs and terminates, independent of any scheduled cluster. For sporadic Jobs with no state, a Job cluster is the right fit.
Q3
You attach an off-hours schedule to a cluster that a 2 AM Job runs on. What does ZopNight do about the Job?
A. Nothing; you check the Jobs yourself
B. Lists the affected Jobs in a warning
C. Moves the Jobs onto Job clusters
D. Skips stops that overlap a Job run
Show answer
Correct: A. ZopNight does not check Job dependencies at attachment time. Audit the Jobs that run on the cluster first (they are discovered as databricks-job resources) and pick Pattern A, B or C for each before you attach the schedule.
5. Apply
ZopNight does not check Job dependencies when you attach a schedule to a Databricks cluster. Schedule design starts from the dependency map you build yourself.
For your team: audit Jobs before scheduling clusters; pick the right resolution per Job.
Module quiz
Complete M2.7 → 10-question module quiz.