Outcome
By the end of this lesson, you will be able to distinguish Databricks autoterm from ZopNight cluster scheduling, combine the two layers effectively, and handle the timing edge cases.
| Prerequisites | M2.7.L1 |
| Time | 9 minutes |
1. Concept
Databricks can already shut down an idle cluster by itself. You set a number of minutes, and after that long doing nothing the cluster stops.
That is Databricks’ own feature and it is worth having. ZopNight’s scheduling does something different, and the two are not in competition.
TWO LAYERS OF COST CONTROL:
AUTOTERM (Databricks-native): Mid-day idle handler Configurable per cluster (10-10000 minutes, or 0 to disable) Default: 120 min in the Databricks UI
ZOPNIGHT SCHEDULE: Daily lifecycle (start/stop times) Independent of activity Handles always-on clusters
Both work together; complementaryThe right setup combines both for best cost control.
Databricks autoterm: the baseline
CLUSTER CONFIG (Databricks-native): autotermination_minutes: 30
BEHAVIOR: Cluster is idle for 30 minutes → cluster terminates Cost stops during the off-period Cluster restarts on next use
EXAMPLE timing: 10:00 AM: cluster active 10:30 AM: no jobs running 11:00 AM: 30 min idle; cluster auto-terminates 2:00 PM: user starts new job 2:00 PM: cluster cold-starts; ~3 min 2:03 PM: cluster ready; job runsThis is the default cost optimization for interactive clusters. Most clusters should have autoterm set.
ZopNight cluster scheduling: daily lifecycle
CLUSTER CONFIG: Autotermination: 30 minutes (tightened from the 120-min UI default) ZopNight schedule: business-hours-staging (8 AM - 8 PM)
BEHAVIOR: ZopNight's schedule fires: Stop at 8 PM: terminates cluster regardless of activity (a Databricks cluster stop IS a terminate, so treat it as destructive for anything running on it; Start restarts the same cluster ID from the configuration Databricks keeps) Start at 8 AM: ensures cluster is ready for the day
Databricks autoterm still works: During business hours, an idle cluster still self-terminates after 30 minutes of no activityThe two layers work together. ZopNight handles the daily lifecycle; autoterm handles mid-day idle.
When ZopNight scheduling matters
For Always-on clusters or clusters with critical activity:
SCENARIO: ml-training-cluster Configured for "always-on" (autoterm = 0) Reason: nightly training jobs run automatically
WITHOUT ZopNight: Cluster runs 168 hours/week Cost: ~$2,800/month
WITH ZopNight: Schedule cluster off weekends Train Monday-Friday only Savings: 48 of 168 hours, ~29% = ~$800/month
SCENARIO: dev-experiments-cluster Used by 5 engineers, weekdays only Currently always-on (autoterm = 0) Engineers leave it running for "fast start"
WITHOUT ZopNight: Cluster runs 168 hours/week Cost: $1,500/month
WITH ZopNight: Schedule off-hours stop Engineers wait 3 min for restart when they start work Cost: $600/month (-60%)The clusters with autoterm=0 are the biggest opportunity. ZopNight adds the missing daily lifecycle.
Cluster start latency
TYPICAL START LATENCY: ML cluster (10 nodes): 3-5 minutes Data engineering cluster (4 nodes): 2-3 minutes Dev cluster (1 node): 1-2 minutes
WITH INSTANCE POOL (pre-warmed): Any cluster size: 1-2 minutes Pool nodes warm; ready to attach to clusterSchedule the start a few minutes before users need the cluster.
Auto-start workflow on schedule
T+0 Cron firesT+1s ZopNight: Databricks API: clusters/startT+2s Databricks acknowledgesT+30s Cluster transitioning to RUNNINGT+3 min Cluster reaches Running stateT+3 min Cluster ready for jobsT+3 min Notification fires (cluster ready)Total start latency: ~3 minutes typical. Schedule the start cron 5 minutes earlier to ensure availability.
Auto-stop workflow on schedule
T+0 Cron firesT+1s ZopNight: Databricks API: clusters/deleteT+5s Databricks acknowledgesT+30s Cluster fully stoppedT+30s Cost meter stopsT+30s Notification firesFaster than start. Total stop time: <1 minute.
Interactions with autoterm
SCENARIO: ZopNight scheduled stop at 8 PM But user is still active at 7:55 PM
BEHAVIOR: ZopNight schedule fires at 8 PM Cluster terminated regardless of activity User's session interrupted
THIS IS INTENTIONAL: Schedule = explicit hours Activity detection = autoterm's job
IF USER NEEDS OVERRIDE: Use the Override system (T1.M1.5) Force-on the cluster for a few hours Schedule resumes after override expiresThe override pattern handles the edge cases. Don’t rely on schedule to detect activity.
What survives a cluster stop
ZOPNIGHT saves no state for a cluster on schedule stop.
DATABRICKS keeps a terminated cluster's configuration under the same cluster ID (size, Spark configuration, libraries, init scripts). That retention is Databricks' own, not ZopNight's.
ON SCHEDULE START: ZopNight restarts the same cluster ID Cold-start latency only (3-5 min typical)
NO DATA LOSS: Cluster is compute-only (state in workspace) Notebooks, code, data unaffectedThe compute is ephemeral; the configuration lives in Databricks.
Schedule patterns for Databricks
PATTERN A: BUSINESS HOURS (most common) Start: 7:55 AM Mon-Fri (5 min before users) Stop: 8:30 PM Mon-Fri Off all weekend On ~62.9 of 168 hours; savings ~63% vs always-on
PATTERN B: CONSERVATIVE OFF-HOURS Start: 7:00 AM Mon-Fri Stop: 11:00 PM Mon-Fri Off weekends On 80 of 168 hours; savings ~52% vs always-on More forgiving of late work
PATTERN C: NIGHTLY BATCH WINDOW Start: 2:00 AM Mon-Fri (for nightly batches) Stop: 8:00 AM Mon-Fri (after batches done) Off all daytime + weekends For batch-only clusters Savings: ~80%
PATTERN D: ON-DEMAND ONLY (no scheduled start) Stop: every weekday at 8 PM (force cleanup) Start: only via user initiation For low-usage clusters Pre-warm via instance pool for fast startThe right pattern depends on usage pattern.
2. Demo
A team’s data engineering cluster scheduling:
WORKLOAD: data-engineering-cluster, 4 nodes Used by 12 data engineers, weekdays 8 AM - 8 PM Currently configured: autoterm = 60 min
OBSERVED COST PATTERN: Average runtime: 8 hours/day × 5 days = 40 hours/week Friday afternoons sometimes idle 30+ minutes; autoterm fires Saturday/Sunday: cluster off entirely (autoterm fires) Cost: ~$140/week
WITH ZOPNIGHT SCHEDULE: business-hours-eng: 8 AM start, 8 PM stop (weekdays)
REVISED for late-afternoon flexibility: business-hours-eng: 7:55 AM start, 8:30 PM stop Reason: Friday afternoons sometimes run late
Cost: MORE hours, not fewer. The cluster is now held on ~62.9 hours/week (12h35m x 5) instead of the observed ~40
Real benefit: Predictable cluster availability for the team No waiting for cluster start when beginning work Clean cost reporting (fewer random autoterm cycles)
COMPARISON: Before ZopNight: Cost-per-week $140 (~40 hours at ~$3.50/hour) Engineer wait time: 3 min x 3/day x 5 days x 12 engineers = 540 engineer-minutes/week (45 per engineer)
After ZopNight: Cost-per-week ~$220 (~62.9 hours) Engineer wait time: near 0 (cluster ready at 7:55 AM; only a mid-day autoterm stop still means a cold start) = ~$80/week buys back up to ~9 engineer-hours/weekHere the schedule raises cost rather than cutting it. That is a fair trade for the team’s time, but make it knowingly: a schedule saves money only when it holds a cluster on for fewer hours than it was already running.
3. Hands-on (5 min)
Schedule a Databricks cluster:
□ STEP 1: Pick a cluster Cluster: __________ Autoterm setting: ___ minutes Currently always-on? □ Yes □ No Usage pattern: __________
□ STEP 2: Choose schedule pattern □ Business hours □ Conservative off-hours □ Nightly batch □ On-demand only
□ STEP 3: Set times Start cron: __________ Stop cron: __________
□ STEP 4: Attach Schedule attached to cluster Effective from: __________
□ STEP 5: Monitor first cycle Did cluster start as expected? □ Yes □ No Engineer impact: __________ Cost impact: $_____/weekA 10-minute exercise per cluster. Track cycle for first week.
Do it through MCP. The same task you just did in the console, asked in one sentence.
BEFORE A ZopNight account with one cloud connected. Azure Databricks connected, with at least one non-production cluster.ASK "Find our Databricks clusters and put the non-production ones on the nightly schedule."CHECK that job clusters were not swept up with interactive ones. The distinction in L1 is the one that matters here.Tools behind it: list_resources (read, Explore), create_schedule (write, tier 2, reversible), attach_resource_to_schedule (write, tier 2, reversible). Connection details are in the MCP integration docs.
4. Knowledge check
Q1
Databricks autoterm vs ZopNight scheduling:
A. Conflict
B. Same thing
C. Autoterm replaces the ZopNight schedule
D. Complementary: autoterm handles mid-day idle
Show answer
Correct: D. Autoterm handles mid-day idle; ZopNight handles the daily window. The two layers cover different cost concerns and work in parallel.
Q2
A cluster takes 3 minutes to start. Schedule cron should fire:
A. ~3-5 minutes earlier than user need (to account for startup latency)
B. At the moment users need it
C. Exactly on the hour, whatever the cluster startup time might be
D. 30 minutes earlier
Show answer
Correct: A. Schedule the start cron a few minutes ahead so the cluster is ready when users arrive. Otherwise they wait briefly at 8 AM.
Q3
A team uses cluster past the schedule’s stop time. The schedule fires and stops it:
A. ZopNight backs off
B. The stop is skipped while a job is running
C. Stop fires regardless
D. Discovery fails
Show answer
Correct: C. Schedules are explicit; activity detection is autoterm’s job. The Override system is the explicit mechanism for “keep this running past the schedule”: use that instead of relying on the schedule to detect activity.
5. Apply
Cluster scheduling is per-cluster. Attach via the cluster’s detail page.
For your team: combine autoterm + ZopNight; use Override for edge cases; document the pattern.