Scheduling Interactive Cluster
Can ZopNight schedule Interactive Cluster?
Databricks offers nothing gentler than Terminate for an interactive cluster, so the scheduled stop is destructive by nature: execution contexts, cached DataFrames and runtime-installed libraries vanish, and restart provisions brand-new VMs. Cluster configuration survives, restarts take 5-10 minutes, and job clusters are out of scope entirely.
How the stop works
| Field | Value |
|---|---|
| Behaviour | Stop issues a cluster Terminate, which is destructive to cluster state; restart recreates compute on resume. Applies to interactive clusters only |
Example schedules
- 0 8 * * 1-5 — Business Hours Start: Start at 8:00 AM on weekdays
- 0 18 * * 1-5 — Business Hours Stop: Stop at 6:00 PM on weekdays
- 0 22 * * * — Night Shutdown: Stop at 10:00 PM every day
- 0 6 * * 1-5 — Morning Startup: Start at 6:00 AM on weekdays
- 0 20 * * 5 — Weekend Shutdown: Stop at 8:00 PM on Friday
- 0 7 * * 1 — Weekend Startup: Start at 7:00 AM on Monday
Terminate is the only stop there is
An interactive cluster cannot be paused or hibernated; the platform’s vocabulary ends at Terminate, and that is the verb the schedule issues. The driver and workers are destroyed and their cloud VMs released, which stops both the DBU meter and the underlying VM meter in the same moment. Restart later recreates compute from the cluster’s stored definition.
What termination erases
Everything that lived on the running cluster: notebook execution state and variables, cached and persisted DataFrames, files written to cluster-local disk, and any library installed at runtime from a notebook session. The configuration-level things survive and are reapplied on restart: cluster spec, cluster-scoped init scripts, libraries declared on the cluster config. A team whose notebooks assume yesterday’s temp tables will discover the difference on the first scheduled morning.
Restart is a rebuild, and it takes minutes
Starting a terminated cluster means acquiring fresh VMs, booting the runtime, running init scripts and reinstalling declared libraries, typically 5-10 minutes before the first cell runs. Users who arrive at 9:00 should find clusters started by a cron at 8:45, or accept pressing Start themselves and waiting.
Interactive only, and that is the right scope
The schedule touches all-purpose clusters, the ones humans attach notebooks to and then abandon at dinner time. Job clusters are excluded because they are already self-cleaning: created per run, terminated at completion. Scheduled production jobs therefore run overnight exactly as before, while the abandoned notebook clusters stop burning double meters.
Auto-termination exists; the schedule closes its gaps
Native auto-termination kills clusters after an idle threshold, and every interactive cluster should have it set. Its blind spot is work that looks busy: a streaming query left running, a cell stuck in a retry loop, a scheduled notebook poking the cluster hourly. The calendar stop is indifferent to apparent busyness, which is exactly what an end-of-day policy needs.