Terminate Databricks Clusters on a Schedule

Databricks compute has no “stop until morning” state. The only off switch is terminate, which releases the cloud VMs and keeps the configuration so the same compute can be started again. This guide finds compute that never terminates by itself, gives it an inactivity limit, and adds a hard evening cutoff.

Know which compute actually needs this

• Job compute starts when its task run begins and terminates after the task runs (or after the last task sharing it). There is nothing to schedule.
• All-purpose compute runs until someone terminates it or auto-termination fires. This is the target.
• SQL warehouses have their own Auto Stop setting and their own stop command.

Databricks is explicit that idle compute keeps accumulating DBU and cloud instance charges during the inactivity period before termination, and that idle SQL warehouses do the same until they stop.

1. List all-purpose compute with no auto-termination

With the Databricks CLI authenticated to the workspace:

databricks clusters list -o json \
  | jq -r '.[] | select((.autotermination_minutes // 0) == 0) | [.cluster_id, .cluster_name, .state] | @tsv'

clusters list returns pinned and active compute plus anything terminated in the last 30 days. An autotermination_minutes of 0 means auto-termination is off.

2. Turn on auto-termination

In the workspace sidebar click Compute, open the cluster’s details page and edit its configuration: tick Terminate after … minutes of inactivity and enter a value between 10 and 10,000. Editing a running cluster (anything beyond size and permissions) restarts it, so do this when nobody is attached.

What counts as activity: Spark jobs, Structured Streaming, JDBC calls and web terminal activity. Commands run over SSH outside Spark do not count, and Databricks says compute running DStreams should not use auto-termination.

3. Add a fixed evening termination

Auto-termination never fires on a cluster that a scheduled notebook or a JDBC client keeps busy. For a guaranteed cutoff, run the terminate command from any scheduler that holds a workspace token:

# 19:30 on weekdays, from your scheduler of choice
databricks clusters delete 0923-164208-meows279 --no-wait

Despite the name, clusters delete terminates: the compute ends in TERMINATED and can be started again. Do not confuse it with clusters permanent-delete.

Start it again in the morning, keeping its ID and last size:

databricks clusters start 0923-164208-meows279

4. Stop SQL warehouses the same way

databricks warehouses stop 1234567890abcdef
databricks warehouses start 1234567890abcdef
databricks warehouses edit 1234567890abcdef --auto-stop-mins 30

Auto Stop defaults to 45 minutes on pro and classic warehouses (minimum 10) and 10 minutes on serverless.

5. Verify

databricks clusters get 0923-164208-meows279 -o json | jq -r .state

After the cutoff it should print TERMINATED. One caution: unless compute is pinned or restarted, Databricks permanently deletes it 30 days after termination, so a cluster used only monthly will lose its configuration unless an admin pins it.

Scheduling it in ZopNight

ZopNight can put all-purpose clusters, pools and SQL warehouses on the same cron schedules, groups and overrides it uses for VMs; its cluster stop is a terminate. Jobs and model-serving endpoints are discovered but not schedulable. AWS and GCP workspaces connect as a standalone Databricks account over an OAuth M2M service principal; Azure workspaces come through the connected Azure subscription with the “Zopnight Databricks Access” role granted. See Databricks cluster scheduling and SQL warehouse scheduling.