Enforce Databricks Auto-Termination with Compute Policies

Setting auto-termination on each cluster works until someone creates the next one without it. Databricks compute policies move that decision to the workspace admin: a policy definition can fix autotermination_minutes, hide the field from the UI, and cap the settings that drive hourly cost. This guide writes one, publishes it with the CLI, and checks who can still bypass it.

Before you start

• Workspace admin rights. By default only admins have permissions on policies; everyone else must be granted access.
• The Databricks CLI configured for the workspace (databricks auth login or a profile in ~/.databrickscfg).

1. Write the policy definition

Save this as team-interactive.json. It fixes auto-termination at 60 minutes and hides it, limits autoscaling to 8 workers, and makes a cost tag mandatory:

{
  "autotermination_minutes": { "type": "fixed", "value": 60, "hidden": true },
  "autoscale.max_workers":   { "type": "range", "maxValue": 8, "defaultValue": 4 },
  "custom_tags.team":        { "type": "unlimited", "isOptional": false }
}

Rules follow Databricks’ policy reference: fixed pins a value, range sets minValue and/or maxValue, unlimited makes an attribute required or gives it a default. In the policy reference, an autotermination_minutes of 0 means no auto-termination, which is exactly what you are preventing.

2. Create it

databricks cluster-policies create \
  --name "team-interactive" \
  --definition "$(cat team-interactive.json)" \
  --max-clusters-per-user 2

--definition takes the policy JSON as a string. --max-clusters-per-user limits how many active clusters one user can run under the policy.

3. Grant it and close the bypass

Open the policy’s overview page, click See all permissions, and grant the relevant groups permission to use it. Then check the entitlement that undoes all of this: Databricks notes that a user with unrestricted cluster creation permission also has access to the Unrestricted policy and can create fully configurable compute. Remove that entitlement from groups that should only use team-interactive.

4. Deal with clusters that already exist

Databricks states that when you edit a policy, compute already created under it is not automatically updated; the policy page offers Enforce policy compliance to bring some or all of it in line. Clusters created before any policy existed need editing onto the policy by hand, so list the ones without a limit:

databricks clusters list -o json \
  | jq -r '.[] | select((.autotermination_minutes // 0) == 0) | .cluster_name'

Edit each onto the policy from its details page. Editing a running cluster restarts it.

5. Keep configurations you still need

Terminated compute that is neither pinned nor restarted is permanently deleted 30 days after termination, and because the system does it, the deletion is not in the audit log. An admin can pin up to 100 compute resources:

databricks clusters pin 0923-164208-meows279

6. Pools: the other idle cost

Instance pools hold ready VMs. Databricks charges no DBUs while instances sit idle in a pool, but the cloud provider still bills those VMs. Keep the pool’s minimum idle instances low outside working hours, or delete the pool; deleting terminates its idle instances, and running clusters attached to it keep running but cannot take new instances from it.

Verify

Create a test cluster as a non-admin member of the granted group. The auto-termination field should not appear, and databricks clusters get CLUSTER_ID -o json | jq .autotermination_minutes should return 60.

What ZopNight adds

ZopNight’s Databricks recommendations run across AWS, Azure and GCP workspaces and flag compute missing auto-termination, SQL warehouses without auto-stop, pools holding too many idle VMs, missing cluster policies and missing cost tags. It can also scale pools down and stop clusters and warehouses on a cron schedule. Rule details are in the Databricks cost optimization guide, and pool scheduling is on Databricks pool scheduling.