Databricks Cost Optimization — Clusters, Pools and SQL Warehouses on Autopilot
What does the Databricks integration do?
Databricks is covered by 15 rules over 6 object types, 3 of them schedulable. Two meters run in parallel: DBUs charged by Databricks and the underlying cloud VM charged by your provider. An idle instance pool bills the VM with no DBU attached, which is both the point and the risk.
Coverage by category
3 categories have fewer than 3 resource types, so they get no category page. Their resources are listed here.
Databricks compute quietly accumulates: all-purpose clusters left running, warm instance pools holding idle VMs, SQL warehouses that never auto-stop. ZopNight connects at the account level with a single OAuth service principal and inventories every workspace on AWS, Azure and Google Cloud alike: clusters, instance pools, SQL warehouses, jobs and model-serving endpoints. It schedules interactive compute off-hours, derives cluster utilization from the underlying VMs, and ships 15 Databricks-specific recommendations per cloud.
Account-level OAuth on AWS and GCP, Entra on Azure
On AWS and GCP, Databricks is a standalone connection: you create an account-level service principal with OAuth machine-to-machine credentials, and ZopNight talks to the account console (accounts.cloud.databricks.com or accounts.gcp.databricks.com) to enumerate workspaces, then to each workspace API for its compute. On Azure, no separate credential is needed. ZopNight finds workspaces through your connected Azure subscription and authenticates with Entra ID, once you add its service principal to each workspace’s admins group. Cluster costs are computed from the underlying VM rates (workers plus driver, with autoscale ranges billed at the midpoint) and reconciled against your cloud bill through the billing overlay, so cluster, pool and warehouse rows re-slice the workspace’s real spend without double counting.
Service principal, OAuth secret, workspace admin
- In the Databricks account console, create a service principal and grant it Account admin (Roles tab).
- Generate OAuth secret (client ID + secret; secrets live up to 730 days).
- For each workspace to manage, add the service principal as a workspace Admin.
- In ZopNight choose Add Cloud Account > Databricks, pick the cloud (AWS or GCP), and enter account ID, client ID, secret. Azure path: connect your Azure subscription first, then just add the SP to the workspace admins groups; no extra credential is required.
Workspace inventory and off-hours scheduling
Account-wide workspace inventory with per-workspace access health. Discovery of clusters (interactive), instance pools, SQL warehouses, jobs and provisioned model-serving endpoints. Off-hours scheduling of clusters, pools and warehouses (workspace-wide stop on Azure). Derived cluster CPU utilization from worker VMs. 15 recommendation rules per cloud: idle and oversized clusters, warm pools, autoscale hygiene, old runtime versions, Spot adoption, missing cost tags and endpoint utilization.
Limits worth knowing before you connect
Stopping a Databricks cluster terminates it (Databricks semantics — configuration is preserved, the VMs are released). Job/pipeline-internal clusters are excluded from inventory by design. The DBU license layer is not modeled separately; VM costs plus your cloud bill cover actuals. Account admin is required to see all workspaces.
Databricks integration: common questions
Do I need a separate Databricks credential on Azure?
No. ZopNight finds Azure workspaces through the Azure subscription you already connected and authenticates with Entra ID, once you add its service principal to each workspace's admins group. On AWS and GCP you create an account-level service principal with OAuth machine-to-machine credentials instead.
What happens when ZopNight stops a Databricks cluster?
It terminates, because that is what stopping means in Databricks. The cluster configuration is preserved and the underlying VMs are released.
Why does an idle instance pool still cost money?
Two meters run in parallel: DBUs charged by Databricks, and the underlying cloud VM charged by your provider. A warm pool holds VMs with no DBU attached, so it looks idle in Databricks while billing on the cloud side. That is both the point of a pool and its risk.
Recommendations
15 live rule families evaluate Databricks spend. Each page documents the metric, threshold, window, and the IAM actions the check needs.
Scheduling
3 Databricks resource types can be stopped and started on a schedule.