# Databricks Cost Optimization — Clusters, Pools and SQL Warehouses on Autopilot

> One service principal connects your whole Databricks account. ZopNight inventories clusters, pools, SQL warehouses and jobs across every workspace on AWS, Azure and GCP.

Source: https://zop.dev/integrations/databricks
Updated: 2026-08-20

---

## Common questions

### Do I need a separate Databricks credential on Azure?

No. ZopNight finds Azure workspaces through the Azure subscription you already connected and authenticates with Entra ID, once you add its service principal to each workspace's admins group. On AWS and GCP you create an account-level service principal with OAuth machine-to-machine credentials instead.

### What happens when ZopNight stops a Databricks cluster?

It terminates, because that is what stopping means in Databricks. The cluster configuration is preserved and the underlying VMs are released.

### Why does an idle instance pool still cost money?

Two meters run in parallel: DBUs charged by Databricks, and the underlying cloud VM charged by your provider. A warm pool holds VMs with no DBU attached, so it looks idle in Databricks while billing on the cloud side. That is both the point of a pool and its risk.

Databricks compute quietly accumulates: all-purpose clusters left running, warm instance pools holding idle VMs, SQL warehouses that never auto-stop. ZopNight connects at the account level with a single OAuth service principal and inventories every workspace on AWS, Azure and Google Cloud alike: clusters, instance pools, SQL warehouses, jobs and model-serving endpoints. It schedules interactive compute off-hours, derives cluster utilization from the underlying VMs, and ships 15 Databricks-specific recommendations per cloud.

## Account-level OAuth on AWS and GCP, Entra on Azure

On AWS and GCP, Databricks is a standalone connection: you create an account-level service principal with OAuth machine-to-machine credentials, and ZopNight talks to the account console (accounts.cloud.databricks.com or accounts.gcp.databricks.com) to enumerate workspaces, then to each workspace API for its compute. On Azure, no separate credential is needed. ZopNight finds workspaces through your connected Azure subscription and authenticates with Entra ID, once you add its service principal to each workspace's admins group. Cluster costs are computed from the underlying VM rates (workers plus driver, with autoscale ranges billed at the midpoint) and reconciled against your cloud bill through the billing overlay, so cluster, pool and warehouse rows re-slice the workspace's real spend without double counting.

## Service principal, OAuth secret, workspace admin

1. In the Databricks account console, create a service principal and grant it Account admin (Roles tab).
2. Generate OAuth secret (client ID + secret; secrets live up to 730 days).
3. For each workspace to manage, add the service principal as a workspace Admin.
4. In ZopNight choose Add Cloud Account > Databricks, pick the cloud (AWS or GCP), and enter account ID, client ID, secret. Azure path: connect your Azure subscription first, then just add the SP to the workspace admins groups; no extra credential is required.

## Workspace inventory and off-hours scheduling

Account-wide workspace inventory with per-workspace access health. Discovery of clusters (interactive), instance pools, SQL warehouses, jobs and provisioned model-serving endpoints. Off-hours scheduling of clusters, pools and warehouses (workspace-wide stop on Azure). Derived cluster CPU utilization from worker VMs. 15 recommendation rules per cloud: idle and oversized clusters, warm pools, autoscale hygiene, old runtime versions, Spot adoption, missing cost tags and endpoint utilization.