Databricks Cost Optimization: DBUs, Clusters and Warehouses

Databricks cost optimization is reducing what a Databricks workspace spends on compute without slowing the jobs, notebooks, and dashboards that use it. It starts with how Databricks bills. A Databricks Unit (DBU) is a normalized unit of processing power used for measurement and pricing, and the DBUs a workload consumes depend on the compute it uses and, for some products, the data it processes. On classic compute, which runs in your own cloud account, total cost is DBUs plus the virtual machine, disk, and network costs your cloud provider charges.

Lever 1: stop idle compute

All-purpose clusters. Set auto termination: an inactivity period in minutes after which Databricks terminates the compute if no command has run. Termination stops the cost and keeps the configuration so the cluster can be started again. Two caveats: unless a compute is pinned or restarted, Databricks permanently deletes it 30 days after termination, and compute running DStreams does not report activity, so it can be terminated mid-stream.

SQL warehouses. Idle SQL warehouses keep accumulating DBU and cloud instance charges until they stop. Auto Stop defaults to 45 minutes for pro and classic warehouses, with a 10-minute minimum, and to 10 minutes for serverless warehouses, with a 5-minute minimum in the UI (1 minute through the API).

Instance pools. Databricks charges no DBUs for idle instances in a pool, but your cloud provider still bills them. A pool’s minimum idle setting is warm capacity you pay for around the clock.

Lever 2: run work on the right compute

Databricks’ cost best practices state that non-interactive workloads cost significantly less on job compute than on all-purpose compute. Scheduled ETL that runs on a shared all-purpose cluster pays the interactive rate and keeps that cluster alive. A new job compute also terminates when the job completes.

Photon can shorten runtimes, but Photon instance types consume DBUs at a different rate than the non-Photon runtime, so evaluate regularly scheduled jobs for both speed and total cost.

Lever 3: cheaper machines

Databricks supports spot instances for cluster workers and recommends that the first instance, the Spark driver, stay on-demand. Spot fits workloads where a longer run after an eviction is acceptable.

Lever 4: guardrails with compute policies

Compute policies are JSON rules over any attribute the Clusters API controls. They can set a default auto-termination time, forbid or require pools, enforce Photon, cap dbus_per_hour with a maximum, and require specific custom_tags values. Tags on compute propagate to usage logs and to cloud provider resources, which is what makes Databricks spend attributable by team.

Databricks in ZopNight

On AWS and GCP, ZopNight connects to Databricks as a standalone account using an OAuth M2M service principal; on Azure, workspaces are discovered through the connected Azure subscription once the workspace-admin access role is granted. Clusters, instance pools, and SQL warehouses can be put on the same schedules, groups, and overrides used for VMs; a cluster stop is a terminate, and jobs and model-serving endpoints are discovered but not schedulable. Recommendations cover the levers above on all three clouds: missing or weak auto-termination, SQL warehouses without auto-stop, pool minimum idle, always-on serving endpoints, oversized clusters, autoscaling or Photon disabled, jobs on all-purpose clusters, on-demand workers that could be spot, missing cluster policy or cost tags, and orphaned jobs or warehouses. See scheduling Databricks clusters and scheduling SQL warehouses.