Databricks
Connect a Databricks workspace on AWS, GCP, or Azure; discover clusters, pools, and warehouses; schedule off-hours stops; and get Databricks cost recommendations.
Databricks compute is one of the largest and least-scheduled line items in a modern data estate; all-purpose clusters and SQL warehouses left running overnight burn money the same way an idle VM does. ZopNight discovers Databricks compute, schedules it off-hours, and surfaces Databricks-specific cost recommendations, across AWS, GCP, and Azure.
The Resources view filtered to the Databricks types; workspaces and their compute discovered across all three clouds, each with cost and running state.
Connecting a workspace
How you connect depends on the cloud the workspace runs on.
AWS & GCP: standalone connection
Add Databricks as its own connection in the credential wizard: Databricks Account ID, an OAuth M2M client ID/secret, and the workspace host. ZopNight recognises the cloud automatically from the host (accounts.cloud.databricks.com for AWS, accounts.gcp.databricks.com for GCP). No separate cloud account is required.
Azure: rides the subscription
There is no separate Databricks connection on Azure. Workspaces are discovered as resources under your connected Azure subscription. Grant the workspace-admin “Zopnight Databricks Access” role so ZopNight can read the workspace’s clusters, pools, and warehouses. (Zopnight Databricks Access is the verbatim role string to enter in Azure; type it exactly as shown.)
What gets discovered
ZopNight discovers six Databricks resource types, grouped under their workspace:
| Type | Notes |
|---|---|
databricks (workspace) | The parent; all other objects nest under it |
databricks-cluster | All-purpose and job clusters, with size/worker config, autoscaling, and auto-termination settings |
databricks-pool | Instance pools, with min-idle configuration |
databricks-sqlwarehouse | SQL warehouses, with auto-stop configuration |
databricks-job | Jobs (discovered, read-only) |
databricks-model-endpoint | Model-serving endpoints (discovered, read-only) |
You’ll see which objects are running vs terminated/stopped, their size and worker config, autoscaling and auto-termination settings, and cost tags; all in the standard Resources view.
Scheduling off-hours stops
Databricks compute uses the same schedules, groups, and overrides as VMs.
- Put SQL warehouses and all-purpose clusters on an off-hours schedule so they stop nights/weekends and start again at business hours.
- Scale instance pools down outside working hours.
Start/stop runs over the workspace’s OAuth M2M service principal (on AWS/GCP there’s no underlying cloud account to act through).
Databricks recommendations
A provider-parameterised rule family fires across all three clouds (Azure RC-22xx, AWS RC-23xx, GCP RC-24xx). It catches:
- Clusters missing auto-termination
- SQL warehouses without auto-stop
- Model-serving endpoints left always-on
- Instance pools holding too many idle VMs
- Oversized clusters
- Autoscaling or Photon disabled where it would cut cost
- Jobs running on expensive all-purpose clusters
- On-demand workers that could be spot
- Missing cluster policy
- Missing cost-allocation tags
- Orphaned jobs and warehouses
Azure adds a metric-driven idle-cluster rule on top of the shared family. See the rule catalogue.
Acting on Databricks findings
One-click stop works on clusters, pools, and SQL warehouses, executed over the workspace’s OAuth M2M service principal. Cluster stop terminates the cluster. Jobs and model endpoints have no start/stop action; their recommendations render as advisory playbooks. See Recommendations and Auto-remediation.