Databricks
Connect a Databricks workspace on AWS, GCP, or Azure, schedule its clusters, pools, and SQL warehouses off-hours, and act on Databricks cost recommendations.
In many data estates, Databricks compute is a large line item that is rarely scheduled. All-purpose clusters and SQL warehouses left running overnight cost money the same way an idle VM does. ZopNight discovers Databricks compute on AWS, GCP, and Azure, schedules it off-hours, and raises Databricks-specific cost recommendations.
Cloud Estate → Inventory filtered to the Databricks types: workspaces and their compute across all three clouds, each with cost, status, and start/stop.
Before you start
- A ZopNight role that can add cloud accounts (the
cloud-account:createpermission; Admin has it by default). - On AWS or GCP: your Databricks Account ID, and an account-level OAuth service principal (client ID and secret) that can list clusters in the workspaces you want covered. ZopNight probes workspace access at connect time, so a principal without it fails the connect.
- On Azure: a connected Azure cloud account for the subscription that holds the workspace, and permission to grant the “Zopnight Databricks Access” role in the workspace. Reading clusters, pools, and warehouses needs workspace-admin permission.
- To stop and start compute from schedules or recommendations: the account’s permission level set to Read + Write. Read-only accounts still get discovery and recommendations.
How it works
How you connect depends on the cloud the workspace runs on.
| Cloud | Connection model |
|---|---|
| AWS & GCP (standalone connection) | Add Databricks as its own connection in the credential wizard: the Databricks Account ID and an OAuth M2M client ID/secret. ZopNight uses the account console for the cloud you pick (accounts.cloud.databricks.com for AWS, accounts.gcp.databricks.com for GCP). No separate cloud account is required. |
| Azure (rides the subscription) | There is no separate Databricks connection on Azure. Workspaces are discovered as resources under your connected Azure subscription. Grant the workspace-admin “Zopnight Databricks Access” role so ZopNight can read the workspace’s clusters, pools, and warehouses. (Zopnight Databricks Access is the verbatim role string to enter in Azure; type it exactly as shown.) |
What gets discovered
ZopNight discovers six Databricks resource types, grouped under their workspace:
| Type | Notes |
|---|---|
databricks (workspace) | The parent; all other objects nest under it |
databricks-cluster | All-purpose and job clusters, with size/worker config, autoscaling, and auto-termination settings |
databricks-pool | Instance pools, with min-idle configuration |
databricks-sqlwarehouse | SQL warehouses, with auto-stop configuration |
databricks-job | Jobs (discovered, read-only) |
databricks-model-endpoint | Model-serving endpoints (discovered, read-only) |
You’ll see which objects are running vs terminated/stopped, their size and worker config, autoscaling and auto-termination settings, and cost tags, all in the standard Inventory view.
Connect a workspace
AWS or GCP: add a Databricks connection
In Settings → Cloud Accounts → Add Cloud Account, pick Databricks, choose the cloud, and enter the Account ID and the service principal’s OAuth client ID and secret.
Azure: connect the subscription and grant the role
Connect the Azure subscription that holds the workspace as a cloud account, then grant the “Zopnight Databricks Access” role in the workspace.
Check the inventory
Workspaces and their compute appear in Cloud Estate → Inventory, grouped under each workspace.
Scheduling off-hours stops
Databricks compute uses the same schedules, groups, and overrides as VMs.
- Put SQL warehouses and all-purpose clusters on an off-hours schedule so they stop at night and at weekends and start again for business hours.
- Scale instance pools down outside working hours.
Start/stop runs over the workspace’s OAuth M2M service principal (on AWS/GCP there’s no underlying cloud account to act through).
Databricks recommendations
The same 15 Databricks rules run on each cloud (Azure RC-22xx, AWS RC-23xx, GCP RC-24xx). They catch:
- Clusters missing auto-termination
- SQL warehouses without auto-stop
- Model-serving endpoints left always on
- Instance pools holding too many idle VMs
- Oversized clusters (AWS and Azure, where per-cluster CPU is available)
- Autoscaling or Photon disabled where it would cut cost
- Jobs running on expensive all-purpose clusters
- On-demand workers that could be spot
- Missing cluster policy
- Missing cost-allocation tags
- Orphaned jobs and warehouses
Cluster rules look only at interactive clusters (not the system clusters Databricks runs for jobs, pipelines, and warehouses), and stopped or terminated compute is skipped, so findings point at compute that is actually billing. See the rule catalogue.
Acting on Databricks findings
Most Databricks findings are configuration changes (auto-termination, auto-stop, pool min-idle, cluster policy, tags), and each links straight to the object in your Databricks workspace, where you make the change. To stop compute, put it on a schedule or use manual start/stop from the Inventory; both run over the workspace’s OAuth M2M service principal, and cluster stop terminates the cluster. Jobs and model endpoints have no start/stop action. See Recommendations.
Troubleshooting
The connect fails on AWS or GCP
The service principal can’t reach the workspace’s data plane, so the ListClusters probe fails and nothing is saved. Give the principal access to list clusters in each workspace you want covered, then connect again.
An Azure workspace appears, but no clusters, pools, or warehouses
Reading a workspace’s compute needs workspace-admin permission. Grant the “Zopnight Databricks Access” role in the workspace, typed exactly as shown.
A cluster was terminated rather than paused
This is expected. Databricks has no pause for clusters, so a scheduled or manual stop terminates the cluster and a start recreates it.
There is no start/stop toggle on a job or model endpoint
Jobs and model-serving endpoints are read-only in ZopNight. They are discovered and get recommendations, but have no start/stop action.