# Databricks

> Connect a Databricks workspace on AWS, GCP, or Azure; discover clusters, pools, and warehouses; schedule off-hours stops; and get Databricks cost recommendations.

Source: https://zop.dev/docs/zopnight/concepts/databricks

---

Databricks compute is one of the largest and least-scheduled line items in a modern data estate; all-purpose clusters and SQL warehouses left running overnight burn money the same way an idle VM does. ZopNight discovers Databricks compute, schedules it off-hours, and surfaces Databricks-specific cost recommendations, across AWS, GCP, and Azure.

![ZopNight Resources view filtered to the Databricks types (databricks, databricks-cluster, databricks-pool, databricks-sqlwarehouse) showing ml-workspace and cluster resources discovered across Azure, AWS, and GCP with per-resource MTD cost and running state](https://storage.googleapis.com/zopdev-blog-resources/1/files/originals/20260707/99c7b5af-c2f4-43ab-85ca-e3e9a01e00a2-databricks.png)

*The Resources view filtered to the Databricks types; workspaces and their compute discovered across all three clouds, each with cost and running state.*

## Connecting a workspace

How you connect depends on the cloud the workspace runs on.

**AWS & GCP: standalone connection**

    Add Databricks as its own connection in the credential wizard: Databricks **Account ID**, an OAuth **M2M** client ID/secret, and the workspace host. ZopNight recognises the cloud automatically from the host (`accounts.cloud.databricks.com` for AWS, `accounts.gcp.databricks.com` for GCP). No separate cloud account is required.

**Azure: rides the subscription**

    There is no separate Databricks connection on Azure. Workspaces are discovered as resources under your connected Azure subscription. Grant the workspace-admin **"Zopnight Databricks Access"** role so ZopNight can read the workspace's clusters, pools, and warehouses. (`Zopnight Databricks Access` is the verbatim role string to enter in Azure; type it exactly as shown.)

**Note**

On AWS and GCP, ZopNight verifies workspace-level (data-plane) access at connect time with a `ListClusters` probe. A successful connect is the proof the credential works.

## What gets discovered

ZopNight discovers six Databricks resource types, grouped under their workspace:

| Type | Notes |
|---|---|
| `databricks` (workspace) | The parent; all other objects nest under it |
| `databricks-cluster` | All-purpose and job clusters, with size/worker config, autoscaling, and auto-termination settings |
| `databricks-pool` | Instance pools, with min-idle configuration |
| `databricks-sqlwarehouse` | SQL warehouses, with auto-stop configuration |
| `databricks-job` | Jobs (discovered, read-only) |
| `databricks-model-endpoint` | Model-serving endpoints (discovered, read-only) |

You'll see which objects are running vs terminated/stopped, their size and worker config, autoscaling and auto-termination settings, and cost tags; all in the standard [Resources](https://zop.dev/docs/zopnight/concepts/resources) view.

## Scheduling off-hours stops

Databricks compute uses the same [schedules, groups, and overrides](https://zop.dev/docs/zopnight/concepts/scheduling) as VMs.

- Put **SQL warehouses** and **all-purpose clusters** on an off-hours schedule so they stop nights/weekends and start again at business hours.
- Scale **instance pools** down outside working hours.

Start/stop runs over the workspace's OAuth M2M service principal (on AWS/GCP there's no underlying cloud account to act through).

**Warning**

Stopping a Databricks **cluster** terminates it; Databricks has no "pause" for clusters, so a stop is destructive in the sense that the cluster is torn down and recreated on start. **Jobs and model-serving endpoints are read-only**: ZopNight discovers them and recommends on them, but does not start/stop them.

## Databricks recommendations

A provider-parameterised rule family fires across all three clouds (Azure RC-22xx, AWS RC-23xx, GCP RC-24xx). It catches:

- Clusters missing auto-termination
- SQL warehouses without auto-stop
- Model-serving endpoints left always-on
- Instance pools holding too many idle VMs
- Oversized clusters
- Autoscaling or Photon disabled where it would cut cost
- Jobs running on expensive all-purpose clusters
- On-demand workers that could be spot
- Missing cluster policy
- Missing cost-allocation tags
- Orphaned jobs and warehouses

Azure adds a metric-driven idle-cluster rule on top of the shared family. See the [rule catalogue](https://zop.dev/docs/zopnight/optimization/recommendation-rules#databricks).

## Acting on Databricks findings

One-click **stop** works on clusters, pools, and SQL warehouses, executed over the workspace's OAuth M2M service principal. Cluster stop terminates the cluster. Jobs and model endpoints have no start/stop action; their recommendations render as advisory playbooks. See [Recommendations](https://zop.dev/docs/zopnight/optimization/recommendations) and [Auto-remediation](https://zop.dev/docs/zopnight/optimization/auto-remediation).