Outcome
By the end of this lesson, you will be able to identify Databricks Workspaces, Clusters, Instance Pools, and SQL Warehouses in the parent-child hierarchy, navigate the discovered resources, and explain which entities drive cost.
| Prerequisites | M2.1 (rule library) |
| Time | 9 minutes |
1. Concept
Everything in Databricks sits under a workspace.
ZopNight finds six kinds of thing: the workspace itself, plus clusters, instance pools, SQL warehouses, jobs and model-serving endpoints.
Clusters, instance pools and SQL warehouses can carry a schedule on every cloud. The workspace itself can carry one on Azure only. Jobs and model-serving endpoints you cannot schedule; they are found so their cost shows up, and nothing more.
DATABRICKS HIERARCHY:
Workspace: prod-databricks-eu├── Clusters│ ├── ml-training-cluster│ ├── data-engineering-cluster│ └── dev-experiments-cluster├── Instance Pools│ └── shared-ml-pool├── SQL Warehouses│ ├── reporting-warehouse│ └── ad-hoc-warehouse├── Jobs (discovered, not schedulable)│ └── nightly-etl└── Model serving endpoints (discovered, not schedulable) └── recommender-v3Six discovered types. Clusters, pools and SQL warehouses can carry a schedule on every cloud, and the workspace can on Azure; jobs and model-serving endpoints are read for cost attribution only.
How the connection works, and why it differs by cloud
AWS and GCP: a standalone Databricks connection own credential row, OAuth machine-to-machine service principal, cloud derived from the workspace host. No underlying cloud account is used for the start/stop call.
Azure: rides the existing Azure subscription workspaces come from Resource Graph, children via ARM plus an AAD token. Enumerating children needs workspace-admin permission inside Databricks, which the subscription role cannot grant; that grant is probed and surfaced separately.This matters operationally: on AWS and GCP a Databricks credential problem is independent of the cloud account, and on Azure a working subscription connection can still show zero children until the workspace-level grant exists.
Each component’s purpose
WORKSPACE The Databricks tenant Per-region Has its own URL and authentication Container for all other entities
CLUSTER A Spark cluster: compute capacity for jobs and interactive notebooks Created on demand or kept running All-purpose or job-specific Major cost contributor
INSTANCE POOL Pre-warmed pool of nodes for fast cluster startup Pools have idle pods waiting Reduces cold-start latency
SQL WAREHOUSE Auto-scaling compute for SQL queries Modern Databricks SQL experience Sizes from 2X-Small to 4X-Large Auto-pause when idle (configurable)The hierarchy is logical; cost-relevant components are at the right level.
Scheduling targets
Clusters, pools and SQL warehouses are schedulable on every cloud. The workspace is schedulable on Azure only; on AWS and GCP, schedule its children instead:
WORKSPACE (Azure only) Stop stops every cluster, pool and SQL warehouse in the workspace Coarse-grained; rarely the right level
INDIVIDUAL CLUSTERS Stop/start specific clusters Most common scheduling pattern Fine-grained control
INSTANCE POOLS Resize (pool of 0 = no idle nodes) Back to N when ready Reduces idle pool cost
SQL WAREHOUSES Stop and start only (no resize) Auto-pause handles much already Additional scheduling for hard-stopThe flexibility allows tuning for different workload patterns.
Cost contribution by component
COMPONENT COST SHAPE─────────────────────────────────────────────────────────────Workspace Modest fixed cost Few dollars/mo per workspace Mostly metadata + control plane
Clusters HIGH variable cost (compute hours) Often the main cost driver $10-1000+/hr depending on size
Instance Pools Idle pool size × hour $50-500/mo per pool typical Hidden cost; easy to forget
SQL Warehouses Active hours × warehouse size Auto-pause helps Still: $5-500/mo per warehouseClusters and SQL Warehouses are the major cost contributors. Pools matter at scale.
Discovery scope
DISCOVERED (in scope): ✓ Workspaces (1+ per region) ✓ All Clusters in each workspace ✓ All Instance Pools in each workspace ✓ All SQL Warehouses in each workspace ✓ Per-resource state (running / stopped / idle) ✓ Configuration (size, autotermination, etc.) ✓ Tags ✓ Jobs and model-serving endpoints (for cost attribution; not schedulable)
NOT DISCOVERED (out of scope): ✗ Notebooks (these aren't billable resources) ✗ Tables, ML models (not billable in this scope) ✗ Users / permissions (not cost-relevant) ✗ MLflow experiments ✗ Delta Lake tables (storage is separate from compute)The focus is on what costs money. Notebooks are just files; jobs are discovered so their cost can be attributed, but the compute cost lives in the cluster running them.
Workspace permission requirements
For ZopNight to discover Databricks resources:
PERMISSIONS NEEDED: Read on Workspace metadata List Clusters List Instance Pools List SQL Warehouses Per-resource: read state + config
ACTIONS for scheduling: Start/Stop Clusters (cluster:can_attach + can_manage) Resize Pools (pool:can_manage) Start/Stop SQL Warehouses (warehouses:can_use + can_manage)
TYPICAL SETUP: AWS / GCP: one OAuth M2M service principal per Databricks Account ID (the account-console host), covering the account's workspaces Azure: the Azure subscription's service principal, plus workspace-admin access inside each workspace No personal access token (PAT) pathOn AWS and GCP you onboard the Databricks account once, not each workspace. On Azure the subscription connection carries it, and each workspace still needs the workspace-level grant.
Discovery frequency
DATABRICKS DISCOVERY: every 6 hours Same cadence as other cloud resources
WHAT CHANGES BETWEEN DISCOVERIES: Cluster state (running/idle/stopped) Cluster size (auto-scaled) SQL Warehouse state
FRESHNESS: Up to 6 hours stale Acceptable for cost optimization Real-time monitoring use different toolsThe 6-hour cadence matches other resource discovery.
How Databricks fits into the broader ZopNight model
DATABRICKS resources behave like other resources: Discovered automatically Show in Inventory page Get cost calculated Get recommendations Can be scheduled Can be in groups Audit log captures actions
DIFFERENT: Compute units (DBU) vs traditional CPU/memory Workspace abstraction (no direct equivalent in AWS/GCP/Azure) Tight integration with Databricks API (not generic cloud)Once discovered, Databricks resources behave like any other ZopNight resource.
2. Demo
A typical Databricks discovery:
DATABRICKS DISCOVERY:
T+0 Discovery cron runs for prod-aws-eu-1 (which has the Databricks Workspace)T+30s Found:
Workspace: prod-databricks-eu (1) Children: Clusters: 3 - ml-training (currently active, 12 nodes) - data-engineering (idle, 0 nodes pending) - dev-experiments (running, 2 nodes) Instance Pools: 1 - shared-ml-pool (idle 4 nodes) SQL Warehouses: 2 - reporting-warehouse (active, X-Small) - ad-hoc-warehouse (auto-paused)
T+30s All children visible in ZopNight Inventory pageT+30s Cost data calculated per resourceT+30s Recommendations begin computing
INSPECTION (5 min later): Inventory page → filter type=databricks Workspace tree visible Each entity has: Current state Cost (latest 30 days) Recommendations (if any) Schedule attachment (if any)The whole Databricks footprint becomes schedulable in 30 seconds of discovery.
3. Hands-on (5 min)
Explore your Databricks estate (if you have one):
□ STEP 1: Open Resources; filter type=databricks Workspaces: _____
□ STEP 2: Pick one workspace; drill in Clusters: _____ Instance Pools: _____ SQL Warehouses: _____
□ STEP 3: Note state of each cluster Currently running: _____ Currently stopped: _____
□ STEP 4: Identify cost contributors Highest-cost cluster: __________ Highest-cost SQL warehouse: __________ Total Databricks spend (last 30 days): $_____
□ STEP 5: Check schedules Clusters scheduled: _____ / total Opportunity: _____ unscheduled clustersA 10-minute exercise reveals the Databricks footprint.
4. Knowledge check
Q1
Databricks Workspace has children:
A. Just Clusters
B. Notebooks only, since they are the unit that actually consumes the compute and therefore the only child worth discovering separately from the parent workspace record
C. Five child types are discovered (clusters, instance pools, SQL warehouses, jobs, model-serving endpoints); clusters, pools and warehouses are schedulable, jobs and serving endpoints are not
D. Jobs
Show answer
Correct: C. Workspace is the container; it is itself schedulable on Azure only. Five child types are discovered, and three of them are schedulable on every cloud.
Q2
Notebooks and Jobs:
A. Are billable; ZopNight schedules them
B. Same as Workspace
C. Notebooks are discovered separately, but the Jobs are never discovered by the discovery pass at any point in the cycle
D. Notebooks run on clusters and are not discovered separately; jobs are discovered as their own type but cannot be scheduled
Show answer
Correct: D. Notebooks are not discovered; jobs are discovered but not schedulable. Either way the compute cost sits on the cluster or warehouse underneath.
Q3
SQL Warehouses contribute cost via:
A. Active hours and the warehouse size
B. Idle time only
C. Storage volume of the queried tables
D. Notebook count
Show answer
Correct: A. Cost is active hours × size. Auto-pause minimizes idle cost by default, but active queries on big warehouses get expensive fast, so sizing and scheduling both matter.
5. Apply
On Azure, Databricks is discovered through the existing Azure subscription. On AWS and GCP it is a standalone connection with its own OAuth service principal. Children appear under the Workspace either way.
For your team: navigate the hierarchy; identify cost contributors; plan schedules for L2-L4.