Fixed-size clusters measurably idling below 20% CPU with autoscaling off
What does ZopNight detect here?
All-purpose clusters running a fixed worker count pay VM and DBU charges for every worker around the clock. ZopNight fires only when the cluster has 2 or more workers, its derived CPU sat below 20% for at least 10% of a 30-day window, and real pricing yields a positive dollar saving.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-2308 · RC-2408 · RC-2208 |
| Category | rightsizing |
| Severity | low |
| Metric | none — pure configuration read |
| Source | autoscaling_disabled.go |
Where it applies
Two meters, both running around the clock
Databricks compute bills twice: the cloud provider charges for the VMs, and Databricks charges DBUs on top for every node-hour. A fixed-size cluster keeps both meters at full width regardless of load: 8 workers at 3 a.m. cost what 8 workers cost at peak. Autoscaling is the mechanism that narrows the meters during slack, shedding workers down to a minimum bound and re-adding them under demand, and this rule finds interactive clusters running without it while measurably idle.
The evidence bar before a dollar is claimed
Detection starts with configuration (no autoscale maximum set, meaning a fixed size), but configuration alone never produces a finding. The cluster must be running, be a user-created all-purpose cluster (system-managed job, pipeline, SQL, and serving clusters are sized by the platform and skipped), and have at least 2 workers, since a single-worker cluster has nothing to shed. Then comes measurement: the aggregator derives a per-cluster CPU series by averaging the worker VMs’ utilization, and the share of that series sitting below 20% CPU over 30 days must reach at least 10%. Below that, the load is effectively steady and a fixed size may be the right call, so the rule abstains.
The savings arithmetic, node by node
The claimed figure is deliberately smaller than “cluster cost times idle time”. With monthly cost priced across the workers plus the driver, the per-node rate is cost divided by workers plus 1; the recoverable capacity is only the workers above the minimum floor of 1 that autoscaling would keep, and only for the measured idle fraction. The driver never scales away, the floor never scales away, and peak capacity is untouched. The autoscale ceiling still serves demand.
Find fixed-size clusters yourself
databricks api get /api/2.1/clusters/list | jq ' .clusters[] | select(.autoscale == null and (.num_workers // 0) >= 2) | {cluster_name, num_workers, state}'Silence is an answer too
No finding appears for stopped or errored clusters, unpriced clusters, or clusters whose CPU series is missing, notably pool-backed AWS clusters whose worker instances carry no cluster tag and therefore cannot be joined into a per-cluster series. In every such case the rule returns nothing rather than a fabricated fraction or a $0 placeholder.