Autoscale clusters whose minimum floor of 4+ workers measurably idles
What does ZopNight detect here?
An autoscale minimum of 4 or more workers is a floor billed 24/7. The cluster can never shrink below it, even overnight. ZopNight recommends lowering the floor to 2 only after the derived cluster CPU shows at least 10% of a 30-day window under 20% load, and values only the idle share.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-2306 · RC-2406 · RC-2206 |
| Category | rightsizing |
| Severity | low |
| Metric | none — pure configuration read |
| Source | cluster_oversized.go |
Where it applies
The minimum is the commitment; the maximum is free
Autoscale bounds are routinely read backwards. A high maximum costs nothing until demand summons those workers; a high minimum is a standing purchase: the cluster holds that many workers through every idle hour, billing VMs and DBUs for capacity that exists because a configuration line says so. This rule targets interactive clusters whose floor is 4 workers or more, a shape that usually starts as “keep it snappy for the demo” and survives long after.
Config gets it flagged; measurement gets it a number
Autoscaling must be on (the fixed-size case belongs to a sibling rule, and the two can never both fire on one cluster), with the minimum at 4 or above. Then the same derived per-cluster CPU series used across ZopNight’s Databricks rules must show a sustained idle window: at least 10% of the 30-day series under 20% CPU. A genuinely busy min-4 cluster shows no such window and is left alone; telling it to drop to 2 workers would trade real capacity for imagined savings, and an earlier version of this rule that fired on configuration alone was corrected for exactly that reason.
Why the delta is halved
Pricing values the cluster at its autoscale midpoint: the average of min and max workers, plus the driver. Lowering the floor from its current value to the target of 2 moves that midpoint by half the floor reduction, so the reclaimed node count is (min − 2) / 2, priced at the per-node rate and then multiplied by the measured idle fraction. Only the idle share of the freed floor is claimed as savings; the busy share was doing work.
Survey your floors
databricks api get /api/2.1/clusters/list | jq ' .clusters[] | select((.autoscale.min_workers // 0) >= 4) | {cluster_name, autoscale, state}'Before lowering the number
Interactive latency is the one real cost: a floor exists so notebooks respond without waiting for scale-up. Check who uses the cluster at which hours, drop the minimum to 2, and watch the first week. The autoscaler will still reach the old worker counts whenever demand is real, just no longer at 3 a.m.