Instance pools keeping warm VMs that no cluster is drawing from

Warm capacity is a VM bill with no DBU line

Instance pools exist to make cluster starts fast: the pool holds pre-provisioned VMs, and clusters grab them instead of waiting on the cloud. Databricks charges no DBUs for an instance sitting warm in a pool, but the cloud provider charges full VM price for it, every hour, attached to a cluster or not. A min-idle floor is therefore a standing infrastructure cost justified only by the start-latency it saves, and this rule finds pools where nothing is being started at all.

Fired on live utilization, not configuration

A minimum-idle setting above 0 alone is not the finding; that was an earlier, noisier version of this rule, and it overstated waste by flagging pools whose warm floor was busy serving clusters. The current gate reads the pool statistics the discoverer captures at each run: usedCount must be exactly 0 (no cluster is drawing from the pool) while idleCount is positive (warm instances exist right now). When the stats keys are absent (older inventory predating them), the rule declines to fire rather than fall back to the config-only signal. And because the entire warm floor is unconsumed VM spend, the pool’s full priced VM cost is reported as the saving; an unpriceable node type means abstention, never a $0 recommendation.

Inspect pool stats yourself

databricks api get /api/2.0/instance-pools/list | jq '
  .instance_pools[]
  | select((.min_idle_instances // 0) > 0)
  | {instance_pool_name, min_idle_instances, stats}'

The stats object carries idle_count and used_count: the same two numbers the rule gates on.

A snapshot, honestly labeled

Both counts are point-in-time values from the last discovery pass, not a time series. A pool photographed between cluster attachments (jobs ended at 09:00, next wave at 09:30) can momentarily show zero usage and trip the rule. The finding claims the pool was unused at snapshot time, nothing more; check the pool’s recent attach pattern before acting on it.

Draining the floor

Lower the pool’s minimum idle toward 0 and set idle-instance auto-termination so surplus warm VMs release themselves. Clusters keep working either way. They fall back to provisioning from the cloud, paying the cold-start seconds the warm floor was buying. If nobody notices the difference, the floor was pure cost.