Scheduled jobs running on all-purpose clusters instead of job clusters

Two DBU rates for the same work

All-purpose compute is priced for interactive use: a cluster you attach notebooks to, that stays warm between commands. Job compute is priced for scheduled execution on an ephemeral cluster that exists only for the run.

The same notebook, on the same instance types, costs more on the first than the second. A job scheduled against an interactive cluster pays that premium on every execution, indefinitely.

Concrete-or-abstain, strictly

This rule follows the platform’s concrete-or-abstain directive, and the source is blunt about it: emitting a $0 finding under a cost-driven category is treated as a defect.

So when the rate data for either compute type is missing, the rule returns nothing. It does not report a governance-flavoured $0 as a placeholder, and it does not estimate the delta from a default ratio. Either both real rates resolve and the saving is quantified, or there is no finding at all.

That is worth knowing when reading an empty result: silence here can mean missing rate data rather than a clean workspace.

Why severity is high

Unlike most rightsizing findings, this is pure pricing arbitrage. Switching a job to job compute changes no code, no output, and nothing about the cluster spec beyond its type. The saving is immediate and the risk is close to zero, rare enough to justify the rating.

Finding jobs with existing_cluster_id set

databricks jobs list --output json \
  | jq -r '.jobs[]
      | select(.settings.existing_cluster_id != null)
      | [.job_id, .settings.name, .settings.existing_cluster_id] | @tsv'

Any job with existing_cluster_id set is bound to an all-purpose cluster. Jobs using new_cluster already create a job cluster per run and sit on the cheaper rate.

What the migration costs you

Replace existing_cluster_id with a new_cluster block carrying the same node type and worker count. The first run is slower because the cluster is created fresh. That startup time is the entire trade, and for anything running longer than a few minutes it is dwarfed by the rate difference.