All-purpose clusters created outside any cluster policy

Freehand clusters drift into expensive shapes

Nothing about an unpoliced cluster is necessarily wrong today. The problem is what it is allowed to become: any node type including GPU families, any worker count, any auto-termination setting including none, any tags including none. Databricks cluster policies are the workspace’s mechanism for bounding those choices at creation time, and every cluster running outside one represents configuration that only stays sane by individual discipline. The recurring pattern in real workspaces is that the oversized, untagged, never-terminating cluster is almost always also the unpoliced one.

A positive probe, not an absence test

The detection has a subtlety worth spelling out. ZopNight’s enricher writes a policyId key onto every all-purpose cluster it collects, using an empty string when no policy is attached. The rule therefore fires only when the key is present and empty: that combination proves the field was collected and the cluster genuinely has no policy. When the key is missing entirely, the honest reading is “not collected”, not “no policy”, and the rule abstains. An earlier implementation fired on both cases and manufactured findings from partial metadata; the current one cannot.

Scope gates

Stopped and errored clusters are skipped (no current compute, no current exposure), and system-managed clusters created by jobs, pipelines, SQL warehouses, and serving are excluded because the platform provisions them without user-facing policies by design. The finding carries no dollar figure: it is governance, and pretending a policy gap has a computable monthly saving would be invented precision.

List unpoliced clusters

databricks api get /api/2.1/clusters/list | jq '
  .clusters[]
  | select((.policy_id // "") == "")
  | {cluster_name, creator_user_name, state}'

What a policy worth attaching pins down

Four bounds cover most of the drift: an allowlist of node types, a ceiling on workers and autoscale ranges, a mandatory auto-termination maximum, and required cost- allocation tags. Attach the policy to existing clusters, then close the loop by restricting the unrestricted-cluster-creation entitlement so new compute must be born inside a policy, at which point this finding stops recurring on its own.