Running interactive clusters whose uptime cannot be billed to a team

Uptime is the attribution unit for clusters

An all-purpose cluster is long-lived compute: someone creates it, teams attach notebooks to it, and it bills by the hour for weeks or months. That longevity is what makes attribution tractable: the cluster’s uptime is the cost, so a correct owner label on the cluster settles every hour it ever runs. The same three keys mean different things on ZopNight’s job and SQL-warehouse tag rules, where the units are per-run bursts and shared endpoints; here the question is simply “whose long-running machine is this?”

What is checked, and what is deliberately not

The rule requires all of team, environment, and cost-center in the cluster’s custom tags and reports whichever of the 3 are missing. Two gates narrow the scope. Stopped and errored clusters are skipped because they bill nothing, so their tag gap is not actionable spend, and it resurfaces the moment the cluster starts again. And only interactive all-purpose clusters are examined: clusters spawned by jobs, pipelines, SQL warehouses, and model serving inherit attribution from the object that owns them, so flagging them would be noise. A cluster whose tag blob fails to parse is left alone rather than guessed at.

Audit custom tags across the workspace

databricks api get /api/2.1/clusters/list | jq '
  .clusters[]
  | {cluster_name,
     missing: (["team", "environment", "cost-center"]
               - ((.custom_tags // {}) | keys))}
  | select(.missing | length > 0)'

Why this reaches the cloud invoice

Databricks propagates a cluster’s custom tags onto the underlying cloud VMs and disks it provisions. Tag the cluster and the cloud provider’s own cost tooling can slice the VM-side spend by team as well. The DBU side and the infrastructure side of the bill finally agree on an owner. Default tags like ClusterName and Creator are not a substitute: a creator who left the team is not a cost center.

Making it stick

Add the missing keys in the cluster’s configuration page or via the API, then encode them in a cluster policy with fixed or required tag values so the next cluster cannot be created without them. Enforcement at creation time is what turns this from a recurring cleanup into a solved problem.