Databricks clusters, SQL warehouses and jobs missing the team, environment or cost-center tag

Untagged DBUs land in nobody’s budget

Databricks spend is only as attributable as its tags. According to the usage attribution guide, custom tags on clusters, SQL warehouses and pools flow into the account’s usage records and are what lets you attribute compute to teams, projects or cost centers and build budgets around them. On AWS, cluster tags also propagate to the EC2 instances behind a cluster (unless it runs from a pool), so the cloud bill can be split the same way.

Usage from a resource with no owner tag reaches the custom_tags column of system.billing.usage without those keys, so a finance report grouped by team can only file it as unallocated.

Spotting gaps in each resource type

Clusters carry custom_tags directly, warehouses nest them under tags.custom_tags as key/value pairs, and jobs keep theirs in the job settings:

databricks clusters list -o json \
  | jq -r '.[] | select(.custom_tags["cost-center"] == null) | .cluster_name'

databricks warehouses list -o json \
  | jq -r '.[] | select(([.tags.custom_tags[]?.key] | index("team")) == null) | .name'

To size the problem in DBUs, total last month’s usage that arrived with no team tag:

SELECT billing_origin_product, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - 30
  AND custom_tags['team'] IS NULL
GROUP BY ALL
ORDER BY dbus DESC;

Three keys, matched exactly

The rule looks for the keys team, environment and cost-center, spelled exactly that way. A resource missing any one of them is reported, and the finding names which keys are absent. A resource with no tags at all is the worst case and is always reported.

Resources the tag check leaves out

• Clusters created by jobs, pipelines, SQL warehouses or model serving; only interactive clusters are checked, since the parent job or warehouse is where their tags belong.
• Clusters and warehouses that are stopped or in error, whose missing tags attach to no live spend.
• Any resource whose tag data could not be parsed.

Jobs have no running state of their own, so they are checked whatever state their compute is in.

Why the saving is zero

Tagging moves no money; it makes every other number traceable. ZopNight therefore shows no saving here and ranks the finding low severity.

Tagging the resource and making it stick

1. Add the missing keys on the cluster, warehouse or job. Keys and values may use letters, numbers and + - = . , _ : @, but no spaces or /.
2. Restart clusters afterwards: tag changes apply only after a cluster restart or pool expansion.
3. Enforce the keys in a cluster policy with fixed custom_tags.<key> entries so new clusters cannot skip them.
4. For clusters launched from a pool on AWS, tag the pool as well; its EC2 instances inherit pool and workspace tags, not cluster tags.