Dataproc clusters running zero YARN applications for a month
What does ZopNight detect here?
Dataproc clusters keep billing master and worker VM-hours between jobs. ZopNight watches the yarn/apps metric with state=RUNNING and flags clusters that stay at or below a 0.5 application floor for 30 measured days, excluding Presto, Trino, HBase and Flink clusters whose work never appears as YARN applications.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-1210 |
| Category | idle |
| Severity | medium |
| Metric | none — pure configuration read |
| Source | dataproc_idle.go |
Where it applies
The cluster bills between jobs
A Dataproc cluster is a set of Compute Engine VMs with persistent disks attached, and all of it bills continuously (the master, every worker, and the disks) whether a job is running or the cluster is waiting for the next nightly batch that stopped being scheduled months ago. The recommendation treats the full cluster cost as recoverable, because the modern pattern is to delete and recreate on demand with Workflow Templates rather than keep a warm cluster.
A floor of 0.5, not an exact zero
The idle signal is the dataproc.googleapis.com/cluster/yarn/apps gauge pinned to
state=RUNNING, harvested via monitoring.timeSeries.list under
roles/monitoring.viewer. The gate is average and peak at or below 0.5 rather than
exactly zero: the gauge samples coarsely, so a short job finishing between samples can
leave a fractional residue on a genuinely dead cluster. Anything above the floor, on the
full series or the 30-day window, clears the cluster. The rule also demands at least 30
days of measured coverage; a freshly discovered cluster with a thin series is never
flagged.
Engines that hide their work from YARN
Clusters running Presto, Trino, HBase, or Flink as optional components are excluded outright: those engines serve queries through their own long-lived processes and can be fully busy while reporting zero YARN applications. Zeppelin and Jupyter notebooks are deliberately not excluded: their Spark work runs through YARN, so an idle notebook cluster reports honestly. Clusters carrying an auto-delete TTL are also skipped; they reclaim their own cost.
List active jobs before deleting
gcloud dataproc jobs list --region us-central1 \ --cluster my-cluster --state-filter activeAn empty list plus a quiet YARN graph is the confirmation the rule already required.
Deletion goes through dataproc.clusters.delete (roles/dataproc.editor); enable
idleDeleteTtl on replacement clusters so this cannot recur.
Coverage and abstention
No recommendation appears when cluster pricing is unknown or zero, when the cluster is ephemeral, when a non-YARN serving component is present, when the YARN metric was never harvested, when any activity clears the floor, or when measured coverage falls short of 30 days. There is no status- or tag-based fallback. A cluster without the measured series simply does not fire.