# GCP Dataproc Cluster Idle

> Dataproc clusters keep billing master and worker VM-hours between jobs. ZopNight watches the yarn/apps metric with state=RUNNING and flags clusters that stay at or below a 0.5 application floor for 30 measured days, excluding Presto, Trino, HBase and Flink clusters whose work never appears as YARN applications.

Source: https://zop.dev/integrations/gcp/recommendations/gcp-dataproc-cluster-idle
Updated: 2026-08-19

---

## The cluster bills between jobs

A Dataproc cluster is a set of Compute Engine VMs with persistent disks attached, and all
of it bills continuously (the master, every worker, and the disks) whether a job is
running or the cluster is waiting for the next nightly batch that stopped being scheduled
months ago. The recommendation treats the full cluster cost as recoverable, because the
modern pattern is to delete and recreate on demand with Workflow Templates rather than
keep a warm cluster.

## A floor of 0.5, not an exact zero

The idle signal is the `dataproc.googleapis.com/cluster/yarn/apps` gauge pinned to
`state=RUNNING`, harvested via `monitoring.timeSeries.list` under
`roles/monitoring.viewer`. The gate is average and peak at or below 0.5 rather than
exactly zero: the gauge samples coarsely, so a short job finishing between samples can
leave a fractional residue on a genuinely dead cluster. Anything above the floor, on the
full series or the 30-day window, clears the cluster. The rule also demands at least 30
days of measured coverage; a freshly discovered cluster with a thin series is never
flagged.

## Engines that hide their work from YARN

Clusters running Presto, Trino, HBase, or Flink as optional components are excluded
outright: those engines serve queries through their own long-lived processes and can be
fully busy while reporting zero YARN applications. Zeppelin and Jupyter notebooks are
deliberately not excluded: their Spark work runs through YARN, so an idle notebook cluster
reports honestly. Clusters carrying an auto-delete TTL are also skipped; they reclaim
their own cost.

## List active jobs before deleting

```bash
gcloud dataproc jobs list --region us-central1 \
  --cluster my-cluster --state-filter active
```

An empty list plus a quiet YARN graph is the confirmation the rule already required.
Deletion goes through `dataproc.clusters.delete` (`roles/dataproc.editor`); enable
`idleDeleteTtl` on replacement clusters so this cannot recur.

## Coverage and abstention

No recommendation appears when cluster pricing is unknown or zero, when the cluster is
ephemeral, when a non-YARN serving component is present, when the YARN metric was never
harvested, when any activity clears the floor, or when measured coverage falls short of 30
days. There is no status- or tag-based fallback. A cluster without the measured series
simply does not fire.