Long-lived Dataproc clusters that ran no YARN applications for 30 days
What does ZopNight detect here?
Dataproc clusters are flagged when active YARN applications, from `dataproc.googleapis.com/cluster/yarn/apps`, stay at or below 0.5 on average and at peak for 30 days. The cluster's Compute Engine VMs and the Dataproc fee of $0.010 per vCPU-hour keep billing regardless, so ZopNight counts the whole cluster cost as the saving.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-1210 |
| Category | idle |
| Severity | medium |
| Metric | dataproc.googleapis.com/cluster/yarn/apps |
| Threshold | YARN active applications at or below 0.5 |
| Evaluation window | 30d |
| Source | ZopNight |
| Permissions used | dataproc.clusters.list · monitoring.timeSeries.list |
Where it applies
What an idle Dataproc cluster keeps paying for
A Dataproc cluster on Compute Engine is billed on uptime, not on work. Google’s Dataproc pricing charges a management fee of $0.010 per vCPU per hour across master, worker and secondary worker nodes, billed by the second, and that fee is in addition to the Compute Engine price of every VM in the cluster. Persistent disks attached to the nodes bill too.
Clusters built for a one-off analysis and never torn down are the common case. They look harmless in the console and cost as much as a busy cluster of the same size.
Checking for YARN activity
gcloud dataproc clusters list --region=REGIONIn Metrics Explorer, chart dataproc.googleapis.com/cluster/yarn/apps for the cluster over 30
days. The metric counts active YARN applications, with a status label for running, pending and
other states. Zero throughout means no Spark, Hive or MapReduce job ran.
What ZopNight requires
- Active YARN applications stay at or below 0.5 on both average and peak for the whole window.
- The metric history covers at least 30 days.
- The cluster has a known monthly cost.
- The cluster has no scheduled deletion configured, such as a max-idle or max-age setting.
Clusters it deliberately ignores
A cluster with scheduled deletion already set is treated as ephemeral and managed, so it is not flagged. Clusters running Presto, Trino, HBase or Flink as optional components are skipped as well, because those services do work outside YARN and would look idle on this metric. When the YARN metric is missing for a cluster, there is no finding.
Saving the whole cluster cost
saving = current monthly cluster cost (VMs + Dataproc fee)cost after deletion = 0Deleting the cluster and preventing the next one
- Check scheduled jobs, workflow templates and orchestration tools such as Cloud Composer for references to the cluster.
- Copy anything left in cluster HDFS to Cloud Storage.
- Delete it:
gcloud dataproc clusters delete CLUSTER_NAME --region=REGION. - For future clusters, add scheduled deletion at creation, for example
gcloud dataproc clusters create CLUSTER_NAME --region=REGION --delete-max-idle=2h, so an idle cluster removes itself.