# Dataproc Cluster

> Dataproc clusters bill for their underlying Compute Engine VMs plus a per-vCPU Dataproc premium for every hour the cluster exists. ZopNight discovers clusters across 30+ regions via Cloud Asset Inventory, counts active jobs on each one, and flags clusters sitting idle between Spark or Hadoop runs for deletion or ephemeral rebuilds.

Source: https://zop.dev/integrations/gcp/dataproc
Updated: 2026-08-19

---

Dataproc provides managed Hadoop and Spark clusters billed for the underlying VMs plus a per-vCPU Dataproc premium. Clusters kept alive between jobs are a classic source of idle analytics spend.

## A premium stacked on ordinary VM meters

Dataproc's cost is layered: the master and worker nodes are ordinary Compute Engine instances with their usual vCPU, memory, and disk meters, and Dataproc adds its own premium per vCPU per hour on top for the management layer. Both layers run whether or not a single job is executing. An idle cluster pays exactly what a busy one does. The design intent is ephemeral clusters created per job and deleted after, but plenty of teams operate them as standing infrastructure, which is where the waste concentrates.

## Counting live jobs region by region

Dataproc's cluster and job APIs are region-scoped with no aggregated endpoint, so ZopDev sweeps the GA regions plus the special global region, 30-plus in all, to enumerate every cluster. Each cluster is then enriched with its count of active jobs (pending, running, or cancelling), giving the idle rule an authoritative no-jobs signal rather than a guess. Clusters running persistent non-YARN engines such as Presto, Trino, HBase, or Flink are deliberately excluded from idle detection, because those serve queries without launching YARN applications and would otherwise look idle while doing real work. Job-activity enrichment is what distinguishes busy clusters from abandoned ones.

## Standing clusters and other Spark-era habits

The costliest pattern is the always-on cluster serving a nightly batch window, paying for 24 hours to work for 2. Close behind: clusters sized for the largest job of the quarter and never scaled back, and orphaned dev clusters created for a one-off exploration whose owner has moved on.

## Reading cluster activity in the console

Google Cloud console → Dataproc → Clusters lists clusters per region with their status and size. The Jobs tab, filtered to a cluster, shows when work last actually ran. A Running cluster whose most recent job finished days ago is the deletion candidate the idle rule exists to catch.