# Dataflow Job

> Dataflow bills each job for the worker VMs it runs: vCPU, memory, and storage metered per second while workers exist. Batch jobs end on completion, but a streaming job runs until cancelled, making it an always-on VM fleet. ZopNight inventories jobs via Cloud Asset Inventory and highlights streaming jobs with sustained low utilization.

Source: https://zop.dev/integrations/gcp/dataflow
Updated: 2026-08-19

---

Dataflow runs managed Apache Beam pipelines for stream and batch processing, billed for the worker VMs, memory, and storage each job consumes. Long-running streaming jobs are effectively always-on fleets of VMs.

## Workers are what the job meter counts

A Dataflow job has no charge of its own. The bill is the sum of what its workers consume: vCPU-seconds, memory, and the Persistent Disk attached to each worker, metered for as long as the workers exist. The job's lifecycle therefore is the cost model. A batch pipeline spins workers up, processes, and terminates on completion, so its cost is bounded by its runtime. A streaming pipeline holds its workers indefinitely, because streaming jobs run until someone cancels them, which turns every forgotten streaming job into a small VM fleet billing around the clock.

## Streaming utilization is the signal that matters

ZopDev inventories Dataflow jobs via Cloud Asset Inventory, attributes worker spend from billing actuals, and highlights streaming jobs with sustained low utilization: the pipeline still faithfully consuming a trickle of messages on workers sized for a flood. Jobs are not schedulable in ZopNight: pausing a Beam pipeline is not a platform operation, and a batch job's end is its own stop. The actionable outcomes are cancelling abandoned streaming jobs and resizing over-provisioned ones.

## Pipelines that keep billing after everyone moved on

Three shapes recur. The proof-of-concept streaming job wired to a test topic, left running months after the experiment concluded. The over-provisioned production stream, where autoscaling floors or fixed worker counts were set during a traffic spike and never revisited. And the retry loop: a batch pipeline relaunched on a schedule that fails partway every run, paying for the same processing repeatedly without ever producing output anyone reads.

## Separating live pipelines from zombies in the console

Google Cloud console → Dataflow → Jobs lists jobs with their type, status, and start time. Filter by status Running and sort by start time: a streaming job started long ago whose throughput graphs sit near zero is the zombie to investigate, and its job graph names the topics and tables it touches for confirmation.