Dataflow Job
Does ZopNight manage Dataflow Job?
Dataflow bills each job for the worker VMs it runs: vCPU, memory, and storage metered per second while workers exist. Batch jobs end on completion, but a streaming job runs until cancelled, making it an always-on VM fleet. ZopNight inventories jobs via Cloud Asset Inventory and highlights streaming jobs with sustained low utilization.
Rules that fire on Dataflow Job
No active rule family targets Dataflow Job today. Rules that used to are retired, and retired rules publish no pages and fire no findings. Scheduling and permissions coverage are unaffected.
Dataflow runs managed Apache Beam pipelines for stream and batch processing, billed for the worker VMs, memory, and storage each job consumes. Long-running streaming jobs are effectively always-on fleets of VMs.
Workers are what the job meter counts
A Dataflow job has no charge of its own. The bill is the sum of what its workers consume: vCPU-seconds, memory, and the Persistent Disk attached to each worker, metered for as long as the workers exist. The job’s lifecycle therefore is the cost model. A batch pipeline spins workers up, processes, and terminates on completion, so its cost is bounded by its runtime. A streaming pipeline holds its workers indefinitely, because streaming jobs run until someone cancels them, which turns every forgotten streaming job into a small VM fleet billing around the clock.
Streaming utilization is the signal that matters
ZopDev inventories Dataflow jobs via Cloud Asset Inventory, attributes worker spend from billing actuals, and highlights streaming jobs with sustained low utilization: the pipeline still faithfully consuming a trickle of messages on workers sized for a flood. Jobs are not schedulable in ZopNight: pausing a Beam pipeline is not a platform operation, and a batch job’s end is its own stop. The actionable outcomes are cancelling abandoned streaming jobs and resizing over-provisioned ones.
Pipelines that keep billing after everyone moved on
Three shapes recur. The proof-of-concept streaming job wired to a test topic, left running months after the experiment concluded. The over-provisioned production stream, where autoscaling floors or fixed worker counts were set during a traffic spike and never revisited. And the retry loop: a batch pipeline relaunched on a schedule that fails partway every run, paying for the same processing repeatedly without ever producing output anyone reads.
Separating live pipelines from zombies in the console
Google Cloud console → Dataflow → Jobs lists jobs with their type, status, and start time. Filter by status Running and sort by start time: a streaming job started long ago whose throughput graphs sit near zero is the zombie to investigate, and its job graph names the topics and tables it touches for confirmation.