Vertex AI Pipeline Job

A pipeline job orchestrates a DAG of ML steps on Vertex AI Pipelines, billed a small per-run fee plus the compute each step consumes. Failed pipelines that re-run expensive training steps are the main waste pattern.

A small orchestration fee plus every step’s compute

Vertex AI Pipelines charges a flat fee per pipeline run, and then each step bills for whatever compute it uses: training jobs, batch predictions, data processing. Step compute dominates; the orchestration fee is minor. That structure means the pipeline object itself is cheap while its contents can be arbitrarily expensive, and the true total only appears when per-step spend is summed across the whole DAG.

Pipeline-job visibility in ZopNight

ZopNight discovers pipeline jobs through the live aiplatform API across 32 Vertex regions rather than relying on Cloud Asset Inventory, whose rows for transient jobs go stale after completion. Per-step compute spend is attributed in ML cost views, so a pipeline’s real cost (the sum of its steps) is visible in one place instead of scattered across anonymous training and prediction charges. Pipeline jobs are not schedulable; each run terminates by itself.

Re-run waste in ML pipelines

The dominant leak is the failed run repeating its expensive early steps: a pipeline that trains for hours and fails at a final deployment step will, without step caching, re-train from scratch on retry. Add cron-scheduled pipelines still executing after their consumers disappeared, and individual steps specified with accelerators sized for a different workload entirely. Reviewing where the DAG fails is worth far more than reviewing the orchestration fee.

Finding pipeline runs in the console

Google Cloud console → Vertex AI → Pipelines lists runs per region with state, duration, and a step-by-step breakdown. That step view is exactly where a failed run’s repeated cost shows up, and where step caching proves whether a retry actually skipped the expensive work.