Skip to main content
← Back to blog

A FinOps Cost Source Choice That Never Shows a Fake Zero

Riya Mittal
Riya Mittal Engineer · Zop.Dev
5 min read
A FinOps Cost Source Choice That Never Shows a Fake Zero

A FinOps Cost Source Choice That Never Shows a Fake Zero

A cost number a customer never agreed to pay for is worse than a missing one. ZopNight’s cost figures used to come from whatever billing access an account happened to have connected, which sometimes meant paid AWS Cost Explorer calls a customer never explicitly chose to incur. Settings → Preferences → Cost Preferences now puts that choice in the open: Estimates only, Your billing export, or Export + billing API, the last one labeled plainly as Billed to you. Any single cloud account can override the org-wide choice on its own.

Two smaller changes carry more weight than they seem to at first glance. A cost with no underlying data now renders as a dash instead of a zero-dollar line, and a cost that’s modeled rather than measured is labeled Rack rate rather than presented as if someone actually billed it. Both exist so a customer reading a number can tell what kind of number it is.

A Policy Resolved the Same Way Everywhere, or Not at All

This shipped as part of a much larger programme spanning 583 files and a diff of over 61,000 added lines, continuing work from an earlier release. At its center is a capture policy: an organization, or a single cloud account inside it, is set to modeled_only, measured_free, or measured_any. Resolving which one applies follows a fixed order. An explicit per-organization override row wins first. Failing that, a stored key-value setting applies. Failing that, a system default takes over.

Architecture diagram

The part worth noticing is that two entirely separate services, the aggregator and internal-api, both implement this same precedence and are proven to agree through a dedicated parity test rather than through code review alone. A policy that can’t be read for any reason defers every kind of capture rather than falling back to a guess. A result list too large to fully return, north of a thousand rows, still runs its free capture normally and defers only the paid portion.

AWS Cost Explorer charges for every call it answers, which makes it a genuinely different kind of dependency than a free billing export. Every call to it now passes through a consent gate that refuses outright if a fresh stored denial exists for that account. A per-run budget bounds how many paid requests can happen at all, floored at 50 and capped at 1000, and the daily capture job gives up on the entire day once it hits 5 failures of one specific kind or 30 failures in total.

None of those numbers are arbitrary safety theater. A budget with no floor could starve a legitimately large account of any paid data at all. A budget with no ceiling could let one misbehaving account run up real charges on ZopNight’s own AWS invoice before anyone noticed.

A Failed Write Fails the Whole Run

Billing records now live only in ClickHouse, not in the MySQL tables that used to hold parts of this data. That’s a real tradeoff, not a free upgrade. A failed write to ClickHouse fails the entire capture run, skips that period’s cost overlay, and raises an alert, rather than quietly succeeding halfway and leaving a customer looking at numbers built from half a day’s data with no indication anything was wrong.

The tradeoff shows up explicitly in how rollback works. Reverting the code is simple. Reverting the data isn’t, because MySQL’s billing_records table received zero rows for the entire window this change was live. A rollback restores the old code path, but it can’t retroactively populate the rows that ClickHouse alone was holding.

Thirteen Bugs a Cross-Checked System Surfaces

Building a system where two independent services have to agree on the same answer tends to surface bugs that a single code path would never expose, and this release found thirteen of them along the way.

BugWhat was actually wrong
Double-counted daily totalsA daily cost aggregation summed both a calculated row and an actual row for the same period, overstating true cost by 2.3 times
Free tier still payingThe measured_free capture tier still made paid Cost Explorer calls on accounts that already had a free billing export available
Silent zero on a paid resourceAzure reserved VM costs read as approximately zero on the free capture lane
Wrong failure directionAn unreadable capture policy used to fall back to paid Cost Explorer calls instead of deferring, exactly backwards from what a fail-safe should do

The double-counting bug is the clearest example of why cross-checking catches what review doesn’t. A calculated estimate and an actual measured figure are supposed to be alternatives for the same period, never both added together. Nothing about either number looked wrong in isolation. The bug only became visible once the two were compared against a third, independent source of truth.

Shipping It Needed a Specific Order of Operations

Deploying this safely wasn’t a single flag flip. It required running a ClickHouse backfill followed by a shadow-compare pass before trusting the new path, setting one previously unset environment variable on the aggregator without which the entire consent write gate silently does nothing at all, and enabling a specific recommender flag before any new recommender pods start, since six of its rules change their own internal contract revision as part of this release.

A same-week follow-up fixed something much smaller but still worth noting: the per-account version of the cost source picker had been showing a heading and then the picker’s own label directly beneath it, repeating the same words twice. The fix hides the redundant label visually while keeping it available to screen readers, so the control’s accessible name survives even though a sighted user no longer sees it twice. Five feature flags that had defaulted to off for months, plus the deletion of nine older cost flags that predated this whole effort, mark the actual moment this became the only cost pipeline ZopNight runs.

Tagged
Riya Mittal

Riya Mittal

Engineer · Zop.Dev

Riya is an AI engineer at ZopDev, working on production LLM pipelines behind the company's content and account-intelligence platforms. She works on the engineering that makes these systems reliable and repeatable, from multi-provider orchestration and structured output validation to evals, idempotent pipelines, and automated recovery. She writes about what it takes to make AI systems reliable enough to run in production.

Stop watching the waste.
Start cutting it.

See. Find. Fix. Automatic.

Connect your first cloud account in under 5 minutes. See your first remediation in under 7. No credit card required.

CDCR continuous detection, continuous remediation
full audit every action traceable
read-only default access
Multi-cloud automation· Production-ready in 30 min· SOC 2 · ISO 27001· 20–60% off the bill, first month· 4 platforms · 1 console· Multi-cloud automation· Production-ready in 30 min· SOC 2 · ISO 27001· 20–60% off the bill, first month· 4 platforms · 1 console·