A FinOps Finding Going Quiet Is Not Proof It Was Fixed
Every recommendations engine has the same convenient bug, and in ZopNight it accounted for 84.6% of all closures. A rule fires, raises a finding, and on the next scan the rule does not fire. The engine closes the finding, adds its projected saving to a running total, and shows somebody a number that went up.
Nothing in that sequence checked whether anything was fixed.
Call it the silence fallacy: treating the absence of a finding as evidence of the thing the finding asked for. It is the cheapest possible verification, it is wrong in a specific direction, and it is wrong most often exactly when the scanner itself is degraded. ZopNight now re-reads the resource to confirm the outcome instead. All 264 rules that can raise a recommendation have a verification path, and a rule that stopped firing is explicitly not accepted as evidence.
Silence Has Four Causes and Only One of Them Is a Fix
When a finding stops appearing, at least four things could have happened, and the engine sees an identical signal for all of them.
| Why the finding went quiet | What actually happened | What the silence fallacy reports |
|---|---|---|
| Somebody acted on it | The disk was resized, the instance was stopped, the snapshot was deleted | Fixed, correctly |
| The scanner lost permission | An IAM role was narrowed and the resource is no longer readable | Fixed, incorrectly |
| The provider renamed a field | The rule’s input no longer resolves, so the condition cannot evaluate | Fixed, incorrectly |
| The resource disappeared for other reasons | An account was moved, a region was dropped from the scan scope | Fixed, incorrectly |
Three of the four produce a false claim, and the failure is not random. A narrowed IAM role silences many rules at once, so a permissions regression looks like a productive week. A renamed provider field silences one rule across the whole estate, so a provider-side change looks like a coordinated cleanup nobody remembers doing.
FinOps is the practice of giving engineering, finance and product teams shared, current visibility into cloud spend, so the people generating cost decide about it while they still can. A savings total nobody can audit is the failure mode that practice exists to prevent. The savings total inherits that bias. It is a figure that grows fastest when the scanner is broken, which is the exact inverse of what it is supposed to measure. Anyone who has tried to reconcile a FinOps tool’s claimed savings against an actual invoice has met the consequence: the tool insists on a number the bill has never heard of, and there is no way to interrogate which findings the number is made of.
A Rule Now Declares What to Re-Read Before It Is Allowed to Close Anything
The fix is structural, and it is imposed on the rules rather than on the closing logic. Each rule declares a FindingObservable: the single resource key the finding fired on. Verification re-reads that exact key.
Declaring the key at rule-definition time is what makes this inexpensive enough to run. Verification does not re-scan the estate looking for the resource, and it does not re-evaluate the rule. It reads one key and compares it against what the finding recorded when it fired. If the key is unreadable now, that is a different answer from “the value changed”, and the engine says so instead of collapsing both into a close.
This also gives the honest answer in the IAM case. A narrowed role means the re-read fails. A failed re-read is not a fix, it is an abstain, and an abstain does not pay into the savings total.
Six Probes, and a Rule That Fits None of Them Abstains by Construction
Six probe types cover what a recommendation can ask a resource to change. Each one re-reads a different property.
| Probe | What it re-reads | Example finding it verifies |
|---|---|---|
| Existence | Whether the resource is still there | An orphaned volume or an unattached address |
| Power state | Whether it is running or stopped | An instance that should be stopped outside business hours |
| Shape | Its size, class or tier | A right-sizing recommendation on an oversized instance |
| Duty cycle | How much of the time it is actually in use | A workload idle enough to schedule or consolidate |
| Setting | A specific configuration value | A retention setting, a flag, an encryption or logging option |
| Rate | The price basis it is billed at | A move onto a commitment or a different pricing tier |
A rule that fits none of the six abstains, by construction rather than by exception. It reports no_baseline, not_observable or posture_no_cost_outcome and takes no credit.
That abstain state is the load-bearing part of the design, and it is the part most engines skip. A verification system with no way to say “I do not know” will manufacture a verdict for every rule it does not understand, and those manufactured verdicts are indistinguishable from the real ones in the total. Once abstain exists and is used, a confirmed verdict means something narrow and checkable: this key was re-read, and it changed the way the finding asked it to.
The counts, measured across the shipped rule set, came out at 264 of 264 rules that can emit a recommendation having a verification path. Of those, 251 declare an observable directly, and 13 are verified through an authoritative lever instead. Five of the 269 rules in the engine cannot emit a recommendation at all, which is why the denominator is 264 rather than 269. On the raw base, per the same count, 251 of 269 is 93.3%.
Most Recommendations Used to Have No Verdict Available at All
Before this work, verification only existed for cost findings, because a cost finding is the only kind where a dollar delta was already being computed. Everything else abstained.
| Before | After | |
|---|---|---|
| Rules with a verification path | Cost findings only | 264 of 264 rules that can emit |
Recommendations abstaining with posture_no_cost_outcome, measured before the change | 84.6% | Reserved for rules that genuinely fit no probe |
| Basis for closing a finding | The rule stopped firing | The declared observable was re-read |
| Savings claimed on an unverified close | Yes | No |
84.6% is the number that reframes the whole feature. A verification system covering a sixth of its own findings is not a verification system with gaps, it is a savings total with a verification anecdote attached. Security, compliance and reliability findings had no verdict available in either direction, which meant the only recommendations anyone could audit were the ones already carrying a dollar figure.
Extending verification to the other categories required inverting the design. Instead of deriving a verdict from a cost outcome, the rule states in advance what would prove the outcome, and the engine checks that. A compliance rule about a logging setting verifies by re-reading the setting. A reliability rule about a backup configuration verifies by re-reading the configuration. Neither needs a price to produce an honest answer.
The Savings Verb Follows the Evidence, Not the Status
Status and evidence are separate axes, and conflating them is how a closed finding becomes a claimed saving. A recommendation can be closed because it was fixed, closed because it was dismissed, closed because the resource was deleted for unrelated reasons, or withdrawn because the rule that raised it changed.
The suppression is applied everywhere the figure appears: the list, the drawer and the subject line of the share email. That last one is worth calling out, because a share email is the surface most likely to be forwarded to somebody who cannot check it. A number that is honest in the product and optimistic in the email is not an honest number.
Rule versioning gets the same treatment from the other direction. When a rule changes, recommendations already applied under the old version are marked auto-optimised rather than withdrawn, because withdrawing them would erase work that genuinely happened. The status changes; the evidence does not.
A History That Outlives the Resource It Describes
The second half of the change is an audit trail. Every recommendation now carries a history covering raised, viewed, applied and closed, each with a reason. It renders in the drawer, it is deep-linkable, and it survives the resource it describes.
Surviving deletion is the requirement that makes the rest useful. The most expensive findings are the ones that end with a resource being deleted, and an audit trail keyed to a live resource loses exactly those. Six months later, when somebody asks why a cluster disappeared in the same week a bill dropped, the history is the only artefact that can answer.
This works when the declared observable is still readable at verification time. It fails when the platform loses access to the resource, and the correct output in that case is an abstain rather than a verdict. The schema work behind it was deliberately additive: ten nullable columns on the recommendations table, two new tables, no drops and no type changes, with every alter running in place and without locking. A verification feature that requires a maintenance window to deploy will be deployed late and rolled back under pressure, which is the wrong pressure to apply to the component whose job is to tell you the truth about your own savings.
