Outcome
By the end of this lesson, you will be able to explain why a verification baseline is written once and never recomputed, describe what a rule declares as its finding observable, and reject “the rule stopped firing” as evidence of a fix.
| Tier | Engineer |
| JTBD | ”Trust a verification verdict enough to put it in front of finance.” |
| Personas | Platform Engineer · FinOps Lead |
| Prerequisites | M2.13.L2 |
| Time | 9 minutes |
| Bloom verb | Explain (Understand), Describe (Understand), Reject (Evaluate) |
1. Concept
A verdict is only as good as what it is measured against. Two mechanisms make the measurement trustworthy, and both are about refusing to let the target move.
Mechanism 1: the frozen before-picture
When a recommendation is emitted, the engine stamps what it asked for and captures a baseline of the resource as it was.
STAMPED AT EMISSION, THEN NEVER RECOMPUTED target_spec for a resize: the shape we asked for finding_target for a posture finding: the thing we objected to finding_baseline the resource as it stood at t0All are written once. The reason is a failure mode worth stating in full:
IF THE TARGET MOVED WITH THE CUSTOMER
Day 0 rec asks: m5.2xlarge -> m5.xlarge Day 12 customer resizes to m5.xlarge. Correct. Day 13 the rule re-evaluates the NEW shape and now asks: m5.xlarge -> m5.large Day 14 verification compares current (m5.xlarge) against target (m5.large) -> NOT ADOPTED
Someone who did exactly what was asked can never match the target, because the target chased them.A recomputing target makes adoption unachievable. So the target is frozen, and a reopen clears both target and baseline so the next observation window starts with a fresh, matching pair.
Mechanism 2: rules declare what to re-read
A posture rule names the single resource key its finding fired on, through FindingObservable. Verification re-reads that exact key.
RULE: "this bucket has no lifecycle policy" FindingObservable: the lifecycle-policy key
VERIFICATION re-reads THAT KEY, and asks: did it flip to the compliant value?This is a declaration by the rule author, not an inference by the verifier. The author knows which field their gate read; a category-wide guess does not.
Why “the rule stopped firing” is not evidence
This is the load-bearing rejection in the whole design, and it is the thing most systems get wrong.
THE TEMPTING SHORTCUT "The rule no longer fires on this resource, so it must have been fixed."
WHY IT FAILS A genuine fix and these all produce IDENTICAL silence:
- a narrowed IAM role, so the rule can no longer read the field it evaluates - a renamed or restructured provider field - the resource dropping out of discovery entirely - a rule edit that changed its own threshold - a metrics fetch that failed for that resource
Silence has many causes and only one of them is success.So the verifier re-reads the declared observable and requires it to have moved to the compliant value. Absence of a complaint is not presence of a fix.
This is the same discipline you met in the Safety Gate (M2.3.L7): absent evidence is not evidence, and it fails closed.
The read-back gap, stated honestly
A resize rule can target any of 19 target-spec keys. The discoverer’s resource row carries only three of them as first-class columns.
FIRST-CLASS COLUMNS instance_type size_gb unit_count
IN metadata JSON, WITH PARTIAL PROVIDER COVERAGE iops, throughput, volumeType, memorySize, target_replicas, desiredCount, ecsTaskCpu, ...So shape verification needs a read-side map from target-spec key to where it can be read back. That map is built per key, with abstention as the default: an unmapped key yields abstained, never verified_unchanged.
The direction of that default is the whole point. An unmapped key means “we cannot see this”, and reporting that as “unchanged” would silently accuse a customer of not acting. Adding a key later upgrades those recommendations from abstained to a real verdict, which is a monotone improvement rather than a correction.
What this costs
Freezing baselines and requiring declared observables means the system verifies less than a naive implementation would claim to. That is the trade, and it is the right one. A smaller set of trustworthy verdicts beats a complete set of guesses, especially when the output is a savings figure somebody has to defend to a CFO.
2. Demo
Two recommendations on the same resource, one verifiable and one not:
RESOURCE: i-0c47a2f9b, an m5.2xlarge in prod-us-east
REC A: RC-003 EC2 Rightsizing emitted 2026-06-02 target_spec: instance_type = m5.xlarge finding_baseline: instance_type = m5.2xlarge (FROZEN at emission)
2026-06-19 the team resizes to m5.xlarge 2026-06-20 verification reads instance_type current = m5.xlarge target = m5.xlarge MATCH verdict = verified_adopted
Note what did NOT happen: the rule re-evaluated the m5.xlarge on 20 June and emitted a NEW recommendation suggesting m5.large. That is a separate recommendation with its own frozen target. It did not retroactively move Rec A's goalposts.
REC B: a posture rule on the same instance emitted 2026-06-02 FindingObservable: detailed_monitoring_enabled finding_baseline: false
2026-06-20 verification re-reads that exact key current = false verdict = verified_unchanged
A WEEK LATER the rule stops firing entirely. Naive reading: "fixed on 27 June." Actual cause: the IAM role was narrowed during an unrelated permissions cleanup, and the provider call that reads that field now returns AccessDenied.
Verification re-read the declared key, could not read it, and recorded `abstained`. It did NOT record a fix.
Permission Visibility (T1.M1.1.L4) showed the denial the same day, which is how the cleanup was caught.The Rec B ending is the case that justifies the whole design. Under a stopped-firing heuristic, a permissions regression would have been reported to leadership as a compliance improvement.
3. Hands-on (6 min)
1. Find a rightsizing recommendation that was applied. Open it. What is its target_spec? ______________________________________________ Does the resource's CURRENT shape match? Y / N
2. Find a recommendation on a resource that has since been resized TWICE. Are there two recommendations with two different frozen targets? Y / N (There should be. One rec never chases.)
3. Find an `abstained` verdict on a resize. Which target-spec key did it want to read? ______________________________________________ Is that key one of the three first-class columns? Y / N
4. Check Permission Visibility for your accounts. Any newly-denied entries in the last 30 days? Y / N If yes, cross-check whether any rules went quiet at the same time: ______________________________
5. Write the one-sentence reason you would give a colleague for why "the rule stopped firing" is not accepted as proof: ______________________________________________Step 4 is the habit worth keeping. A permissions change and a wave of quiet rules on the same day is the signature this design exists to catch.
4. Knowledge check
Q1
Why is target_spec written once at emission and never recomputed?
A. To reduce database writes
B. Because instance types are immutable
C. Because recomputation would break the audit trail, and an auditor reading the record months later would have no way of telling what the recommendation originally asked for at the time it was raised
D. Because a target that moved with the customer would make adoption unachievable: someone who resized exactly as asked would be re-evaluated at their new shape, given a new target, and measured against that instead
Show answer
Correct: D. A reopen clears the target and baseline so the next window starts with a fresh matching pair. The rule genuinely does re-evaluate the new shape and may emit a further recommendation, but that is a separate recommendation with its own frozen target. It does not retroactively move the first one’s goalposts.
Q2
A posture rule stops firing on a resource. Verification treats this as:
A. Not evidence
B. Evidence the finding was fixed
C. A reason to reopen the recommendation
D. A reason to close the recommendation as optimised
Show answer
Correct: A. A genuine fix, a narrowed IAM role, a renamed provider field, a resource dropping out of discovery and a failed metrics fetch all produce identical silence. Verification re-reads the key the rule declared through FindingObservable and requires it to have moved to the compliant value. This is the same fail-closed discipline as the Safety Gate: absence of a complaint is not presence of a fix. Under a stopped-firing heuristic, a permissions regression gets reported to leadership as a compliance improvement.
Q3
A resize recommendation targets a key that has no read-side mapping. The verdict recorded is:
A. verified_unchanged, since the change cannot be confirmed
B. abstained
C. pending, until a mapping is added
D. verified_diverged
Show answer
Correct: B. An unmapped key means “we cannot see this”, and reporting that as unchanged would silently accuse the customer of not acting. Adding a mapping later upgrades those recommendations to a real verdict. The direction of the default is the design decision. Only three of the 19 target-spec keys are first-class columns on the resource row; the rest live in metadata JSON with partial provider coverage, so abstention is common and honest rather than exceptional.
5. Apply
When a verdict surprises you, check the frozen baseline before questioning the probe. Most surprising verdicts are a resource that changed twice, and the frozen target is doing exactly what it should.
When a batch of rules goes quiet at once, check Permission Visibility the same day. Silence with a permissions cause looks identical to silence with a fix cause, and only one of them is good news.
Related lessons
- L2: Status is a claim, evidence is a fact
- L4: The seven verification buckets (next)
- T2.M2.3.L7: The Recommendation Safety Gate
- T1.M1.1.L4: Permission Visibility
Glossary terms touched
Frozen baseline · target_spec · FindingObservable · Abstain