Outcome
By the end of this lesson, you will be able to explain the fail-closed evaluation contract, predict why a watch policy never fires on missing data, and name what is deferred from the current version so you do not design around something that is not there.
| Tier | Engineer |
| JTBD | ”Know whether my policy producing nothing means the estate is clean or the policy is broken.” |
| Personas | Platform Engineer · FinOps Lead · SRE |
| Prerequisites | M2.14.L3 |
| Time | 9 minutes |
| Bloom verb | Explain (Understand), Predict (Apply), Name (Remember) |
1. Concept
Two rules govern evaluation, and both fail in the safe direction.
Rule 1: matching fails closed
MatchResource fails closed on a malformed or unknownselector.
A scope you typed wrong does not quietly match EVERYTHING.It matches NOTHING.The alternative would be catastrophic: a terminate outcome on an accidentally-broad scope is exactly the failure you cannot afford. So a selector that cannot be resolved produces an empty match set.
The diagnostic consequence: a policy producing zero findings might have a broken selector. That is the first thing to check, not the threshold.
Rule 2: a policy never fires on missing data
EvaluateSignal treats an ABSENT or ALL-GAP series asNO-FIRE.This is the same discipline you met in the Bedrock rules (M2.11) and in verification (M2.13.L3), and it is worth stating as a general principle because it recurs everywhere in this product:
Absence of data is not evidence of a condition.
A resource that emits no CPU metric at all is not a resource with 0% CPU. It might be a resource whose metrics ingestion is broken, whose permissions were narrowed, or which the aggregator has not reached yet. Firing “avg cpu lt 5%” on it would recommend deleting things on the strength of a monitoring gap.
THE COUNTER-CASE THAT MAKES THE POINT
A policy: avg cpu lt 5% over 14 days, outcome terminate An account whose CloudWatch permission was revoked three weeks ago.
WITHOUT the no-fire rule: every resource in that account now has an absent CPU series, reads as 0%, and gets a delete recommendation with a full-monthly-bill savings figure attached.
WITH it: nothing fires, and Permission Visibility shows you the denial.Combining, and what the decision does
all-of every signal must fireany-of any signal firing is enough
A signal that NO-FIRES because its data is missing issimply not firing. Under all-of, that means the policydoes not fire. Under any-of, the other signals still can.That asymmetry is worth knowing when you choose a decision type. An any-of policy is more robust to a single metric going missing, and correspondingly more likely to fire on a partial picture.
Why a new policy may produce nothing yet
Policies resolve through a change event, published per edit and processed asynchronously. There is an org-wide admin backfill for evaluating a new policy against existing resources rather than only from the point of creation.
SO, "MY NEW POLICY HAS NO FINDINGS" HAS FOUR CAUSES,IN THE ORDER WORTH CHECKING:
1. The change event has not resolved yet, or the backfill has not been run. 2. The selector is malformed and matched nothing (fail-closed). 3. The metric is absent for resources in scope, so every signal no-fires. 4. The threshold is genuinely not met. The estate is clean on this axis.
Most people assume 4 and it is usually 1, 2 or 3.What is deferred
Stated plainly so you do not design a workflow around it:
NOT AVAILABLE TODAY notify dispatch from a watch policy Jira / ticket dispatch from a watch policy nested boolean decisions, (A AND B) OR C uptime_pct / uptime_source stamped on the custom row for later audit (the fold IS applied, just not recorded) a realizability tierThe first two matter most in practice. A watch policy produces a recommendation, and that is where it stops. If your intended workflow is “policy fires, ticket opens”, that chain does not exist yet: the built-in open_ticket policy effect covers built-in recommendations, not watch findings.
What it shares with built-in findings
Worth restating, because it is easy to assume a user-authored finding is second class:
SAME drawerSAME savings basis and pricing helpersSAME RBACSAME $5 low-savings floorSAME Dismiss and Mark Resolved actionsA watch finding is a first-class recommendation in every respect except which engine produced it and which tab it appears on.
2. Demo
Debugging a policy that produced nothing:
THE POLICY scope type = ec2, tag environment = staging signal avg cpu lt 10% over 14 days outcome rightsize FINDINGS 0
THE TEAM'S ASSUMPTION "Our staging fleet must be well-sized."
CHECK 1: has the change event resolved? Policy saved 20 minutes ago. Resolution is async. -> waited, re-checked. Still 0. Not this.
CHECK 2: is the selector matching anything? Scope preview: 0 resources. FOUND IT. The tag key in the estate is `Environment`, capitalised, and the selector said `environment`.
MatchResource failed closed, as designed. The scope matched nothing rather than matching everything, which is the behaviour you want when the outcome could have been terminate.
CHECK 3 (after fixing the selector): 41 resources matched FINDINGS 0, still.
Metrics check: 41 of 41 have an ABSENT cpu series. Permission Visibility on that account: CloudWatch GetMetricData -> DENIED, 19 days ago.
Every signal no-fired because the data was missing. Exactly correct: had it fired, 41 staging instances would have carried rightsize recommendations built on a monitoring outage.
AFTER RESTORING THE PERMISSION FINDINGS 12. Nine were real.
THE TIMELINE THIS PRODUCED A "policy is broken" ticket surfaced a three-week-old permissions regression that nothing else had flagged, because no built-in rule in that account was metric-dependent enough to go quiet visibly.3. Hands-on (6 min)
1. Your policy's scope preview: how many resources match? ______ If 0, check for a case-sensitivity or typo issue in the selector before touching anything else.
2. For 3 matched resources, confirm the metric exists: resource ____________ series present? Y / N resource ____________ series present? Y / N resource ____________ series present? Y / N
3. Check Permission Visibility for the accounts in scope. Any metric-related denials? Y / N Date of the oldest one: ____________
4. Work the four causes in order for any policy of yours producing nothing: [ ] 1 event not resolved / backfill not run [ ] 2 selector matched nothing [ ] 3 metric absent, signals no-fired [ ] 4 threshold genuinely not met
5. Confirm the deferred list against your intended workflow. Does your plan depend on a policy opening a ticket? Y / N If Y, redesign now: that chain does not exist.4. Knowledge check
Q1
A watch policy’s selector contains a typo and cannot be resolved. The policy will:
A. Match every resource in the org, since no filter applies
B. Match nothing
C. Be rejected at save time
D. Fall back to the previous version of the selector
Show answer
Correct: B. MatchResource fails closed on a malformed or unknown selector, because a broad accidental match combined with a terminate outcome is the failure that cannot be afforded. The diagnostic consequence is the useful part: a policy producing zero findings might have a broken selector, and that is the first thing to check rather than the threshold. Case sensitivity in tag keys is the most common cause.
Q2
A resource in scope emits no CPU metric at all. A signal reading avg cpu lt 5%:
A. Fires, since no CPU usage means the resource is idle
B. Fires with reduced severity
C. Causes an evaluation error for the whole policy
D. Does not fire
Show answer
Correct: D. An absent or all-gap series is treated as no-fire, because absence of data is not evidence of a condition: the resource might have broken metrics ingestion, narrowed permissions, or simply not have been reached yet. Consider the counter-case: an account whose CloudWatch permission was revoked would have every resource read as 0% CPU and receive a delete recommendation with a full-monthly-bill savings figure. The no-fire rule is what prevents a monitoring gap becoming a deletion worklist.
Q3
Which of these workflows can a watch policy support today?
A. Fire, then automatically open a Jira ticket
B. Fire, then surface a recommendation on the User-Defined tab which a human triages
C. Fire, then notify a Slack channel
D. Fire on a nested boolean condition such as (A AND B) OR C combined together
Show answer
Correct: B. Notify and ticket dispatch from watch policies are deferred; the built-in open_ticket policy effect covers built-in recommendations, not watch findings. C and A are the same deferred capability, and D is the deferred decision syntax; today the decision is all-of or any-of. A policy needing nested logic is usually two policies.
5. Apply
When a policy of yours produces nothing, work the four causes in order rather than adjusting the threshold. Most “broken policy” reports are a selector typo or an absent metric, and the absent-metric case is frequently a permissions regression worth finding on its own merits.
Check the deferred list against any workflow you are planning before you build around it.
Related lessons
- L3: Outcomes and the savings basis
- L1: When to write a watch policy
- T1.M1.1.L4: Permission Visibility
- T5.M5.9.L3: Policy-gated auto-ticketing
Glossary terms touched
Fail closed · No-fire · Scope selector · Backfill