Outcome
By the end of this lesson, you will be able to write a signal from the concept, aggregation, operator and window vocabulary, and avoid the exact-equality trap that makes a policy structurally incapable of firing.
| Tier | Engineer |
| JTBD | ”Express the condition I actually mean, in a language small enough to reason about.” |
| Personas | Platform Engineer · FinOps Lead · SRE |
| Prerequisites | M2.14.L1 |
| Time | 9 minutes |
| Bloom verb | Write (Apply), Avoid (Analyze) |
1. Concept
A signal is four things:
<aggregation> <concept> <operator> <threshold> over <window>
avg cpu lt 5% 14 days max db_connections eq 0 30 days p95 queue_depth gt 800 7 daysConcepts
A concept is a canonical metric name that resolves to the right provider-native metric on each cloud, so one policy works across AWS, GCP and Azure.
COMPUTE cpu, memory, gpu, gpu_memoryNETWORK network_in, network_outDISK disk_read_ops, disk_write_ops, iops_utilQUEUEING queue_depth, messages, throttled_opsDATABASE db_connections, db_qpsSERVERLESS invocations, durationQUALITY error_rate, cache_hitLIFECYCLE uptimeCAPACITY request_unitsYou may also name a raw provider-native metric directly. That is the escape hatch when a concept does not exist for what you need, and the cost of using it is that the policy stops being portable across clouds.
Aggregations
avg the mean over the windowmax the per-hour EXTREME, then the max of thosemin the per-hour extreme, then the min of thosep95 the 95th percentile OF PER-HOUR PEAKSp99 the 99th percentile of per-hour peaksThe percentile definition matters and is easy to misread. p95 is not the 95th percentile of raw samples; it is computed over per-hour peaks. A workload that spikes for two minutes every hour looks far busier under p95 than the raw samples would suggest. That is what you want when sizing for peak. It is the opposite of what you want when asking whether the thing is idle.
Operators
lt <gt >eq == EXACT FLOAT EQUALITYThe eq trap, and how it is prevented
eq is exact float equality, and that makes it a hazard with continuous aggregations.
REJECTED AT THE WRITE BOUNDARY eq paired with avg, p95 or p99
"avg cpu eq 0" can never be true in practice: an average over a fortnight of floating-point samples will not land on exactly 0.0. The policy would save cleanly and then never fire, forever.
Validation rejects it at save rather than letting you discover it in three months.
ALLOWED eq paired with max or min
"max db_connections eq 0" is meaningful and correct: the peak was exactly zero, so nothing connected at all during the window. That is a genuinely useful idle signal for a database.This is the clearest example of the authoring surface protecting you from a policy whose failure mode is silence.
The window
windowDays: 1 to 90The ceiling is the metric store’s serving limit, not an arbitrary choice. Note that it is wider than some per-provider metric lookbacks, so a 90-day window on a provider whose retention is shorter will simply have less data to work with rather than erroring.
Choose the window to match the cycle of the thing you are measuring, not the confidence you want:
A DAILY batch workload 14 days (two weeks of runs)A WEEKLY report warehouse 30 days (four cycles)A MONTH-END process 90 days (three closes)An always-on service 7 daysA window shorter than one full cycle of the workload will produce findings that are artefacts of when you looked.
Byte-rate thresholds
Thresholds expressed as byte rates (“1 MB/s”) are canonicalised to base units when the policy is ingested, and the drawer humanises them back for display. So you type what you mean and read what you typed, and the stored spec is unambiguous.
Combining signals
DECISION all-of every signal must fire (AND) any-of any signal firing is enough (OR)Nested boolean logic such as (A AND B) OR C is not available; it is deferred. In practice this is rarely limiting, because a policy that needs nested logic is usually two policies.
The multi-signal case is where watch policies earn their keep. A single low-CPU threshold produces noise in most estates. Low CPU and near-zero network and zero database connections is a very different claim, and it is one no shipped rule makes.
2. Demo
Three signals, one of which was rejected:
ATTEMPT 1 "find idle databases"
scope type = rds signal avg db_connections eq 0 over 30 days decision all-of
REJECTED AT SAVE. eq with a continuous aggregation. An average over 30 days of floating-point connection counts will not land on exactly 0.0, so this policy could never fire.
ATTEMPT 2 the fix
signal max db_connections eq 0 over 30 days
ACCEPTED. The per-hour peak being exactly zero means nothing connected during the entire window, which is precisely the claim intended.
RESULT: 4 findings. All 4 genuinely unused: two post-migration leftovers and two dev instances from a project that ended in April.
ATTEMPT 3 "under-filled batch nodes", refined
scope resource_group = batch-compute signals avg cpu lt 15% avg network_in lt 5 MB/s decision all-of window 14 days
WHY TWO SIGNALS CPU alone produced 23 findings, of which 14 were nodes doing legitimate low-CPU I/O-bound work. Adding the network floor cut it to 9, of which 6 were real.
WHY 14 DAYS The batch cycle is daily, so 14 days is two weeks of runs. At 3 days the same policy produced findings that were artefacts of a bank holiday.
THE PATTERN WORTH TAKING AWAY The first draft of a policy is almost always one signal and too short a window. Both failure modes produce findings you then have to hand-triage, which is the cost you were trying to avoid.3. Hands-on (6 min)
1. Write your signal in longhand first: ______ ______________ ______ ________ over ____ days (agg) (concept) (op) (thresh) (window)
2. Check the eq rule: Are you using eq? Y / N If Y, is it paired with max or min? Y / N If N to the second, it will be rejected. Rewrite it.
3. Window check: what is the natural CYCLE of the workload you are measuring? ______ days Is your window at least two cycles? Y / N
4. Single-signal check: run your policy mentally against three resources you KNOW are healthy. Would it fire on any of them? Y / N If Y, add a second signal with all-of rather than loosening the threshold.
5. If you need (A AND B) OR C, split it: policy 1: ____________________________ policy 2: ____________________________4. Knowledge check
Q1
A policy specifies avg cpu eq 0 over 14 days. What happens?
A. It fires on any resource with no CPU activity
B. It is accepted but converted to lt 0.01
C. It fires only on stopped resources
D. It is rejected at the write boundary
Show answer
Correct: D. eq is exact float equality, and an average over a fortnight of floating-point samples will not land on exactly 0.0, so the policy could never fire. Validation rejects it at save rather than letting you discover the silence months later. eq with max or min is allowed and useful: max db_connections eq 0 means the per-hour peak was exactly zero, so nothing connected during the whole window. The rejection is specifically for continuous aggregations (avg, p95, p99).
Q2
p95 in a watch policy signal computes:
A. The 95th percentile of raw metric samples
B. The mean of the top 5% of samples
C. The 95th percentile of per-hour peaks
D. The 95th percentile over the last hour only
Show answer
Correct: C. A workload spiking for two minutes every hour looks far busier under this definition than a raw-sample percentile would suggest, which suits capacity questions and does not suit “is this basically idle”. Reading it as a raw-sample percentile is the common misreading and it leads to thresholds that never fire on bursty workloads. max and min likewise read the per-hour extreme.
Q3
A first-draft policy on low CPU produces 23 findings, 14 of which are legitimate I/O-bound workloads. The better fix:
A. Lower the CPU threshold until all the false positives disappear
B. Add a second signal with an all-of decision, such as a network floor
C. Narrow the scope to exclude those 14 resources
D. Switch the aggregation to p95
Show answer
Correct: B. The combination is the claim no shipped rule makes; tightening a single threshold trades false positives for false negatives without making the policy more specific. Multi-signal all-of policies are where watch policies earn their keep. C is a maintenance trap: an exclusion list drifts, and the next I/O-bound node added to the group re-introduces the false positive.
5. Apply
Write your signal in longhand before opening the wizard, and check the eq rule and the window-versus-cycle question before you save. Those two account for most first-draft policies that either never fire or fire on everything.
If your first pass produces more findings than you can hand-check, add a signal rather than moving the threshold.
Related lessons
- L1: When to write a watch policy
- L3: Outcomes and the savings basis (next)
- L4: Evaluation and its limits
- T2.M2.2.L1: The Metrics drawer
Glossary terms touched
Concept metric · Per-hour peak · windowDays · all-of decision