Outcome
By the end of this lesson, you will be able to pick the right dimension for an anomaly investigation, cascade from broad to narrow, and recognize the noise-reduction patterns.
| Prerequisites | M2.1 (rule library) |
| Time | 9 minutes |
1. Concept
ZopNight detects anomalies across 7 dimensions: org, cloud account, resource type, resource group, resource, team, and the Azure-specific hierarchy (resource group parsed from the ARM resource id, and tenant resolved through the parent-account hierarchy). Each dimension catches different kinds of events.
THE SEVEN DIMENSIONS:
org-level Org-wide spike Multi-team event; cloud-wide pricing
cloud-account Per-account spike New environment in one account
resource-type Per-type spike One service class moved together: every NAT gateway, every GPU node
resource-group Per-group spike Group's workload scaled up unexpectedly
resource Per-resource spike Single resource misbehaving
team Per-team spike Team's spend pattern changed
azure hierarchy Per Azure resource-group, per tenant Parsed from the ARM resource id and the parent-account chain, so an Azure estate is attributable the way its owners organise itEach dimension catches different patterns. The investigation cascade is the technique.
How dimensions cascade
TIME T: Org-level spike detected at +200%
Drill 1: cloud-account dimension Account "prod-aws" spike +150% Other accounts: normal → Focused: prod-aws is the source
Drill 2: team dimension on prod-aws Team "platform" spike +180% Team "data" normal Team "search" normal → Focused: platform team's resources
Drill 3: resource dimension on platform team Resource "i-0xyz" spike +1,200% Other resources: normal → Diagnosed: i-0xyz is driving the org spikeInvestigating from broadest to narrowest dimension identifies the root cause. The cascade is the technique.
When to use which dimension
QUESTION DIMENSION TO USE──────────────────────────────────────────────────────────"Are we spending more than yesterday?" org-level"Which cloud is the issue?" cloud-account"Which environment is spiking?" cloud-account or resource-group"Which team is over-spending?" team"Which specific resource is misbehaving?" resource"Which workload?" resource-groupThe right dimension matches the question.
Resource-level records are not capped
The resource dimension has no top-10 cap. Every resource that crosses the warning band is recorded; only the root-cause enrichment is capped, at 20 anomalies per org per run:
EXAMPLE: 20 resources spike simultaneously All 20: recorded as resource anomalies Root-cause text: added for up to 20 per org per run
NOTE: Every dimension below org runs only when the org has a dimension-level anomaly subscription. Org-level detection runs for every org.Noise is controlled by the bands, the guards and your subscriptions, not by a fixed cap.
Adaptive batching
Anomaly detection runs on different cadences based on org size:
ORG SIZE BATCH SIZE FREQUENCY──────────────────────────────────────────────────────────≤ 5,000 resources 25 orgs per batch Daily 20:55 UTC≤ 20,000 resources 5 orgs per batch Daily 20:55 UTC> 20,000 resources 1 org at a time Daily 20:55 UTCLarger orgs process more carefully to manage memory and API load. The detection scales sub-linearly with resource count.
What anomaly detection IS NOT
NOT real-time alerting: Daily cadence; not minutes-level For real-time use cloud-native alarms (CloudWatch, etc.)
NOT replacing budget alerts: Budget alerts: threshold-based ("spent X% of budget") Anomaly detection: deviation-based ("spending 200% of normal") Different signals
NOT predictive: Reactive (anomalies after the fact) Not "will spend"; "spent unexpectedly"
NOT actionable directly: Anomaly = diagnostic surface + root cause analysis (L4) + remediation through other mechanisms Not a one-click "fix the anomaly"Anomaly detection is one tool; use it for what it’s designed for.
Combining with budget alerts
BUDGET ALERTS catch when: Spend exceeds threshold (e.g., "team-X used 80% of $50K budget") Forecasted to exceed by month-end
ANOMALY ALERTS catch when: Spend deviates from normal pattern (e.g., "today is 200% of typical") Statistical anomaly regardless of absolute amount
USE BOTH: Budget for "are we within plan?" Anomaly for "is something wrong?" Catch different scenariosThe two systems are complementary. Most teams use both.
Severity bands (one global set, not per-dimension)
The severity bands are the same for every dimension. There is a single global set of thresholds on percent deviation:
SEVERITY PERCENT DEVIATION RESPONSE──────────────────────────────────────────────────────warning 30% to under 100% note; don't pagecritical 100% to under 500% investigate within hoursemergency 500% or more page on-call; immediate responseAt the org level and for team, resource-group and Azure dimensions, a z-score path runs in parallel (emergency at z >= 5) and the higher severity of the two wins; cloud account, resource type and resource use percent only. The bands are not tuned per dimension.
When dimensions don’t help
NEW WORKLOADS: Insufficient history; anomalies fire too easily Solution: ZopNight uses 7-day rolling baseline New workloads have warmup period
SEASONAL PATTERNS: E-commerce spikes in November ML workloads run nightly No day-of-week baseline: ZopNight compares against a flat 7-day rolling average that excludes days already flagged
LARGE BURSTS: Single huge planned event (Black Friday) Still raises anomalies: detection has no event calendar Expect them, and note the event when you close them outDetection handles new workloads and flagged days for you. Planned bursts are yours to recognise.
2. Demo
A team’s anomaly investigation:
T+0 Daily anomaly cron runs (20:55 UTC)T+30s System detects: Org spend yesterday: $4,200 above 7-day average +32% above baselineT+30s Org-level WARNING anomaly firedT+30s Slack notification to #finops-alerts
T+5 min FinOps engineer opens anomaly feedT+5 min Drilling: cloud-account dimension prod-aws-us-east-1: +95% (significant) prod-aws-eu-west-1: normal staging-aws: normal → Focused: us-east-1 is the source
T+10 min Drilling deeper: team dimension on prod-aws-us-east-1 team=ml: +180% (major) team=platform: +10% (background normal) team=search: normal team=billing: normal → Focused: ML team
T+15 min Drilling: resource dimension on team=ml i-0ml-train-99: was $0/hr yesterday, $560/hr today (vs prev 30 days average: $0/hr: instance was off) Other team=ml resources: normal → Identified: i-0ml-train-99 specifically
T+18 min DIAGNOSIS: A long-running ML training job spawned 7 large GPU instances overnight that weren't terminated Verified via CloudWatch logs + audit trail
Action: manual termination of the 7 instances Cost stopped; recovery: 5 min
T+25 min Cost back to normalT+25 min Anomaly note: incident-2026-05-21T+1 day Postmortem: CI bug fixed; review process improved15 minutes from detection to diagnosis using dimensional drill-down.
3. Hands-on (5 min)
Practice anomaly investigation:
□ STEP 1: Open Cost Reports and click an anomaly marker on the daily cost chart (or use View All on the dashboard anomaly banner) Anomalies in range: _____
□ STEP 2: For one anomaly, identify dimension Started at: __________ Dimension: __________ Severity: __________
□ STEP 3: Practice cascade Drill 1: __________ → narrowed to __________ Drill 2: __________ → narrowed to __________ Drill 3: __________ → identified __________
□ STEP 4: Estimate time Total investigation time: ___ minutes Diagnostic confidence: _____%
□ STEP 5: Note pattern How did the cascade help vs random search? Future improvement: __________A 10-minute practice builds the cascade habit.
4. Knowledge check
Q1
An org-level anomaly fires. The right first drill is:
A. resource dimension immediately
B. service dimension, to find the offending service first
C. cloud-account dimension: narrow from broadest to narrowest
D. team dimension
Show answer
Correct: C. Go broad to narrow: once the account is identified, drill into team or resource as needed. Skipping levels often misses the cause.
Q2
Twenty resources in a subscribed org each spike above 30% on the same day. Resource-level detection:
A. Records only the top 10 by deviation
B. Records all 20 as resource anomalies
C. Records none until the org total moves
D. Records only the single largest spike
Show answer
Correct: B. The resource path has no top-10 cap: every resource over the warning band is recorded. Only root-cause enrichment is capped, at 20 per org per run.
Q3
Anomaly detection runs:
A. Real-time
B. Hourly
C. Daily at 20:55 UTC: reactive, not real-time
D. Every 6 hours, alongside the discovery run
Show answer
Correct: C. For real-time alerting, use cloud-native alarms or budget thresholds. Anomaly detection identifies unexpected deviations after the fact. Daily cadence.
5. Apply
Anomalies show as markers on the Cost Reports daily chart and in the dashboard anomaly banner. Dimensional drill-down available.
For your team: practice the cascade. Build the muscle memory for broad → narrow investigation.
Related lessons
- L2: Detection methods (next)
- L3: Severity bands
- L4: Root cause
- L5: Redistribution suppression
- T5.M5.5: Anomaly response