Outcome
By the end of this lesson, you will be able to map each anomaly severity to the right operational response, use notification subscriptions to control who hears each severity, and follow up on anomalies that persist.
| Prerequisites | M2.10.L1 · M2.10.L2 |
| Time | 9 minutes |
1. Concept
Spikes come in three sizes.
Each size has a response that suits it, somewhere it should be sent, and a sense of how quickly somebody ought to look.
SEVERITY DEVIATION RESPONSE SUGGESTED SUBSCRIPTION──────────────────────────────────────────────────────────────────────────────warning 30% to under 100% Investigate this week Slack channelcritical 100% to under 500% Action within 24 hours Slack + Teamsemergency 500% or more Action immediately Slack + webhook to your pagerThe severity drives the response intensity.
Warning: investigate-this-week
EXAMPLE: prod-database costs jumped from $80/day to $130/day (62%)
INTERPRETATION: meaningful change but not crisis-level
RESPONSE PLAN: Investigate within the week Determine if change is one-time or persistent Document the cause No immediate action needed
TYPICAL OUTCOMES: New workload deployment (expected; document) Auto-scaling kicked in for normal load (acceptable; verify) Marketing event traffic (expected; document) Configuration drift (investigate further)
RESOLUTION TIME: days; not hoursWarnings often resolve without remediation: just understanding.
Critical: action within 24 hours
EXAMPLE: dev-staging cluster costs went from $300/day to $1,400/day (367%)
INTERPRETATION: significant unexpected change
RESPONSE PLAN: Investigate within 24 hours Identify root cause Take corrective action Document for postmortem
TYPICAL OUTCOMES: Forgotten resource still running (kill it) Unauthorized resource provisioning (remove + alert) Stuck job consuming resources (investigate) Misconfigured autoscaler (adjust) Failed deployment leaving leaked resources
RESOLUTION TIME: hours; not daysFOLLOW-UP: no timer escalation; chase it through your own processCritical anomalies require human attention within a day.
Emergency: page on-call immediately
EXAMPLE: ml-training cost went from $400/day to $4,800/day (1,100%)
INTERPRETATION: extreme deviation; likely cost runaway
RESPONSE PLAN: IMMEDIATE: page on-call engineer Investigate within minutes Take corrective action ASAP Pause billing in worst case
TYPICAL CAUSES: Stuck multi-instance job spawning GPU instances Misconfigured autoscaler max exhausted Compromised credentials provisioning resources Buggy CI pipeline launching workloads Marketing tool runaway Bot/abuse driving consumption
RESOLUTION TIME: minutes; not hoursESCALATION: incident commander; security if compromise suspectedEmergencies require immediate human action.
Notification routing per severity
ZopNight has no built-in routing per severity. You build it with notification subscriptions, one per dimension and severity, each sent to a channel (Slack, Microsoft Teams, Google Chat or a webhook). There is no PagerDuty channel type; point a webhook at your pager instead.
SEVERITY EXAMPLE SUBSCRIPTION─────────────────────────────────────────────────────────warning Slack #finops-alerts Logged; visible; no page
critical Slack #finops-alerts + Teams channel Two places to see it
emergency Slack #finops-alerts + webhook to your pager Maximum signal Engineer woken up if neededMatch the subscriptions to the severity. Don’t page on warning; do page on emergency.
Quieting non-prod with subscriptions
The bands are fixed. Every org, team and dimension uses the same thresholds, and there is no setting to override a severity. What you control is who hears about each one:
Settings → Notifications: one subscription per dimension and severity
RESOURCE GROUP "dev" Subscribe: critical, emergency Skip: warning (dev spikes at warning level are routine)
RESOURCE GROUP "prod-payment" Subscribe: warning, critical, emergency Emergency also goes to the webhook that pages on-callNon-prod hears less; production hears everything. The anomaly itself is still detected and still visible in Cost Reports either way.
No timer escalation
ZopNight does not escalate on a timer. A critical anomaly that sits unresolved for a day does not become an emergency, and nobody is paged at 48 or 72 hours.
WHAT ACTUALLY HAPPENS:
Day 1 Critical anomaly fires; subscribers notifiedDay 1 A re-run the same day that lands it in a different severity notifies againDay 2-7 The same anomaly key is held back: no repeat alerts for 7 days after a notification
THE GAP THIS LEAVES: "We missed this critical alert; nobody followed up" The product will not chase it for you Follow-up is your process: an owner, a ticket, a review datePersistent anomalies stay at the severity they fired with. Your process is the safety net.
Subscription tuning best practices
DO NARROW A SUBSCRIPTION WHEN: The severity doesn't match business impact for that scope Non-prod workloads (critical and emergency are often enough) High-stakes prod resources (add the paging webhook to emergency) After repeated warning alerts nobody acts on
DON'T NARROW WHEN: You just want alerts to go away because they're "annoying" (find out why they fire first)
The current routing already matches your situation
You haven't observed enough alerts to knowNarrowing is a deliberate choice. Document the rationale.
Severity vs urgency interaction
SEVERITY = ZopNight's classification of magnitudeURGENCY = team's interpretation of business impact
EXAMPLE: Critical anomaly on a non-prod dev environment Severity: critical (technical classification) Urgency: medium (business impact lower)
Customer can: Subscribe non-prod to critical and emergency only Or: route non-prod warnings to a quieter channel
THE CHOICE IS YOURS Severity is suggestion; team interpretsThe numeric classification + team context = actual response.
Anti-pattern: blanket silencing
DON'T silence anomalies without investigation
SCENARIO: Team disabled all dev environment anomaly alerts "Too noisy"
PROBLEM: Lost visibility into dev workloads Bug spawned 50 instances in dev; not detected $5K wasted
BETTER: Investigate noisy patterns Narrow subscriptions per dimension and severity Keep critical and emergency, not blanket-silence Maintain visibility while reducing noiseSilencing is the lazy option. Narrowing subscriptions is the right option.
2. Demo
A team’s severity-based response:
EVENT 1 (warning, 75% deviation): Resource: prod-api group
Action: Investigated next day Found: new feature deployment drove modest cost increase Expected behavior Documented in change log Closed: no further action
EVENT 2 (critical, 280% deviation): Resource: ml-training group
Action: Investigated within 24h Found: stuck training job Killed orphaned instances (6 GPU instances) Cost recovered Postmortem documented Action items: Add job-watchdog Reduce default training timeout
EVENT 3 (emergency, 1,200% deviation): Resource: i-0compromised
Action: Page on-call at 03:00 UTC Engineer investigated within 5 min Detected: compromised IAM credentials provisioning GPU instances Killed instances; rotated credentials Security incident filed (separate process) Total cost: $4,200 (caught early) Without emergency severity: could have been $50K+Each severity matched its response. The routing worked.
3. Hands-on (5 min)
Audit your severity distribution:
□ STEP 1: Open Cost Reports for the last 30 days and count the anomaly markers on the daily chart, by severity: Warnings: _____ Critical: _____ Emergency: _____
□ STEP 2: Per-severity health Warnings resolved: ___% (target: 80%+ resolved) Critical resolved within 24h: ___% Emergency response time: avg ___ minutes
□ STEP 3: Identify miscategorized Any "noisy" warnings you could drop from a subscription? _____ Any critical that should be emergency? _____ Any non-prod patterns over-rated? _____
□ STEP 4: Plan tuning Subscription changes to make: __________ Process improvements: __________
□ STEP 5: Set quarterly review Date: __________ Owner: __________A 15-minute audit reveals routing changes worth making.
4. Knowledge check
Q1
An anomaly with 200% deviation. The severity:
A. warning
B. emergency
C. info, since it is under 500%
D. critical (100-500% range)
Show answer
Correct: D. 200% falls in the critical band (100% to under 500%): investigate within 24 hours.
Q2
A team wants fewer alerts from non-prod spikes without losing production warnings. What should they do?
A. Ask support to lower the severity bands for their org only
B. Switch off anomaly detection for the non-prod groups
C. Subscribe non-prod dimensions to critical and emergency only
D. Raise the warning threshold to 100% in anomaly settings
Show answer
Correct: C. Severity bands are fixed for every org and there is no threshold setting. Subscriptions per dimension and severity decide who hears what, so production keeps its warning subscription.
Q3
A critical anomaly fires at 03:00 UTC. The team’s response should:
A. Wait for morning
B. Investigate within 24 hours per the severity SLA
C. Page immediately
D. Escalate straight to the FinOps lead at once
Show answer
Correct: B. Production-related critical may warrant immediate action; non-prod can wait. 24-hour SLA is the contract. ZopNight will not escalate it on a timer, so the follow-up is yours.
5. Apply
Severity bands are fixed. Routing is set by your anomaly subscriptions in Settings → Notifications, one per dimension and severity.
For your team: quarterly severity audit. Narrow subscriptions deliberately, give every critical an owner, and document the rationale.