Outcome
By the end of this lesson, you will be able to assign the right severity to each event class and design a routing strategy that delivers the right signal without alert fatigue.
| Tier | Operator |
| JTBD | ”Stop my Slack from being unreadable.” |
| Personas | Platform Engineer · FinOps Analyst · Security/Compliance |
| Prerequisites | L3 |
| Time | 10 minutes |
| Bloom verb | Assign (Apply) and Design (Create) |
1. Concept
Severity is the lever for routing notifications correctly. Three levels, INFO, WARNING, CRITICAL, drive what channel an event goes to, whether it pages someone, and how often the team should expect to see it.
The three severity levels
LEVEL MEANING EXAMPLES─────────────────────────────────────────────────────────────────────────INFO Something happened. No action needed. Schedule fired Override applied Override expired Auto-rem succeeded Resource discovered─────────────────────────────────────────────────────────────────────────WARNING Something happened that may need attention Anomaly detected within hours. Investigate. Budget threshold crossed Permission Denied Auto-rem failed Discovery degraded─────────────────────────────────────────────────────────────────────────CRITICAL Action needed now. Service impact, audit Schedule action failed evidence, or significant cost event. after 3 retries Cost anomaly emergency (>500% deviation) Cost > 2× monthly budget Connection / cred failThe severity is configurable per event class in the org settings. Defaults are conservative; tune to your org’s tolerance.
Why severity matters for routing
Different audiences tolerate different signal/noise ratios:
AUDIENCE TOLERATES OPTIMAL CADENCE────────────────────────────────────────────────────────────Engineer's own team INFO frequency Many per day, fineOn-call rotation CRITICAL only Few per week, ideallyOrg-wide #finops-alerts WARNING+ Few per dayAudit / compliance channel CRITICAL only Few per weekPersonal DMs (paging) CRITICAL only Rare, paging-worthyA routing strategy aligns severity to audience. Engineers see their team’s INFO; on-call sees only CRITICAL; auditors see CRITICAL with optional WARNING for context.
Designing a routing strategy
For a typical 50-engineer SaaS org:
CHANNEL SUBSCRIBES TO FREQUENCY─────────────────────────────────────────────────────────────────────────#finops-info INFO only, all scopes 100+/day (engineers opt-in for awareness)#finops-alerts WARNING + CRITICAL, all scopes 5-15/day (everyone in finops/platform)#dev-platform Events on dev-platform group varies (dev-platform team only)#staging-services Events on staging-services group varies (services team only)#prod-shared Events on prod-shared group rare (production-aware engineers)#ops-oncall CRITICAL only, production scope 1-2/week (paging-worthy)#compliance CRITICAL on production <5/month + auth/audit events (compliance / security)Each channel has a clear audience and tolerance. Notifications aren’t duplicated across channels unless multiple audiences need awareness.
The severity-tune-down move
Most orgs need to tune severity DOWN, not up, as they mature.
DEFAULT AFTER 30 DAYS OF OPERATION─────────────────────────────────────────────────────────────────Cost anomaly: WARNING Cost anomaly: INFO (visible but don't page)Anomaly emergency: CRITICAL Anomaly emergency: CRITICAL (keep)Budget threshold: WARNING Budget threshold: WARNING (keep)Schedule action failed after CRITICAL Schedule action CRITICAL (keep) retries failed after retriesDiscovery degraded: WARNING Discovery degraded: WARNING (keep)Auto-rem failed: WARNING Auto-rem failed: INFO (don't page, just log)Severity tuning is iterative. A team that sees too many WARNINGs and starts ignoring them is paying alert fatigue cost. The fix is to downgrade events that are informational, leaving only the genuinely actionable ones as WARNING+.
When to escalate severity
The opposite move, upgrade severity, is rarer but legitimate:
SCENARIO: A team's production has cost-anomaly emergencies (>500% deviation)two-three times per quarter. The current routing has them as CRITICAL goingto #ops-oncall. The on-call engineer is sometimes asleep.
UPGRADE: Add a "Cost anomaly emergency on production" routing rule that alsofires a PagerDuty notification (via webhook), waking up the on-call engineer.
The event severity stays CRITICAL. The routing changes (additional destinationon top of Slack).Severity itself is the event’s classification. Routing is what each channel does with each severity. The two are independent.
Per-channel filtering
Each channel can filter by severity, scope, event type, and time-of-day:
ROUTING RULE FOR #ops-oncall─────────────────────────────────────────────────────────Event types: Schedule action failed, Cost anomaly criticalSeverity: CRITICAL onlyScope: Production resources (tag env=prod) OR group prod-*Time-of-day: Any (24/7)Format: Compact (one-line summary, single emoji)ROUTING RULE FOR #compliance─────────────────────────────────────────────────────────Event types: Auth events, Permission changes, Audit-relevantSeverity: CRITICAL onlyScope: All (org-wide)Time-of-day: AnyFormat: Verbose (full event detail for audit evidence)Two channels, two different filtering strategies, each serves its audience.
The audit-evidence routing
For compliance use cases, a specific routing pattern delivers events to a SIEM or audit channel without depending on humans to read them:
ROUTING RULE FOR webhook:siem─────────────────────────────────────────────────────────Event types: AllSeverity: AnyScope: AllTime-of-day: AnyFormat: JSON (structured for ingestion)
DESTINATION: https://api.siem-tool.com/zopnight/eventsThe webhook receives every event, structured. SIEM ingests, correlates, alerts on its own rules. The notification channel is the data plane; the SIEM is the alerting plane. This decoupling is the right pattern for compliance-driven orgs.
2. Demo
A team’s day-30 retro on notification routing:
T+0 FinOps lead opens Settings → Notification Routing.
T+30 sec Reviews: 5 active channels, 18 routing rules.
T+1 min Identifies issues: - #finops-alerts is at 47 notifications/day, mostly INFO - #ops-oncall got paged for a non-prod anomaly (severity wrong) - #compliance is missing some audit-relevant events
T+5 min Adjusts: #finops-alerts: filter to WARNING+ only Move INFO events to #finops-info (new channel) Anomaly severity: downgrade non-prod anomalies to INFO Add routing for auth events to #compliance
T+10 min Saves. Tests with a manual trigger to verify routing.
T+30 days After 30 more days: #finops-alerts: 5-8 actionable notifications per day Engineers actually read it #ops-oncall: 1-2 per week, all genuine paging #compliance: ~3 per week, all audit-relevant Alert fatigue: resolvedTen minutes of tuning, 30 days of better-than-default routing. The cost of the tuning is small; the cost of NOT tuning is alert fatigue and missed signal.
3. Hands-on (6 min)
For your existing notification channels (or sandbox):
1. Open Settings → Notification Routing.2. Count current channels and rules.3. For each channel, note the per-day notification count (Settings has the analytics, or check the channel directly).4. Identify the noisiest channel.5. Ask: is everything in this channel actually actionable? - If yes, leave it - If no, identify which event class is noisy6. For the noisy event class: - Downgrade severity in the org settings (if too high) - OR move it to a different channel (if it's informational but valuable to track)7. Save and observe over the next week.4. Knowledge check
Q1
A team’s #finops-alerts channel receives ~50 notifications per day, mostly successful scheduled actions (INFO). The team has stopped reading it. The right fix:
A. Disable notifications
B. Downgrade routing: #finops-alerts subscribes to WARNING+ only. Move INFO events to a separate #finops-info channel (opt-in). The actionable channel gets 5-15/day; the informational channel gets the rest.
C. Increase the channel
D. Set up an automation to summarize
Show answer
Correct: B. Severity-based routing solves alert fatigue. INFO does not belong with WARNING/CRITICAL in an actionable channel.
Q2
A non-prod cost anomaly fires as CRITICAL and pages the on-call engineer at 3 AM. The fix is:
A. Disable cost anomaly notifications
B. Downgrade non-prod cost anomalies to WARNING in org settings. Route WARNING to #finops-alerts (visible, not paging). Reserve CRITICAL for production anomalies that genuinely warrant a 3 AM page.
C. Page during business hours only
D. Tell the engineer to mute
Show answer
Correct: B. Severity should reflect business impact. Non-prod anomalies are visible but rarely page-worthy.
Q3
A compliance team needs every CRITICAL event sent to their SIEM for correlation. The right pattern:
A. Send to compliance team’s Slack channel
B. Add a webhook channel pointing at the SIEM’s ingest endpoint, subscribed to CRITICAL across all scopes. The SIEM ingests structured data and applies its own correlation rules. This decouples the data plane (ZopNight notifications) from the alerting plane (SIEM).
C. CC the compliance team on every email
D. Export to CSV manually
Show answer
Correct: B. Webhook to SIEM is the right pattern for compliance-driven alerting. Decoupled, scalable, and respects the SIEM as the canonical alerting tool.
5. Apply
Severity and routing:
- Settings → Event Severity: assign severity per event class
- Settings → Routing Rules: define per-channel filters
- Notification analytics: per-channel volume
For the audit-log foundation that feeds the SIEM pattern, continue to L5.
Related lessons
- L5: Where audit logs live (next: closes the track)
- T3.M3.3: Audit logging
Glossary terms touched
Severity · Routing rule · Alert fatigue · SIEM · Time-of-day filter