Routing with subscriptions · ZopDev University

Outcome

By the end of this lesson, you will be able to route each event to the right channel with subscriptions (channel × event key × scope) and keep each channel’s noise level right for its audience.


Prerequisites	L3, L4
Time	9 minutes

1. Concept

Routing in ZopNight is one idea applied many times: a subscription.

SUBSCRIPTION = CHANNEL  ×  EVENT KEY  ×  SCOPE
─────────────────────────────────────────────────────────────────────
CHANNEL     Where it goes: a Slack, Teams or Google Chat channel,
            or a custom webhook
EVENT KEY   What it is: resource.retry_exhausted, budget.exceeded,
            cost.anomaly.emergency, permission.denied, ...
SCOPE       Which part of the estate:
              org             every matching event
              resource        events about one resource
              resource_group  events about one resource group
              cloud_account   events about one cloud account

An event goes to every channel that has an enabled subscription for its key at a matching scope. An org-wide subscription matches everything; a scoped one matches only its resource, group or account. If no subscription matches, the event is still recorded (in history, on the anomalies report, on the budget) but nobody is pinged.

That is the whole routing model. There is no routing-rule page, no severity filter, no tag-based scope, no exclude list and no time-of-day or message-format option. Every routing decision is “add this subscription” or “remove that one”.

Where severity fits

ZopNight events do carry a severity (CRITICAL, WARNING, INFO), and the message shows it. But severity is fixed by whatever produces the event. You cannot change it in settings.

For cost anomalies, the severity is built into the key:

KEY                       FIRES WHEN
─────────────────────────────────────────────────────────────
cost.anomaly.warning      spend is moderately above its baseline
cost.anomaly.critical     spend is well above its baseline
cost.anomaly.emergency    spend is more than 500% above its baseline

The same three suffixes exist per dimension: cost.anomaly.cloud_account.*, cost.anomaly.resource_group.*, cost.anomaly.resource.* and cost.anomaly.team.*. So “send only emergencies to on-call” is not a severity setting. It is a subscription to the .emergency key and nothing else.

Which scopes an event honours

Not every event can be narrowed by every scope. Scheduling events (resource.failed, resource.retry_exhausted, the start and stop keys) are matched on the resource and its cloud account, so scope them to one resource or one cloud account. Group-level routing is for cost alerts: the cost.anomaly.resource_group.* keys, scoped to the group you care about. Account-level alerts such as permission.denied match on the cloud account.

If you want “production failures go to on-call”, the dependable way is a production cloud account and a subscription scoped to it.

Designing channels by audience

Different audiences tolerate different noise. You serve each by choosing its keys and scopes:

CHANNEL            SUBSCRIPTIONS                                    AUDIENCE
─────────────────────────────────────────────────────────────────────────────
#finops-info       resource.started, resource.stopped (org)         opt-in awareness
#finops-alerts     resource.failed, resource.retry_exhausted,       finops + platform
                   budget.warning, budget.exceeded,
                   cost.anomaly.warning, cost.anomaly.critical (org)
#ops-oncall        resource.retry_exhausted (scope: prod account),  paging
                   cost.anomaly.emergency (org)
#dev-platform      cost.anomaly.resource_group.* (scope: the        dev-platform team
                   dev-platform groups), resource.failed
                   (scope: dev account)
#security          permission.denied (org)                          security / compliance

Overlap is a choice. A production retry-exhausted event lands in both #finops-alerts (org-wide) and #ops-oncall (prod account). If that is too much, remove one.

Where you set it

In the console, open Settings → Notifications, pick a channel and open its Channel Settings:

Event toggles. Each event type has a toggle. Turning it on subscribes the channel to that key org-wide.
Cost anomaly section. One severity slider per dimension (organization, cloud account, team, resource group, resource). The slider is cumulative: Warning subscribes .warning; Critical adds .critical; Emergency adds .emergency, so the rightmost step subscribes all three. The dimension’s entity picker scopes those subscriptions to the accounts, teams or groups you select.
Test Notification. Sends a test message to the channel.

Two things the console does not do: scope a scheduling event to one resource or account, and subscribe to .emergency without .warning and .critical. For both, create the exact subscription through the API or the MCP tool create_notification_subscription (it takes the channel, the event key, the scope type and the scope id).

Feeding a SIEM

For compliance, add a Custom Webhook channel pointing at the SIEM’s ingest endpoint, and subscribe it to the keys the SIEM needs (for example permission.denied, resource.retry_exhausted, remediation.completed, remediation.failed). The SIEM applies its own correlation rules; ZopNight is the data source. For a full record of who changed what, pull the audit log too (L6); notifications are not a copy of it.

2. Demo

A team’s day-30 review of its notification setup:

T+0       FinOps lead opens Settings → Notifications and goes through the
          channels one by one.

T+1 min   Finds three problems:
            - #finops-alerts gets ~45 messages a day. It is subscribed to
              resource.started and resource.stopped org-wide, so every
              scheduled start and stop lands there.
            - #ops-oncall was paged at 3 AM for a staging anomaly. Its cost
              anomaly slider is on Emergency, which subscribes all three
              anomaly keys, warning included.
            - #security never hears about permission problems: nobody
              subscribed it to permission.denied.

T+4 min   Fixes them:
            #finops-alerts: turns off resource.started and resource.stopped.
            New channel #finops-info: turns those two keys on there.
            #ops-oncall: turns the anomaly slider off, then asks the MCP
              assistant to subscribe #ops-oncall to cost.anomaly.emergency
              at org scope, and to resource.retry_exhausted scoped to the
              production cloud account.
            #security: turns on permission.denied.

T+6 min   Sends a Test Notification to each changed channel.

T+30 days #finops-alerts: 5-8 actionable messages a day. People read it.
          #ops-oncall: 1-2 a week, all production or emergency.
          #security: hears about each newly denied permission once.

Nothing about any event’s severity changed. Only the subscriptions did.

3. Hands-on (5 min)

For your notification channels (or a sandbox):

1. Open Settings → Notifications.
2. For each channel, open Channel Settings and list the keys it is
   subscribed to, and at what scope.
3. Pick the noisiest channel. For each key it hears, ask: does this
   audience act on it?
   - If yes, keep it
   - If no, turn it off here, and turn it on in an awareness channel
     if someone still wants it
4. Check the cost anomaly slider on any paging channel. If it is on
   Warning or above, it hears warnings too. Decide whether that is
   what you want.
5. If a channel should only hear one account, list the subscriptions
   you would scope to that account (channel, key, cloud_account, id).
   Create them with the MCP tool create_notification_subscription.
6. Send a Test Notification to each channel you changed, then watch
   the volume over the next week.

4. Knowledge check

Q1

A team’s #finops-alerts channel receives ~50 messages a day, mostly scheduled starts and stops. The team has stopped reading it. The right fix:

A. Lower the severity of scheduled starts and stops to INFO in the organization’s settings page
B. Turn off the start and stop keys on #finops-alerts and subscribe an opt-in channel to them
C. Delete the channel and create it again with the same webhook
D. Add an exclude rule to #finops-alerts that filters every INFO-level event out of the channel

Show answer

Correct: B. A channel hears exactly the keys it subscribes to, so the fix is to change its subscriptions. There is no severity setting and no exclude rule, and recreating the channel changes nothing.

Q2

The on-call channel should be paged only for cost anomaly emergencies, never warnings. Its anomaly slider is set to Emergency, yet it still gets warnings. Why, and what is the fix?

A. The slider is cumulative, so Emergency also subscribes .warning; subscribe only cost.anomaly.emergency via the API or MCP
B. Emergency is the lowest step on the slider, so moving it to Warning subscribes the channel to emergencies only
C. Anomaly severity is set per organization, so lower the warning threshold in the settings
D. Warnings are always sent to every channel, so mute the channel in Slack overnight instead

Show answer

Correct: A. Each slider step adds a key: Emergency means warning, critical and emergency. A channel that should hear emergencies alone needs that single subscription, which you create through the API or the create_notification_subscription MCP tool.

Q3

A compliance team wants permission and remediation events in their SIEM for correlation. The right pattern:

A. Forward every message from the compliance team’s own Slack channel into the SIEM by hand
B. CC the compliance team on every email notification that ZopNight sends
C. Add a Custom Webhook channel at the SIEM’s ingest endpoint and subscribe it to those keys
D. Export the notification list to CSV once a month and upload it to the SIEM

Show answer

Correct: C. A webhook channel is a subscriber like any other. Subscribe it to the keys the SIEM needs, and the SIEM applies its own rules. Manual forwarding and monthly uploads lose events and timing.

5. Apply

Routing lives in:

Settings → Notifications: channels, event toggles, the cost anomaly sliders and entity pickers, and Test Notification
MCP: list_notification_event_types lists every key; create_notification_subscription and bulk_update_subscriptions create exact, scoped subscriptions

For the audit-log foundation that a SIEM also needs, continue to L6.

L6: Where audit logs live (next: closes the module)
L4: The subscribable event catalogue
T4.M4.3: Audit logging

Glossary terms touched

Severity · Alert fatigue

Outcome

1. Concept

Where severity fits

Which scopes an event honours

Designing channels by audience

Where you set it

Feeding a SIEM

2. Demo

3. Hands-on (5 min)

4. Knowledge check

Q1

Q2

Q3

5. Apply

Related lessons

Glossary terms touched

Start with the bill.