Slack / Teams / Google Chat notifications

Outcome

By the end of this lesson, you will be able to configure notification channels (Slack, Teams, Google Chat, webhooks) and route different event types to different channels.

1. Concept

Notifications go out when a schedule acts, when somebody sets or clears an override, when spending jumps, and when something fails.

Each one can go to as many places as you like. A place is a channel, and a channel is just a web address ZopNight posts to: Slack, Microsoft Teams, Google Chat, or anything else that accepts a message that way.

Setting up a channel

SETTINGS → Notifications → Channels & Alerts → add a channel
─────────────────────────────────────────────────────────
Name:           [team-finops-slack]
Type:           [ Slack ▾ ]    (or Teams / GChat / Webhook)
Webhook URL:    [https://hooks.slack.com/services/T.../B.../...]
Test:           [Send test notification]

[Cancel]                                              [Save channel]

The webhook URL is provided by the customer’s Slack / Teams / GChat workspace admin. The channel itself is held by ZopNight’s notification service; it is not kept in the cloud-credential vault from M1.1.

What notifications fire

EVENT                                   KEY
─────────────────────────────────────────────────────────────────
Resource started / stopped (scheduled)  resource.started, resource.stopped
Resource started / stopped (manual)     resource.started.manual, resource.stopped.manual
Group started / stopped                 resource.group.started.manual, resource.group.stopped.manual
Action failed                           resource.failed
Retries exhausted                       resource.retry_exhausted
Override set / cleared / expired        resource.override, resource.override.cleared, resource.override.expired
Remediation approval / done / failed    remediation.approval_required, remediation.completed, remediation.failed
Cost anomaly                            cost.anomaly.warning, cost.anomaly.critical, cost.anomaly.emergency
Budget warning / exceeded               budget.warning, budget.exceeded
Billing import failed                   billing.sync_degraded
Cloud permission newly denied           permission.denied

That is a selection; L4 walks through the full catalogue. There is no event for a discovery sync finishing or failing.

Severity is not a setting. Each event’s severity is fixed by whatever produces it, and for cost anomalies it is part of the key itself (.warning, .critical, .emergency). What you control is which keys each channel subscribes to (L5 goes deeper).

Subscriptions

A channel receives nothing until it has subscriptions. A subscription is three things:

SUBSCRIPTION = CHANNEL  ×  EVENT KEY  ×  SCOPE
─────────────────────────────────────────────────────────
Channel:    #ops-oncall
Event key:  resource.retry_exhausted
Scope:      org (every resource) | one resource | one resource group | one cloud account

One key, one channel, one scope. There is no severity filter, no tag-based scope, no exclude rule and no time-of-day option. To send a channel more, you add subscriptions; to send it less, you remove them.

Common patterns:

CHANNEL                          SUBSCRIBES TO
─────────────────────────────────────────────────────────
#finops-alerts                   resource.failed, resource.retry_exhausted,
                                 budget.warning, budget.exceeded,
                                 cost.anomaly.critical, cost.anomaly.emergency
#finops-info                     resource.started, resource.stopped
                                 (scheduled starts and stops: awareness only)
#ops-oncall                      resource.retry_exhausted, scoped to the
                                 production cloud account
webhook:datadog                  every key the receiving tool needs

In the console, open Settings → Notifications, pick a channel, and open its Channel Settings. Each event type has a toggle, which subscribes the channel org-wide. The cost anomaly section adds a severity slider per dimension (organization, cloud account, team, resource group, resource), and its entity picker scopes the subscription to the accounts, groups or resources you choose.

Which scopes an event honours depends on the event. Scheduling events such as resource.failed can be scoped to one resource or one cloud account; the group-level scope is used by cost alerts such as the resource-group anomaly keys. A scoped scheduling subscription is created through the API or the MCP tool create_notification_subscription, since the console toggles are org-wide.

One event, many channels

A single event goes to every channel with a matching subscription. If #finops-alerts subscribes to resource.retry_exhausted org-wide and #ops-oncall subscribes to the same key scoped to the production account, a production failure lands in both, and a staging failure lands only in #finops-alerts. The team chooses how much overlap to accept.

What a notification looks like

SLACK MESSAGE EXAMPLE
─────────────────────────────────────────────────────────
[CRITICAL] Schedule action failed: dev-platform-eu

Resource:    dev-cluster-1 (eks)
Action:      Stop
Schedule:    business-hours-eu (firing at 20:00 ET)
Error:       InsufficientPermissions: missing eks:UpdateNodegroupConfig
Retries:     3 of 3 (exhausted)
Detected:    2 minutes ago

[View details →]   [View schedule →]   [Diagnose →]

The notification carries:

• Severity (color-coded: red for critical, yellow for warning, gray for info)
• Resource and group context
• Action attempted and outcome
• Error message (for failures)
• Retry status (for retries)
• Detected timestamp
• Action links for follow-up

The notification is actionable: the engineer can click “View details” to get to the resource without searching.

Avoiding alert fatigue

Three principles:

1. Separate routine keys from actionable ones. Don’t put every key in one channel. Scheduled starts and stops (resource.started, resource.stopped) are useful for awareness but noisy next to failures. Two channels: one subscribed to the routine keys, one to the failure, budget and anomaly keys.

2. Scope tightly. A team’s channel should not receive notifications about other teams’ resources. Scope its subscriptions to that team’s cloud account or resources.

3. Unsubscribe from noise. If a key fires too often for a channel, turn that key off for that channel. You cannot change an event’s severity, but you can choose which keys each channel hears (see L5).

Most teams start with too many keys and remove some as their channels mature.

What ZopNight does NOT notify

A few deliberate non-notifications:

• Discovery completions (too frequent: every 6 hours per account)
• Cost trends (these go on the dashboard, not in notifications)
• Resource creation/deletion (these are CMDB-style events; subscribe via audit log if needed)
• Audit log activity (the audit log is the full record; notifications would be too noisy)

A custom webhook can pull these if the team has a use case.

2. Demo

A team’s notification setup, day 1 through month 1:

DAY 1 (initial setup):
  Channel #1: #finops-alerts (Slack)
  Subscribed to every Scheduling key
  Test notification fires. Engineer confirms it lands.

WEEK 1:
  Team gets ~30 notifications per day. Most are scheduled starts and stops.
  Alert fatigue setting in. People start muting the channel.

WEEK 2 (tuning):
  Setup change:
    Channel #1: #finops-alerts (failure, budget and anomaly keys): actionable
    Channel #2: #finops-info (resource.started, resource.stopped): opt-in awareness

  Notification count in #finops-alerts: drops to ~3 per day.

WEEK 3 (team-specific):
  Setup change:
    Add Channel #3: #dev-platform (resource.failed scoped to the dev
                    cloud account, plus resource-group anomaly keys for
                    dev-platform-eu and dev-platform-aws)
    Add Channel #4: #ops-oncall (resource.retry_exhausted scoped to the
                    production cloud account, cost.anomaly.emergency)

  Dev-platform team owns their channel. Ops-oncall is dedicated to prod failures.

MONTH 1:
  Three actionable channels. Each gets ~3-5 notifications per day.
  Engineers actually read the channels.
  Alert fatigue resolved.

The lesson: the first set of subscriptions is a starting point. The first month of operation tunes it for the org’s actual signal/noise tolerance.

3. Hands-on (6 min)

If you have admin access:

1. Set up a notification channel (use a sandbox Slack channel, not your
   real ops channels).
2. Add the webhook URL.
3. Test the notification.
4. Subscribe the channel to resource.stopped.manual (manual actions
   have their own key), then manually stop a sandbox resource.
5. Confirm the notification lands in the test channel.
6. In the channel's Channel Settings, turn off the keys you do not
   want this channel to hear, and leave on resource.stopped.manual.
7. Test again: only the subscribed keys should arrive.

Do it through MCP. The same task you just did in the console, asked in one sentence.

BEFORE  A ZopNight account with one cloud connected. A Slack, Teams or Google Chat workspace you can post into.
ASK     "Set up a Slack channel for schedule failures and send it a test message."
CHECK   the test message arrived before you rely on it. An untested channel is not a channel.

Tools behind it: create_notification_channel (write, tier 2, reversible), test_notification_channel (write, tier 2, reversible), list_notification_subscriptions (read, Govern). Connection details are in the MCP integration docs.

4. Knowledge check

Q1

A team’s #finops-alerts channel is receiving ~30 notifications per day, mostly successful scheduled actions. The team has started muting it. The right fix:

A. Split it: #finops-alerts keeps failure and budget keys; #finops-info gets start and stop keys
B. Disable notifications entirely
C. Reduce how often the schedules fire, so that there are fewer events to send overall
D. Disable the schedule

Q2

A team wants production-CRITICAL events to go to BOTH #finops-alerts (org-wide visibility) AND #ops-oncall (paging). The setup:

A. Pick one channel
B. Forward the messages from one channel through to the other one automatically on every single event that ever fires
C. Set up a relay channel
D. Subscribe both channels to the failure keys for production. One event goes to every channel with a matching subscription

Q3

A team wants a Slack message every time a discovery sync finishes. What should they know?

A. There is no discovery-sync event to subscribe to; the Inventory page shows when each sync last ran
B. Subscribe to resource.started, which is the event that fires each time a discovery sync completes
C. Subscribe to permission.denied, which fires once on every discovery cycle for every account
D. Ask for discovery to run less often first, so that the completion event fires fewer times a day

5. Apply

Notifications are configured in:

• Settings → Notifications: manage channels and, in each channel’s Channel Settings, its subscriptions
• MCP: create_notification_subscription to scope a subscription to one resource or cloud account

For the full list of events you can subscribe to, continue to L4.

Related lessons

• L4: The subscribable event catalogue (next)
• L5: Routing with subscriptions
• L6: Where audit logs live

Glossary terms touched

Notification channel · Alert fatigue