The first 15 minutes · ZopDev University

Outcome

By the end of this lesson, you will be able to triage a cost anomaly within 15 minutes, distinguish events from incidents using the severity-times-expectedness matrix, and decide whether to escalate, document, or watch.


Prerequisites	T1 (operator surface) · M5.1-M5.2 (maturity context)
Time	9 minutes

1. Concept

The first fifteen minutes decide what kind of thing you are dealing with.

An event is recorded and closed. An incident needs somebody to respond. The same alert can be either, and which one it turns out to be depends entirely on what you find when you look.

Doing those fifteen minutes properly is the difference between closing it with a note and finding out three weeks later that it cost $14,000.

MIN 0-1     IDENTIFY the anomaly
            Read the alert (severity, scope, magnitude)
            Confirm it actually fired (not a duplicate)
            Acknowledge to the team channel

MIN 1-3     DRILL into the right dimension
            Start org-level: which provider / account
            Cascade to team / resource
            Use the Sankey to find the dominant flow

MIN 3-8     IDENTIFY the resource(s) driving the anomaly
            From the drill, pinpoint specific resources
            Check the state history for those resources
            Cross-reference with the audit log for recent changes

MIN 8-12    DETERMINE if action is needed
            Is this expected (marketing campaign, scheduled launch)?
            Is this unexpected (autoscaler, compromised cred)?
            What's the dollar impact (your team's tiers)?
            Who's the owner?

MIN 12-15   DECIDE
            Investigation continues (low severity, document)
            Escalate (critical or emergency, page on-call)
            Close as expected (document; not an incident)

The 15-minute target is for triage, not resolution. Resolution may take hours or days; triage answers “should anyone act on this right now, and at what severity.”

Triage decision matrix

The two axes are severity (how far above baseline: ZopNight’s daily 20:55 UTC run grades the day’s percentage deviation from the 7-day average, +30% as warning, +100% as critical, over +500% as emergency) and expectedness (was this on the calendar):

                            EXPECTED                UNEXPECTED
─────────────────────────────────────────────────────────────────
WARNING                    Note + watch             Investigate +
(+30% above baseline)      Document in triage log   document this week

CRITICAL                   Note + bounded watch     Investigate now +
(+100%)                    Confirm scope            escalate if compounding

EMERGENCY                  Treat as critical        PAGE on-call
(+500%)                    (even expected is        Incident commander
                           large)                   Cut affected resources

The “expected” qualifier is what often turns a critical-looking anomaly into a benign note. Confirming with the affected team takes 30 seconds and saves hours of false-incident response.

The “expected” qualifier: confirming fast

Many anomalies are expected. They look anomalous to the algorithm because they deviate from baseline, but the team that owns the workload knows about them. Examples:

EXPECTED PATTERNS:
  - Marketing campaign (CDN spike, Lambda burst)
  - Product launch (compute scale-up planned in advance)
  - Quarterly data ETL (large but predictable batch)
  - Scheduled ML training (GPU spike)
  - Black Friday / peak day traffic
  - Migration in progress (temporary doubling)

The fast confirmation pattern:

ON-CALL HAS 30 SECONDS:
  Slack the affected team's #ops or #finops channel:
    "Anomaly on prod-aws-us-east-1, +$2K above baseline.
    Is this expected? Recent launch / campaign / migration?"

  If yes within 5 min: close as expected; document.
  If no answer in 5 min: assume unexpected; proceed to investigation.

The 5-minute timeout matters. Waiting for confirmation forever lets the anomaly grow if it’s actually unexpected.

Drilling efficiently

The most-common time sink is drilling in the wrong order. Cost anomalies have a natural hierarchy:

DRILL ORDER (top-down):
  1. Provider (AWS / GCP / Azure)
  2. Account / project / subscription
  3. Service (EC2, RDS, K8s, etc.)
  4. Team (via tag)
  5. Resource (specific instance / cluster / bucket)
  6. State change (what changed at the inflection point)

Skipping levels wastes time. Drilling provider-first usually narrows the search space by 60-80% before you look at any specific resource.

Severity vs dollar impact

ZopNight's severity is a percentage, not dollars. The daily
20:55 UTC run compares each day's cost with its 7-day average
(a z-score check can raise the grade):

  warning     +30% to +100%
  critical    +100% to +500%
  emergency   over +500%

Dollar impact is a separate read: a +40% warning on a big
account can cost more than an emergency on a small one. Set
your own dollar tiers (e.g. under $1K, $1K-$10K, over $10K)
and use them with severity to decide who acts and how fast.

Common 15-minute traps

TRAP                                  AVOID BY
──────────────────────────────────────────────────────────────────
"Let me investigate more before        Set a 15-min hard stop;
acting"                                escalate or close at the timer

"I'll page if it gets worse"           Define worse: dollar threshold,
                                       not vibes

"This is probably the marketing        Ask, don't assume; takes 30 sec
campaign"                              and saves hours of false ops

"Let me wait for more data"            ZopNight has the data; the data
                                       you need is in Cost Flow +
                                       audit log right now

"I'll handle it after this meeting"    A 15-min triage IS the meeting
                                       priority for critical anomalies

Documentation as triage output

Every anomaly needs a written triage record. ZopNight’s audit log records API changes, not anomaly triage, so keep the record in your triage log or incident channel. Even if you close as “expected, no action,” that entry is the record:

GOOD TRIAGE RECORD:
  Anomaly: prod-aws-us-east-1 cost +$2,400 above baseline
  Triage: 11 minutes
  Cause: marketing campaign Q2-launch (confirmed with @marketing)
  Action: close as expected
  Documented by: jane@finops at 2026-05-15 03:47 UTC

BAD TRIAGE RECORD:
  "Resolved."

Specific entries compound into pattern recognition over months.

How ZopNight supports 15-minute triage

SURFACE                          USE IN TRIAGE
──────────────────────────────────────────────────────────────────
Anomaly drawer                    Shows the anomaly + root-cause hint
Cost Flow Sankey                  Drill from provider to resource fast
State history per resource         What changed when
Audit log filter                   Recent mutations affecting the spike
Team tag                          Owner identification
Notification routing              Posts each severity to its subscribed channel

The same flow works in cloud-native tools (AWS Cost Anomaly Detection, Azure cost alerts, GCP budget alerts plus Cost Explorer or Cost Management for the drill). Whichever you use, the on-call’s job is to follow the drill order.

2. Demo

A real critical anomaly from the daily detection run (20:55 UTC):

T+0:00     CRITICAL ANOMALY fires (daily run, 20:55 UTC)
            +200% deviation on prod-aws-us-east-1 for the day
            Webhook alert pages on-call

T+0:01     On-call ACK; opens the anomaly detail in ZopNight

T+0:02     Anomaly summary:
              day's cost ~$3,600 vs ~$1,200 baseline
              dimension: prod-aws-us-east-1
              dollar impact: ~$2,400 above baseline

T+0:03     DRILL into Cost Flow filtered to prod-aws-us-east-1
            Sankey shows EKS dominant

T+0:05     Drill into EKS: team=ml dominates

T+0:06     Drill into team=ml: cluster prod-ml-eks-cluster

T+0:07     Check state history for the cluster:
              new resource provisioned at 02:55 UTC
              HPA max increased from 12 to 30 at 02:54 UTC

T+0:08     CONFIRMATION FAST PATH:
              Slack #ml-team: "Anomaly on prod-ml-eks-cluster,
              +200% spike at 02:55. Planned ML batch?"

T+0:10     @ml-lead responds:
              "Yes: overnight ML batch (training new model).
              HPA increase was deliberate. Ran 4 hours, now done."

T+0:11     DECISION: expected event; close as anomaly-expected
              Triage record written with @ml-lead's confirmation.

T+0:14     Triage COMPLETE.

ELAPSED:   14 minutes from page to close.
OUTCOME:   No incident. Documented expected event.

Without the 15-minute discipline, the on-call might have investigated for an hour, paged the FinOps lead, and discovered the same thing an hour later.

3. Hands-on (5 min)

For a past anomaly your team triggered:

ANOMALY (what / when):
  __________________________________________________________

TIMELINE:
  T+0       Anomaly fired:        __________
  T+_____   First responder ack:   __________
  T+_____   Drill completed:       __________
  T+_____   Cause identified:      __________
  T+_____   Decision made:         __________

WOULD 15-MINUTE TRIAGE HAVE HELPED?
  □ Faster identification
  □ Faster close-as-expected
  □ Same outcome, less stress
  □ Wouldn't have helped (this case)

KEY GAP in your team's current practice:
  __________________________________________________________

ACTION (what to set up):
  □ Define the 15-min hard stop
  □ Document confirmation channels per team
  □ Update notification routing
  □ Build the triage-record template

If your team doesn’t have a triage SLA today, set 15 minutes as the target. Adjust based on operational reality after the first month.

Do it through MCP. The same task you just did in the console, asked in one sentence.

BEFORE  A ZopNight account with one cloud connected. At least one anomaly open, which usually means 30 days of history.
ASK     "What anomalies are open, and close the one already explained by the release."
CHECK   that you closed it with a reason. An anomaly closed silently teaches the next responder nothing.

Tools behind it: get_anomalies (read, Cost), list_alerts (read, Govern), resolve_alert (write, tier 2, reversible). Connection details are in the MCP integration docs.

4. Knowledge check

Q1

An emergency anomaly arrives from the daily 20:55 UTC run. First action by the on-call:

A. Wait until morning
B. Ignore until business hours
C. Acknowledge the alert promptly; begin the drill (Cost Flow → resource → state history) right away; engage on-call escalation if the pattern continues to grow
D. Send it to the FinOps lead’s queue and wait for them to pick it up in the morning, since nobody else has the authority to act on it at that hour of the night

Show answer

Correct: C. Emergency severity means action now, not after coffee.

Q2

A critical anomaly turns out to be a marketing campaign:

A. Treat as a full incident anyway
B. Document as expected; close
C. Reclassify it as a security incident
D. Escalate to leadership

Show answer

Correct: B. Investigate only if the spike compounds on top of expected (e.g., expected $5K + unexpected $8K = investigate the $8K). The fast confirmation pattern (Slack the affected team) takes 30 seconds and saves hours of false-incident response.

Q3

The 15-minute target is for:

A. Closing out every single anomaly completely before moving on to the next
B. Filing the postmortem
C. Cosmetic SLA
D. Triage: determining whether action is needed and at what level

Show answer

Correct: D. Resolution (fixing the underlying issue, applying a remediation) may take hours or days; the 15-min target is for the decision, not the fix.

5. Apply

Build your team’s 15-minute triage runbook. Document the drill order, the confirmation channels, your dollar-impact tiers. Test it monthly via tabletop exercises (simulated anomaly).

ZopNight’s Anomaly drawer + Cost Flow + audit log are the canonical surfaces for the drill. Bookmark them.

Glossary terms touched

Anomaly

Outcome

1. Concept

Triage decision matrix

The “expected” qualifier: confirming fast

Drilling efficiently

Severity vs dollar impact

Common 15-minute traps

Documentation as triage output

How ZopNight supports 15-minute triage

2. Demo

3. Hands-on (5 min)

4. Knowledge check

Q1

Q2

Q3

5. Apply

Related lessons

Glossary terms touched

Start with the bill.