Outcome
By the end of this lesson, you will be able to read the root-cause note on an anomaly, verify each cause it names, and investigate further when the note falls back to a generic message.
| Prerequisites | M2.10.L1 - L3 |
| Time | 9 minutes |
1. Concept
Finding a spike is the easy half. ZopNight also writes a short note on what changed.
That note turns the first twenty minutes of an investigation, the part where you work out which direction to look in, into a few lines you read.
ANOMALY: org-level +200% yesterday
Summary: 6 resources had a cost increase. Primary reason: 4 new resources were deployed (+$1,850/day).Insights: • New resources added in prod-aws-us-east-1 • 1 resource ran longer than usualDetails: 1. experiment-cluster-node-1: $0 to $410/day New resource: first tracked 1 day ago ...The note is deterministic. It is built from a fixed set of checks run in the same order every time. There are no scores, no rankings and no probabilities: a cause is listed because its check matched, and it is left out because its check did not.
The checks, in order
For each resource that drove the spike, ZopNight compares the anomaly day with the previous 14 days of that resource’s cost records and runs these checks:
1. RATE INCREASE (resize or repricing) Matches: hourly rate is more than 50% above its recent average Note names the user if the cloud activity log shows a resize or modify call near that day; otherwise it says "pricing updated"
2. USAGE OR SCHEDULE CHANGE Matches: running hours are more than 50% above the usual With a ZopNight schedule attached: "Scheduled shutdown may have failed" With no schedule: "No schedule configured" Plain "Usage increased": when the schedule lookup is unavailable or truncated (so ZopNight cannot be sure), and always on resource-group anomalies
3. RESERVATION OR COMMITMENT EXPIRY Matches: pricing moved from Reserved, Savings Plan or Committed Use Discount to On-Demand
4. NEW RESOURCE Matches: first cost record is no more than 3 days old
5. COST NEWLY APPEARING Matches: at least 3 days in a row at $0, then cost today (a resource restarted after a long stop, for example) Skipped when the resource is already flagged as new
6. BILLING-DRIVEN CHANGE (only if 1 to 5 found nothing) Matches: usage-billed resources (object storage, registries) whose daily cost rose although they have no hourly rateMore than one check can match the same resource. The note lists every match, separated by semicolons. One rule trims the list: when a reservation expiry matches, the rate-increase line is dropped, because losing a discount raises the rate on its own.
When nothing matches
If no check matches a resource, its line reads:
Cost increased: review resource configuration, usage patterns,and pricing for recent changesThat is the honest outcome. ZopNight does not guess. The fallback tells you the cause sits outside what these checks can see, so the next step is to drill into narrower dimensions (L1) and your own change records.
Org-level extras
An org-level note can carry three more sections:
NEW CLOUD ACCOUNT "New cloud account onboarded" or "N new resources added to <account> cloud account", followed by a resource type breakdown
SUMMARY LINE (when more than one resource spiked) "Primary reason" is the cause group with the largest dollar increase, not the one with the most resources
COST INCREASE BY SERVICE Added when over half of the increase is billed spend that is not tied to any one resource (traffic, requests, transfer)Where you read it
CONSOLE Dashboard (Cost Over Time widget) or Cost Reports (daily view) > anomaly banner > View All > pick one The Cost Anomaly drawer shows Actual, Expected / Forecast and Deviation, then Insights, Cost Increase by Service, Resource Type Breakdown and Affected Resources. Expand a resource row to see its "Why:" line.
ALERTS The same note is appended to the anomaly notification sent to your Slack, Teams, Google Chat or webhook channel
API/MCP rootCause field on GET /orgs/{orgID}/reports/anomalies, also returned by the get_anomalies MCP toolVerifying each cause
The note says what changed. You still confirm it and decide what to do:
RATE INCREASE Check the resource's current size and its activity history. Named user: ask them. "Pricing updated": confirm the rate on the bill.SCHEDULE FAILED Open the schedule's history. Did the stop run?NO SCHEDULE Should this resource be on a schedule?RESERVATION EXPIRY Check the commitment's end date. Renew or accept.NEW RESOURCE Find who created it and whether it should stay.COST APPEARED Who started it again after the idle period?BILLING-DRIVEN Look at the service's request or storage growth.FALLBACK Drill into team, resource group and resource.2. Demo
A team’s anomaly with its root-cause note:
T+0 Daily anomaly run detects: prod-payments resource group up 250%
T+1 min Notification arrives with the note: Details: 1. experiment-cluster-1 nodes: $0 to $2,400/day New resource: first tracked 1 day ago 2. payments-worker-3: $60 to $190/day Scheduled shutdown may have failed: ran 24.0h instead of usual 10.0h (+140%)
T+5 min Engineer verifies cause 1: Cluster created by the CI pipeline yesterday It was meant to be torn down after an experiment Clean-up step failed; cluster still running
T+8 min Engineer verifies cause 2: Schedule history shows last night's stop failed with a cloud API error
T+15 min Actions: Terminate the experiment cluster Stop payments-worker-3 and re-run the scheduleT+20 min Follow-ups: Fix the CI clean-up step Subscribe the team to the resource failed event
OUTCOME: Both causes were in the note; both were real Two separate fixes, found in 15 minutesContrasting example: the note falls back
T+0 Anomaly: org-level +45%
T+1 min Note: Summary: 38 resources had a cost increase. Each resource line reads "Cost increased: review resource configuration, usage patterns, and pricing"
T+5 min No check matched, so the engineer drills down: Cloud account: prod-aws-us-east-1 carries most of it Team dimension: team=search up 60%
T+12 min Team=search confirms a feature launch yesterday Many small services, each a little busier No single resource changed size or schedule
OUTCOME: The fallback was correct: no single resource-level cause Dimensions found it; the launch was expected Noted in the change log; anomaly closedThe note narrows the search when a check matches and is honest when none does.
3. Hands-on (5 min)
Investigate an anomaly using its root-cause note:
□ STEP 1: Open a recent anomaly Dashboard or Cost Reports > anomaly banner > View All Anomaly: __________ Severity: __________
□ STEP 2: Read the note Summary line (if any): __________ Causes listed: __________ Fallback only? yes / no
□ STEP 3: Verify the first cause listed Check used (activity, schedule history, commitment, owner): ______ Confirmed or not: __________
□ STEP 4: Verify any other causes, or drill into dimensions if the note fell back Finding: __________
□ STEP 5: Document Actual root cause: __________ Resolution: __________ Action items: __________A 15-minute exercise builds the habit of reading, then verifying.
Do it through MCP
The same task you just did in the console, asked in one sentence.
BEFORE A ZopNight account with one cloud connected. At least 4 days of daily cost data (15 for the z-score path), or detection has nothing to compare against.ASK "What anomalies fired this week, what does each root-cause note say, and who changed the resources it names?"CHECK that every cause in the answer comes from the rootCause text, not from the assistant's own guess.Tools behind it: get_anomalies (read, Cost), get_resource_activity (read, Cost), get_state_history (read, Operate). Connection details are in the MCP integration docs.
4. Knowledge check
Q1
An anomaly’s note lists one resource as “New resource: first tracked 1 day ago”. Best first step:
A. Close it, since new resources are always expected
B. Raise the budget so that it comfortably covers the new resource going forwards
C. Stop every resource in the affected cloud account at once
D. Find the resource in Affected Resources and confirm who created it and why
Show answer
Correct: D. The note tells you which check matched. Your job is to verify it: find the resource, confirm who created it, and decide whether it stays.
Q2
Every resource line reads “Cost increased: review resource configuration, usage patterns, and pricing for recent changes”. What does that tell you?
A. None of the checks matched; drill into narrower dimensions to find the source
B. The analysis is still running and will update with a ranked cause later
C. The spike is billing noise and can be closed without further review
D. ZopNight has paused detection for this org until the data settles down
Show answer
Correct: A. That is the fallback line, written when no check matches. ZopNight does not guess. Drill into team, resource group and resource, and check your own change records.
Q3
The note says a resource’s hourly rate rose from $0.20 to $0.80, “pricing updated”, with no user named. What does that mean?
A. Someone resized it, and ZopNight is withholding the user’s name
B. The resource is new, so its first rate is being compared to zero
C. The rate rose over 50%, but no resize event was found in the activity log
D. A reservation expired, so the resource fell back to on-demand pricing
Show answer
Correct: C. The rate check matched, but no resize or modify call was found near that day, so the note stays neutral instead of claiming a resize. A reservation expiry would show its own line and drop the rate line.
5. Apply
Every anomaly carries a root-cause note: in the Cost Anomaly drawer, in its notification, and in the rootCause field of the API.
For your team: read the note, verify each cause it names, and drill into dimensions when it falls back. Record the confirmed cause when you close the anomaly.
Related lessons
- L1: The seven dimensions
- L2: Detection methods
- L3: Severity bands
- L5: Redistribution suppression (next)