Root cause analysis

ANOMALY: org-level +200% spike yesterday
ROOT CAUSE ANALYSIS suggests:
  Most likely (60%): New resource provisioned in prod-aws-us-east-1
  Less likely (25%): Instance rate change (rate increase)
  Less likely (10%): Schedule failure (resource that should have stopped)
  Less likely (5%):  Reservation expiry

1. INSTANCE RATE CHANGE (resource resize)
   Detection: instance type changed; new rate applies
   Example: m5.large → m5.2xlarge mid-day
   Common: rightsizing decisions; experimentation

2. NEW RESOURCE
   Detection: resource UID not in yesterday's billing
   First time seeing it
   Common: provisioning by CI/CD; new feature

3. RESERVATION / SAVINGS PLAN EXPIRY
   Detection: known reservation end date matches the day
   Cost shifts to on-demand rate
   Common: end-of-quarter; forgot to renew

4. SCHEDULE FAILURE
   Detection: resource was supposed to be off but is running
   ZopNight schedule was meant to stop it
   Common: schedule misconfigured; override expired wrong

5. UNSCHEDULED RESOURCE USAGE INCREASE
   Detection: usage hours up but instance type unchanged
   Resource ran more hours than expected
   Common: stuck job; runaway loop; auto-scale to max

FOR EACH ANOMALY:
  Compare yesterday's cost record to recent average
  Compare resource attributes:
    Instance type / class changed?
    Resource was previously unseen?
    Reservation coverage changed?
    State history shows unusual on-state?
    Activity log shows unusual operations?

  Score each cause class based on signal strength
  Surface top 3 causes with probability

SIGNAL SOURCES:
  cost_records (historical cost data)
  resource attributes (instance type, etc.)
  state history (when resources started/stopped)
  activity logs (CloudTrail equivalent)

ROOT CAUSE 1: NEW RESOURCE (60% probability)
  Action: identify the new resource by UID
  Investigate:
    Who provisioned it?
    What's its purpose?
    Is it expected?
  Decision: keep / terminate / schedule

ROOT CAUSE 2: INSTANCE RATE CHANGE (25%)
  Action: check the resource's instance type now vs yesterday
  Investigate:
    Who or what changed it?
    Was it intentional?
  Decision: revert / keep / schedule

ROOT CAUSE 3: RESERVATION EXPIRY (5%)
  Action: check reservation status; renewal needed?
  Decision: renew / switch to SP / accept on-demand

ROOT CAUSE 4: SCHEDULE FAILURE
  Action: check schedule + override status
  Investigate:
    Why didn't the stop fire?
    Was override accidentally extended?
  Decision: fix schedule / clean up override

ANOMALY: ml-training spike +300%
ANALYSIS suggests:
  New resource (60%)
  Schedule failure (30%)
  Other (10%)

INVESTIGATION:
  Check #1 first: New resource
  Look for new resource UIDs
  None found
  → Top candidate ruled out

  Check #2: Schedule failure
  Look at ml-training schedule + recent fires
  Schedule meant to stop at 8 PM yesterday
  Schedule didn't fire (failed)
  → Found the actual cause!

OUTCOME:
  Schedule failure was the right answer
  Ranked #2 in analysis
  Investigation confirmed it
  Operators should read all candidates, not just #1

ANOMALY: org-level spike +50%
ANALYSIS: no clear single cause
  Possible: distributed small increases across multiple resources

SUGGESTED INVESTIGATION:
  Drill into team or resource-group dimensions
  For finer-grained signals
  Look for patterns of many small increases

INTERPRETATION:
  The cost spike isn't from one source
  Could be: 50 small resources each adding $20/day
  Or: aggregate growth from many teams
  Or: feature rollout affecting many services

HONEST OUTCOME:
  System surfaces "no clear cause"
  Doesn't fabricate a confident answer
  Lets the operator do deeper investigation

SOMETIMES MULTIPLE CAUSES contribute:

ANOMALY: org spike +400%
ANALYSIS:
  New resource (45%): new ML cluster
  Schedule failure (35%): dev environment didn't stop
  Both contributing

REALITY:
  ML cluster contributes ~$2,000
  Schedule failure contributes ~$1,500
  Together: $3,500 of the spike

BOTH need addressing:
  Investigate new resource → may be legit; keep
  Investigate schedule failure → fix the failure
  Two action items

STANDARD INVESTIGATION:

1. Read top candidate (probability)
2. Drill into the suggested signal
3. Verify or refute
4. If verified: take suggested action
5. If refuted: move to candidate #2
6. Repeat
7. If all candidates refuted: deeper investigation needed
   Check team dimension, resource-group dimension
   Cross-reference with deploy logs, change tickets

TYPICAL TIME:
  Top candidate correct: 5-10 min
  Move to candidate #2: 10-20 min
  Deep investigation: 30+ min

T+0       Daily anomaly cron detects:
          prod-payments cost yesterday up 250%

T+1 min   ROOT CAUSE ANALYSIS surfaces:
            65% New resource (prod-aws-us-east-1)
            20% Schedule failure
            10% Unscheduled usage increase
            5%  Reservation expiry

T+5 min   Team drills into "New resource" (top candidate):
            Detected: new EKS cluster "experiment-cluster-1"
            Provisioned by ci-cd-tool@zopcloud.com at 14:32 yesterday

T+10 min  Investigation:
            Cluster is a temporary experiment cluster
            Was supposed to terminate after the experiment
            CI tool failed to clean up; cluster still running

T+15 min  Action: terminate the cluster
T+20 min  Notification: cluster terminated; cost recovered
T+20 min  Postmortem:
          - Improve CI tool's cleanup logic
          - Add Lambda watchdog for orphaned experiment clusters
          - Add cleanup audit job

OUTCOME:
  15 minutes to root cause + remediation
  Top candidate was right
  Action items captured for prevention
  Cost recovered (~$2,500 in this case)

T+0       Anomaly: ml-training spike +400%

T+1 min   ROOT CAUSE ANALYSIS:
            New resource (50%)
            Schedule failure (30%)
            Other (20%)

T+5 min   Drill into "New resource":
            No new resources detected
            All resources known

T+8 min   Drill into "Schedule failure":
            Schedule was supposed to stop ml-cluster at 8 PM
            Schedule logs: failed to apply (cloud API error)
            Cluster ran all night

T+12 min  Investigation complete:
            Schedule failure was the cause
            Cloud API was rate-limited; retry exhausted

T+15 min  Action:
          - Kill the cluster (cost stopped)
          - Add retry logic to schedule application
          - Set up cloud API rate monitoring

OUTCOME:
  Top candidate ruled out; #2 was correct
  Investigation took 4 minutes longer than ideal
  But: still much faster than blind search
  Process improved (retry logic + monitoring)

□ STEP 1: Open active anomaly
  Anomaly: __________
  Severity: __________

□ STEP 2: Read root cause analysis
  Top candidate: __________   Probability: ___%
  #2: __________               Probability: ___%
  #3: __________               Probability: ___%

□ STEP 3: Investigate top candidate
  Check the suggested signal
  Verified or refuted: __________
  Time spent: ___ min

□ STEP 4: If refuted, move to #2
  Verified or refuted: __________
  Time spent: ___ min

□ STEP 5: Document
  Actual root cause: __________
  Resolution: __________
  Action items: __________

BEFORE  A ZopNight account with one cloud connected. At least 30 days of billing history, or detection has nothing to compare against.
ASK     "What anomalies fired this week, and set up alerting for the account that keeps appearing."
CHECK   the five dimensions from L1. An anomaly with no dimension attached is noise you will learn to ignore.