When cost becomes a SEV · ZopDev University

Outcome

By the end of this lesson, you will be able to define criteria for treating cost as an incident, distinguish SEV-1 through SEV-4 cost severity levels, and decide between “kill immediately” and “investigate first” patterns.


Prerequisites	M3.5 (reliability vs cost) · T5.M5.5 (anomaly response)
Time	9 minutes

1. Concept

Cost can be an emergency.

A workload that got stuck in a loop. A stolen credential mining cryptocurrency on your account. A test that spun up a thousand machines and never stopped. None of these is a FinOps problem in the usual sense: they are outages that happen to show up on the bill instead of on a status page.

Treat them the way you treat outages. Decide in advance what counts, and what the response is at each level.

COST-INCIDENT TRIGGERS                    SEVERITY
─────────────────────────────────────────────────────────────────
Daily spend >2× normal sustained          SEV-2
Daily spend >5× normal (clear runaway)    SEV-2/1
Compromised credentials → cost spike      SEV-1 (security + cost)
Misconfigured autoscaler → instance flood  SEV-2/1
Stuck job consuming GPU/expensive          SEV-2
Quota exhausted from rapid growth          SEV-3
Compliance violation discovered            SEV-2/3
Budget threshold breached                  SEV-3/4

The framing: cost spikes are signals, sometimes of larger issues (security incidents, runaway automation). Treat as engineering incidents, not just FinOps concerns.

SEV definitions for cost incidents

SEV-1: EXISTENTIAL
  Cost trajectory threatens business viability
  OR indicates security compromise

  RESPONSE:
    Page everyone (on-call SRE + FinOps lead + Security)
    Stop the bleeding NOW (kill workloads, rotate credentials)
    Communicate to leadership immediately
    Customer comms if security incident

  EXAMPLES:
    Compromised key launching crypto miners ($10K+/hour)
    Runaway pipeline launching 1000s of GPU instances
    Suspected fraudulent customer activity at scale

SEV-2: SIGNIFICANT
  Cost is 2-5× normal
  Or: clear runaway pattern though sustainable for hours

  RESPONSE:
    Investigation + corrective action within hours
    Notify finance + leadership
    Page FinOps on-call
    May require kill-stop of identified runaway workloads

  EXAMPLES:
    CI/CD bug launching too many test instances
    Misconfigured HPA scaling to max indefinitely
    New feature traffic 10× projection

SEV-3: CONCERNING
  Spike or growth pattern requires attention
  Not immediate crisis

  RESPONSE:
    Document; address within days
    Investigation during business hours
    No paging
    Weekly Operate review item

  EXAMPLES:
    50% sustained increase on a workload
    New cost line item ($1K-$10K/mo)
    Budget pacing concerning

SEV-4: TRENDING
  Modest deviation; monitor

  RESPONSE:
    Note in weekly Operate review
    Investigate if persists
    No immediate action

  EXAMPLES:
    Mild growth on known feature
    Seasonal patterns within normal range

The severity drives the response intensity. Don’t overreact; don’t underreact.

Cost SEV vs traditional outage SEV

TRADITIONAL OUTAGE SEV          COST SEV
─────────────────────────────────────────────────────────────
Customer impact                  Financial impact
Page on-call SRE                 Page FinOps lead + on-call
SRE leads investigation          FinOps + SRE collaborate
Production stays running         May require kill-stop
Customer status pages             Internal comms primarily
Focus: restore service           Focus: stop the bleeding
                                  (then restore intent)

The response differs because cost runaways often involve:

Possibly compromised credentials (security implication)
Need to kill rogue workloads (mutation that’s risky)
Finance escalation alongside engineering
Different stakeholders (CFO/CTO vs CTO/VPE alone)
Damage that grows at the burn rate for every minute the runaway continues: by minute 50 it is ten times what it was at minute 5

Common cost-SEV scenarios

SCENARIO 1: COMPROMISED AWS KEY LAUNCHING CRYPTO MINING

  Initial signal: Cost Anomaly alert fires at 3 AM
                  $6,400 cost spike in 2 hours
                  All from us-east-2 GPU instances

  Investigation: 32 GPU instances launched
                  Tagged with placeholder/fake tags
                  Created by AWS access key X-X (not familiar)

  SEVERITY: SEV-1
    Stop the bleeding (kill instances)
    Rotate credentials immediately
    Engage security incident process

  ACTIONS:
    T+0:00 Page on-call
    T+0:05 Identify the workload (clearly malicious)
    T+0:10 Kill all 32 GPU instances
    T+0:15 Rotate AWS access keys org-wide for affected scope
    T+0:30 Security forensics begins
    T+2:00 Initial root cause: compromised CI/CD token
    T+24h Postmortem; new credential hygiene policies

SCENARIO 2: CI/CD BUG LAUNCHING TEST INSTANCES

  Initial signal: Cost spike $8,500 in 6 hours
                  ML training instances in non-prod
                  Created by CI/CD service account

  Investigation: ML pipeline change deployed last night
                  Bug: each test run launches new instance, doesn't terminate

  SEVERITY: SEV-2
    Kill the workloads
    Pause CI/CD pipeline
    Fix root cause

  ACTIONS:
    T+0:00 Cost alert fires
    T+0:30 Identify source (CI pipeline)
    T+0:45 Pause pipeline; kill orphaned instances
    T+1:30 Engineer identifies bug; develops fix
    T+3:00 Fix deployed
    T+4:00 Pipeline re-enabled; monitor
    T+24h Postmortem; CI cost-monitoring added

SCENARIO 3: SUDDEN 50% SPIKE ON NON-PROD

  Initial signal: deviation alert (non-prod workload up 50%)
                  $2,500/mo increase

  Investigation: feature team ramping up testing
                  Documented in change log
                  Expected for the next 2 weeks

  SEVERITY: SEV-4
    Document
    Track via Operate
    No immediate action

Most cost concerns are SEV-3 or SEV-4; SEV-1 and SEV-2 are rarer but consequential.

The kill-vs-investigate decision: the critical moment

KILL IMMEDIATELY when:
  ✓ Cost trajectory clearly unsustainable
    (will exhaust budget in hours/days)
  ✓ Suspected credential compromise
    (every minute the workload runs = more cost)
  ✓ Source is OBVIOUS (one specific resource group)
  ✓ Risk of kill (data loss) is LOW
    (training jobs with checkpoints, test workloads)
  ✓ Off-hours (no engineer to investigate live)

INVESTIGATE FIRST when:
  ✓ Cost growth is sustainable / explained
  ✓ Source ISN'T identified yet
  ✓ Killing could cascade (production dependencies)
  ✓ Risk of kill is HIGH (data loss possible)
  ✓ Business hours (engineers available)
  ✓ Workload is customer-facing production

WRONG DECISION CONSEQUENCES:
  Kill when should have investigated: data loss, service impact
  Investigate when should have killed: cost continues to spike;
    potentially $50K+ wasted while you "investigate"

DEFAULT for ambiguous cases:
  Page the team lead; collective decision
  Default to "investigate" if any production involvement
  Default to "kill" if security incident suspected

This decision is the highest-stakes part of cost-incident response.

Detection signals

ZopNight + cloud-native alerts trigger:

  ZOPNIGHT COST ANOMALY (daily):
    Runs once a day on daily cost rollups
    Flags % deviation from a 7-day rolling average
      (plus a z-score check)
    Severity: warning ≥30%, critical ≥100%, emergency ≥500%
    Catches "yesterday was wrong"; too slow for a 3 AM runaway
    Triggers SEV-3+ investigation

  BUDGET THRESHOLD:
    Configured per team/workload
    Triggers SEV-2/3 based on threshold severity

  CLOUD-NATIVE (AWS Cost Anomaly Detection):
    Catches patterns outside normal
    Latency: runs about three times a day; detection can take
    up to 24 hours after usage
    Use as backup signal

  CUSTOM:
    Hourly cost rate check (catches faster than daily)
    Resource count anomaly (1000 new instances = anomalous)
    Specific resource-type spike (sudden GPU costs)

The detection latency varies; design for the worst case.

Response runbook structure

COST INCIDENT RUNBOOK (per-team document):

1. DETECTION
   What alerts trigger which severity?
   Who gets paged?

2. INITIAL TRIAGE (first 5 minutes)
   What information to gather?
   Identify: source, scope, trajectory

3. KILL-OR-INVESTIGATE DECISION
   Decision tree based on signal
   Authority to kill (team lead? on-call?)

4. KILL PROCEDURE
   Step-by-step for common runaways
   Approval requirements
   Rollback if killed wrong thing

5. INVESTIGATION PROCEDURE
   What logs to check
   Who to contact
   How to track cost while investigating

6. ESCALATION
   When to escalate to leadership
   When to engage security
   Customer comms if applicable

7. POSTMORTEM
   Template
   Cost impact summary
   Process improvements

Print the runbook; reference during incidents. Update post-incident.

2. Demo

A real SEV-1 cost incident:

INCIDENT (2026-05-21, 3 AM UTC):

T+0:00    Cloud cost anomaly alert fires
          Subject: "Cost up 800% in last 3 hours in us-east-1"
          Auto-routed to FinOps on-call

T+0:01    FinOps on-call paged (Slack + PagerDuty)
          On-call: alice@platform

T+0:03    Initial investigation:
          Cost Explorer → us-east-1 → EC2, hourly granularity
          Spike entirely from EC2 GPU nodes in the prod EKS cluster
          47 new GPU nodes (p4d-class) in last 3 hours

T+0:05    Identify origin:
          Nodes added by the Cluster Autoscaler for pending pods
          Pods: payment-team's order-processor
          HPA order-processor: 480 replicas, maxReplicas=500
          (expected max: 50)

T+0:08    Suspected: HPA misconfiguration, not an attack
          Risk assessment:
            Cost trajectory: $4,200 in 3 hours = ~$33K/day continued
              (47 GPU nodes at ~$29/node-hour)
            Source: legitimate automation with a wrong config
            Risk of capping: low (normal peak needs <50 replicas)

T+0:10    DECISION: SEV-1 (clear runaway; ~$33K/day trajectory), cap now
          Authority: on-call has authority for SEV-1 kill
          Brief platform-lead via Slack

T+0:12    Execute:
          kubectl patch hpa order-processor \
            -p '{"spec":{"maxReplicas":50}}'
          Excess pods terminate; the 47 extra GPU nodes are drained
          and terminated

T+0:18    Cost rate confirms drop:
          Replicas back to ~40; extra GPU nodes gone
          Current spend rate: normal

T+0:30    Root cause confirmed:
          Last week's HPA refactor set maxReplicas=500 (meant 50)
          Passed review; nothing caps HPA max in prod

T+1:00    Hotfix merged to Git (max=50), so the next deploy
          cannot revert the cap
          Cost contained; audit of other clusters that got the
          same refactor begins

T+1 hour Communication:
          Slack to #eng-platform: "Cost incident contained;
            details to follow in postmortem"

T+24h     Postmortem:
          Total spend above forecast: ~$4,620 ($4,200 before
            detection + ~18 minutes at ~$1,400/hour)
          Time from detect to mitigate: 18 minutes
          Time from first spike to detection: ~3 hours
            (this is what we need to improve)

          Action items:
            1. Cap HPA max for prod payment-team workloads at 50
            2. Cost-rate alarm at >5× baseline for prod accounts
            3. Hourly cost-spike detection (catch faster than daily)
            4. Runbook section: identify and stop a runaway autoscaler
            5. Quarterly autoscaler config audit

OUTCOME:
  Cost runaway stopped (would have been ~$33K/day)
  Root cause fixed in code, not just in the cluster
  Process improvements identified
  Team confidence in runbook

The runbook works because it was documented, practiced, and trusted.

3. Hands-on (5 min)

Define cost-incident criteria for your team:

□ STEP 1: Define SEV thresholds
  SEV-1: __________ (what triggers it?)
  SEV-2: __________
  SEV-3: __________
  SEV-4: __________

□ STEP 2: Identify response responsibilities
  SEV-1: who pages? __________
  SEV-2: who pages? __________
  SEV-3/4: who tracks? __________

□ STEP 3: Document kill-vs-investigate rules
  Kill criteria: __________
  Investigate criteria: __________

□ STEP 4: Build runbook outline
  □ Detection signals
  □ Initial triage
  □ Decision tree
  □ Kill procedure
  □ Investigation procedure
  □ Escalation
  □ Postmortem

□ STEP 5: Schedule test
  Annual drill: practice cost incident response
  Date: __________

A 30-minute exercise defines the framework. Annual drill validates it.

4. Knowledge check

Q1

A 5× daily cost spike on a single resource group, sustained:

A. Random fluctuation; ignore
B. SEV-4
C. Not actionable
D. SEV-2

Show answer

Correct: D. A sustained 5× spike usually exceeds budget tolerance and points to a real problem: investigate within hours, take corrective action, and notify finance and leadership. It is not SEV-1 unless the trajectory is unsustainable for the business.

Q2

Suspected credential compromise driving cost:

A. Investigate first to confirm
B. Rotate the credential and keep watching
C. Kill the workloads IMMEDIATELY
D. Wait for more data

Show answer

Correct: C. Stop the bleeding first: every minute the workload runs adds cost. Then run the security incident process, because the cost is incidental to the bigger issue, the compromised credential. Compromise scenarios default to SEV-1.

Q3

A 50% spike with clear cause (new feature ramping):

A. SEV-1
B. SEV-3 or 4
C. SEV-2, since the spend is material
D. Ignore it; growth is never tracked

Show answer

Correct: B. The growth is expected and documented in the change log, so no incident response is needed: track it in the Operate review and watch the trend. Not all cost growth is an incident.

5. Apply

Cost SEV criteria + response playbook in team runbook. Test annually with a drill.

For your team: define SEV thresholds; document runbook; reference during incidents; update post-incident.

Outcome

1. Concept

SEV definitions for cost incidents

Cost SEV vs traditional outage SEV

Common cost-SEV scenarios

The kill-vs-investigate decision: the critical moment

Detection signals

Response runbook structure

2. Demo

3. Hands-on (5 min)

4. Knowledge check

Q1

Q2

Q3

5. Apply

Related lessons

Start with the bill.