# Cost anomalies

> Always-on anomaly detection across org, account, group, resource, and team; with root cause analysis on every flag.

Source: https://zop.dev/docs/zopnight/reports/anomalies

---

A "surprise bill" is rarely a single bad cloud day; it's usually a small spike that compounded for a week before anyone noticed. ZopNight's anomaly detector runs daily, compares every dimension against its own baseline, and surfaces deviations the day they happen. Every flag comes with a root cause analysis so the investigation starts at "why" instead of "where."

## Five dimensions watched

Anomaly detection runs against each of these every day:

| Dimension | What "anomalous" means at this level |
|---|---|
| **Org-wide** | Total spend deviates from the 7-day rolling average |
| **Cloud account** | One account's spend deviates from its own 7-day average |
| **Resource group** | A group's spend deviates from its own average |
| **Resource** | An individual resource's daily cost deviates; capped to the top 10 per org per day to avoid noise |
| **Team** | A team's allocated spend deviates; with redistribution suppression (see below) |

## Detection methods

Two methods run in parallel; the higher severity wins.

| Method | What it computes |
|---|---|
| **Percent deviation** | Today's value vs the 7-day rolling average, as a percentage |
| **Z-score** | Today's value's deviation from the mean in standard deviations |

Both methods need at least 4 data points and a minimum $1/day baseline before they flag. The stddev must be at least 10% of the mean; this filters out flapping noise on very stable baselines.

## Severity bands

| Severity | Threshold (percent deviation) |
|---|---|
| **Warning** | 30%–150% above baseline |
| **Critical** | 150%–500% above baseline |
| **Emergency** | More than 500% above baseline |

A z-score of 3+ overrides the percent band to at least Critical even if the percent deviation is smaller; useful for cases where the baseline is high but the variance is normally low.

## Root cause analysis

Every flagged anomaly carries an RCA tag identifying the most likely cause:

**Instance resize**

    The hourly rate on a resource changed. Usually means someone upsized.

**New resource**

    A resource that didn't exist a day ago is contributing to today's spend.

**Reservation expiry**

    A Reserved Instance or Savings Plan term ended. The same workload is now on-demand.

**Schedule failure**

    A ZopNight schedule was supposed to stop the resource but didn't. The audit log will have the error.

**Unscheduled usage increase**

    The resource is running more hours than it did last week without a schedule attached.

The RCA tag is rule-based; derived from the resource's state history, cost records, and metric series. It's a starting point, not a final answer. The anomaly drawer shows the underlying data so you can verify.

## Team redistribution suppression

A common false positive: shared resources getting reassigned between teams. Team A's cost spikes, team B's cost drops by roughly the same amount, net change is near zero. ZopNight detects this; if the absolute net change across teams is under 20%, both anomalies are suppressed as a redistribution rather than fired as separate alerts.

## Evaluation vs notification

These are two separate layers and you can run one without the other.

### Evaluation

The anomaly cron always evaluates and persists anomalies for every org, regardless of notification setup. So:

- Anomalies appear on the dashboard's CostTrendWidget markers
- `/reports/anomalies?from=&to=` returns them via the API
- The anomaly drawer in the UI shows them

…whether or not you have a notification channel configured. This was a deliberate change; earlier behaviour silently short-circuited evaluation when no channels existed.

### Notification (optional)

Layered on top of evaluation. Subscriptions are per dimension and per severity:

- Subscribe channel A to `cost.anomaly.team.critical`
- Subscribe channel B to `cost.anomaly.resource.emergency`
- …etc.

Notifications deduplicate by date + entity so you don't get re-paged for the same anomaly on the same day. Severity escalation re-notifies; if today's anomaly was Warning yesterday and Critical today, the Critical fire goes out.

## Filtering and exporting

The Anomalies tab on the Reports page supports filtering by:

- Date range (`from`, `to`)
- Dimension (org / cloud account / resource group / resource / team)
- Severity
- RCA tag

Pagination is server-side. Export to CSV runs as a background job; your browser doesn't need to stay open. ZopNight notifies you with a signed download link when the file is ready.

## Performance; why it's fast on big orgs

The cron is adaptive. Small orgs (≤5K resources) run in batches of 25, medium orgs (≤20K) in batches of 5, large orgs (>20K) one at a time. This caps peak memory.

Inside each org, ZopNight builds an in-memory cost-record index keyed by resource UID. RCA lookups against the index are O(1); no per-resource database round trip when the analyser asks "what was this resource's rate yesterday?"

Pipeline prefetch overlaps IO with computation; while one org's anomalies are being computed, the next org's cost records are being fetched. Group members are fetched once per org and shared across the resource-group and team detectors.

## Schedule

The anomaly cron runs daily at **04:30 UTC**. You can trigger a manual run via `POST /internal/run-cron?job=anomaly-detection` if you need to evaluate immediately; useful right after a billing sync backfill, when historical anomalies will surface for the first time.