Cost anomalies
Always-on anomaly detection across org, account, group, resource, and team; with root cause analysis on every flag.
A “surprise bill” is rarely a single bad cloud day; it’s usually a small spike that compounded for a week before anyone noticed. ZopNight’s anomaly detector runs daily, compares every dimension against its own baseline, and surfaces deviations the day they happen. Every flag comes with a root cause analysis so the investigation starts at “why” instead of “where.”
Five dimensions watched
Anomaly detection runs against each of these every day:
| Dimension | What “anomalous” means at this level |
|---|---|
| Org-wide | Total spend deviates from the 7-day rolling average |
| Cloud account | One account’s spend deviates from its own 7-day average |
| Resource group | A group’s spend deviates from its own average |
| Resource | An individual resource’s daily cost deviates; capped to the top 10 per org per day to avoid noise |
| Team | A team’s allocated spend deviates; with redistribution suppression (see below) |
Detection methods
Two methods run in parallel; the higher severity wins.
| Method | What it computes |
|---|---|
| Percent deviation | Today’s value vs the 7-day rolling average, as a percentage |
| Z-score | Today’s value’s deviation from the mean in standard deviations |
Both methods need at least 4 data points and a minimum $1/day baseline before they flag. The stddev must be at least 10% of the mean; this filters out flapping noise on very stable baselines.
Severity bands
| Severity | Threshold (percent deviation) |
|---|---|
| Warning | 30%–150% above baseline |
| Critical | 150%–500% above baseline |
| Emergency | More than 500% above baseline |
A z-score of 3+ overrides the percent band to at least Critical even if the percent deviation is smaller; useful for cases where the baseline is high but the variance is normally low.
Root cause analysis
Every flagged anomaly carries an RCA tag identifying the most likely cause:
Instance resize
The hourly rate on a resource changed. Usually means someone upsized.
New resource
A resource that didn’t exist a day ago is contributing to today’s spend.
Reservation expiry
A Reserved Instance or Savings Plan term ended. The same workload is now on-demand.
Schedule failure
A ZopNight schedule was supposed to stop the resource but didn’t. The audit log will have the error.
Unscheduled usage increase
The resource is running more hours than it did last week without a schedule attached.
The RCA tag is rule-based; derived from the resource’s state history, cost records, and metric series. It’s a starting point, not a final answer. The anomaly drawer shows the underlying data so you can verify.
Team redistribution suppression
A common false positive: shared resources getting reassigned between teams. Team A’s cost spikes, team B’s cost drops by roughly the same amount, net change is near zero. ZopNight detects this; if the absolute net change across teams is under 20%, both anomalies are suppressed as a redistribution rather than fired as separate alerts.
Evaluation vs notification
These are two separate layers and you can run one without the other.
Evaluation
The anomaly cron always evaluates and persists anomalies for every org, regardless of notification setup. So:
- Anomalies appear on the dashboard’s CostTrendWidget markers
/reports/anomalies?from=&to=returns them via the API- The anomaly drawer in the UI shows them
…whether or not you have a notification channel configured. This was a deliberate change; earlier behaviour silently short-circuited evaluation when no channels existed.
Notification (optional)
Layered on top of evaluation. Subscriptions are per dimension and per severity:
- Subscribe channel A to
cost.anomaly.team.critical - Subscribe channel B to
cost.anomaly.resource.emergency - …etc.
Notifications deduplicate by date + entity so you don’t get re-paged for the same anomaly on the same day. Severity escalation re-notifies; if today’s anomaly was Warning yesterday and Critical today, the Critical fire goes out.
Filtering and exporting
The Anomalies tab on the Reports page supports filtering by:
- Date range (
from,to) - Dimension (org / cloud account / resource group / resource / team)
- Severity
- RCA tag
Pagination is server-side. Export to CSV runs as a background job; your browser doesn’t need to stay open. ZopNight notifies you with a signed download link when the file is ready.
Performance; why it’s fast on big orgs
The cron is adaptive. Small orgs (≤5K resources) run in batches of 25, medium orgs (≤20K) in batches of 5, large orgs (>20K) one at a time. This caps peak memory.
Inside each org, ZopNight builds an in-memory cost-record index keyed by resource UID. RCA lookups against the index are O(1); no per-resource database round trip when the analyser asks “what was this resource’s rate yesterday?”
Pipeline prefetch overlaps IO with computation; while one org’s anomalies are being computed, the next org’s cost records are being fetched. Group members are fetched once per org and shared across the resource-group and team detectors.
Schedule
The anomaly cron runs daily at 04:30 UTC. You can trigger a manual run via POST /internal/run-cron?job=anomaly-detection if you need to evaluate immediately; useful right after a billing sync backfill, when historical anomalies will surface for the first time.