The state-history timeline

Outcome

By the end of this lesson, you will be able to trace any resource’s history of state changes and explain what each transition represents.

1. Concept

Every time a resource changes state, it is written down. When it changed, what it was before, what it became, and what caused it. That record is kept; ZopNight does not purge it automatically.

This is where you go when the question is “what actually happened”. It is what you use to explain a bill, to chase a spike, and to answer an auditor.

What gets recorded

EVENT                                            RECORDED?
─────────────────────────────────────────────────────────
Schedule fires a start cron                       Y
Schedule fires a stop cron                        Y
Manual start (via UI or API)                      Y
Manual stop (via UI or API)                       Y
Group (bulk) start or stop, per member              Y
Override set (force-on starts, force-off stops)    Y
Override expires or is removed (corrective action)  Y
Change made outside ZopNight (console, pipeline)   N
Remediation action (resize, delete, snapshot)      N
Action that still fails after its retries           Y (one "failed" row)
Resource creation (first appearance in discovery)    N
Resource deletion (last appearance in discovery)    N

Every start or stop that ZopNight carries out is recorded, once it has succeeded, as started or stopped. An action that still fails after its retries leaves one failed row with the error. Discovery events such as a resource appearing or disappearing are not state changes and are not recorded here. Remediation actions are not part of state history either: a remediation runs as its own workflow (precondition, optional approval, action, validation) and is tracked on the recommendation and in the audit log. Track 2 covers it. The trigger source is recorded too (covered in L2).

The history view

RESOURCE: i-0abc123def456
─────────────────────────────────────────────────────────────────────
TIMESTAMP             STATE      SOURCE
─────────────────────────────────────────────────────────────────────
2026-05-20 08:00 ET   started    Scheduled: business-hours-eu
2026-05-19 20:00 ET   stopped    Scheduled: business-hours-eu
2026-05-19 14:23 ET   stopped    Manual: jane@zopcloud.com
... (continues back to the first recorded action)

Each row is one action ZopNight carried out. There are no in-between “Starting” or “Stopping” rows: a row is written once the action succeeds, with the state it left the resource in.

How far back it goes

State history is not purged automatically, so the timeline goes back to the first action ZopNight recorded for the resource. Use the date filter to narrow it.

The Audit Log (covered in L6) is not a second copy of it. The audit log records API requests that change something, plus MCP tool calls. A start or stop that a schedule makes is not an API request, so it appears in state history and not in the audit log. A manual start through the UI or API appears in both.

Reading the timeline for one resource

The resource detail panel has a History section that lists the five most recent changes, each with its state, source and time. That is the quick look.

For anything older, click View full history. It opens the History page already filtered to that resource, where you can narrow further by state, source and date. That is the drill-down.

Why this matters operationally

Three questions the history answers in <30 seconds:

Question 1: Why did this resource stop at 3 AM?

Open the resource → check the history → find the transition at 3 AM → read the trigger column. Most likely:

• “Schedule: weekend-stop-all” → an aggressive weekend schedule fired
• “Manual: api-call-from-user@team” → a human stopped it via API
• “Override Expired” → a force-on override ended, and ZopNight stopped the resource to return it to its schedule state

In 30 seconds, the team knows the cause without escalation.

Question 2: How often does this resource actually run?

Read the started and stopped rows over a week. A non-prod resource that “should run business hours” but shows ON 18 hours/day suggests the schedule isn’t firing: investigate why.

Question 3: Did the schedule fire as expected?

Pick the expected firing time. Find the transition (or absence of one). Confirm or diagnose.

Per-resource vs estate-wide history

The History section on the resource detail panel shows that one resource. For an estate-wide view (e.g., “all transitions in the last 24 hours”), the History page lists every recorded change, filterable by:

• State (started, stopped, failed)
• Source (scheduled, manual, bulk, override, override expired, override removed)
• Resource
• Date range

Useful for “what happened across the estate last night?” investigations.

Changes made outside ZopNight

A change made outside ZopNight (the cloud console, a pipeline, a script) is not written to state history, because ZopNight did not carry it out. State history only records actions ZopNight performed.

That makes a gap in the history a clue in its own right:

RESOURCE IS RUNNING NOW, BUT ITS HISTORY SAYS:
─────────────────────────────────────────────────────
2026-05-19 20:00 ET   stopped    Scheduled: business-hours-eu
(no later row)

If a resource is in a state its history cannot explain, something outside ZopNight changed it. Look in the cloud provider’s own log: CloudTrail on AWS, Cloud Audit Logs on GCP, the Activity Log on Azure. Repeated unexplained changes on the same resource usually point to an unmanaged process (a script, a CI/CD pipeline, an engineer in the console).

2. Demo

A team investigating a cost spike:

T+0       FinOps lead notices Saturday's spend was $1,400 higher than usual
          on the staging-eu account.

T+1 min   Opens Costs → Reports, drills into Saturday's spend on staging-eu.
          Sees EC2 was the driver. Specifically: i-0abc..., i-0def..., i-0ghi...
          ran for 14 extra hours each compared to the typical pattern.

T+3 min   Clicks i-0abc... to open the resource and reads its History section.

T+3:30    Sees the gap:
            2026-05-15 20:00  stopped   Scheduled: business-hours-eu
            (no row since)

          The instance is running now, and the bill says it ran from
          about 14:30 on Saturday. No ZopNight row explains it, so
          ZopNight did not start it.

T+5 min   Opens AWS CloudTrail for Saturday afternoon.
          Finds: a CI/CD pipeline (terraform apply via GitHub Actions) ran at 14:28
          and started the instance.

T+6 min   Diagnosis: an engineer's PR included a terraform module that
          accidentally started staging instances outside the schedule. The team
          fixed the PR and reverted.

T+7 min   Cost incident closed. The missing row in the state history was the key.

Six minutes from “the bill is wrong” to root cause. The state-history is the diagnostic backbone.

3. Hands-on (6 min)

For one of your scheduled resources:

1. Open the resource detail page.
2. Find the History section (the five most recent changes).
3. Click View full history and read the last 7 days of rows.
4. For each transition, identify the trigger source from the column.
5. Look for:
   - Expected schedule firings (every weekday 8 AM start, 8 PM stop)
   - Any manual events (who made them?)
   - Any override events (why?)
   - Any gap: the resource's current state does not match its last row
6. If you find a gap, that's your investigation hook: something
   outside ZopNight is touching this resource. Check the cloud
   provider's own log for that window.

For estate-wide:
7. Open the History page.
8. Filter to Status = Failed for the last 30 days.
9. Count the failed rows. If any resource fails repeatedly,
   investigate its permissions or capacity.

4. Knowledge check

Q1

A team asks: “Why did our staging RDS stop at 4 AM Monday?” The fastest path to answer is:

A. Search the Slack history
B. Open the RDS resource → State History timeline → find the 4 AM Monday transition → read the trigger column
C. Check AWS CloudTrail
D. Ask the on-call engineer what they happen to remember about that particular night in question

Q2

A resource is running, but its state history shows its last row as stopped. The most likely explanation:

A. The state history skipped writing the started row for the most recent run
B. The schedule fired twice in that window and the second row was merged away
C. Something outside ZopNight started it, so no row was written
D. Discovery has not yet synced the newest state history rows

Q3

Which of these is written to a resource’s state history?

A. A start or stop carried out by a schedule, a person, a group action or an override
B. The resource first appearing in a discovery sync after it was created in the cloud
C. A rename of the schedule that the resource happens to be attached to
D. A read-only MCP tool call that looked up the resource’s current details

5. Apply

State-history is in:

• Resource detail → History: the five most recent changes, with a link to the full list
• History page: every recorded change, filterable by state, source, resource and date
• Audit Log: who changed what through the API or MCP

For interpreting the trigger column, continue to L2.

Related lessons

• L2: Reading the trigger column (next)
• L6: Where audit logs live