Audit Whether Your Start/Stop Schedules Actually Ran

A start/stop schedule fails in a few quiet ways: a permission is removed and the call is denied, someone starts an instance by hand at 23:00 and forgets it, a new instance is launched without the tag the schedule keys on. None of these raise an alarm by default. This page sets up three checks: did the call happen, is the estate in the state it should be, and who changed it.

Check 1: did the stop call happen?

CloudTrail records StopInstances and StartInstances as management events. The lookup API covers the last 90 days in one Region and accepts a single lookup attribute per call.

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=StopInstances \
  --start-time 2026-09-23T18:00:00Z --end-time 2026-09-23T20:00:00Z \
  --query "Events[].[EventTime,Username,Resources[0].ResourceName]" --output text

Run it for the window your schedule should fire in. No events means the scheduler did not call, or called in a different Region. If events come back, open the CloudTrailEvent JSON string on each one: an errorCode there means the call was made and refused, which after an IAM change is the first thing to rule out.

Check 2: is anything running that should be off?

After the stop time, sweep for tagged instances that are still up:

aws ec2 describe-instances \
  --filters "Name=tag:env,Values=dev" "Name=instance-state-name,Values=running" \
  --query "Reservations[].Instances[].[InstanceId,LaunchTime,Tags[?Key=='Name']|[0].Value]" \
  --output table

Anything listed is either a failed stop, a manual start after the stop, or a new instance. Compare LaunchTime with your stop time to tell a restart apart from a survivor. For Google Cloud instance schedules, Google’s own guidance is to check the audit logs regularly, because revoking the service agent’s role can make schedules stop working without notice.

Check 3: who started it again?

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=i-0abc123def4567890 \
  --max-items 20 \
  --query "Events[].[EventTime,EventName,Username]" --output text

A StartInstances by a person shortly after the scheduled stop usually means the schedule does not match how that team works. Change the schedule or give them an override path, rather than re-stopping it every night.

Turn the checks into a routine

Run check 2 on a timer the morning after each stop window and post the output to a channel the owning team reads. Review the list of repeat offenders monthly; the same five instances appearing every week point at a schedule that needs different hours, not at people who need reminding.

What ZopNight records and alerts on

ZopNight keeps a state history of every start and stop with what triggered it: the schedule by name, a manual action by user, or an override. Failed actions are retried automatically up to three times and then routed to a dead-letter queue, and notifications can go to Slack, Teams, Google Chat or a webhook for failures, exhausted retries and override changes as well as for each scheduled start and stop. A dashboard widget reports the schedule success rate, counting a firing on a resource already in the target state as skipped rather than missed. ZopNight does not currently alert when a resource’s actual state drifts from its schedule, so the running-instance sweep above is still worth keeping. See also audit trails and sending schedule alerts to Slack.