CloudWatch alarms stuck in INSUFFICIENT_DATA because the monitored resource is gone

An alarm keeps billing after its target is deleted

CloudWatch pricing charges per alarm metric each month; its worked example prices four standard-resolution alarms at $0.10 per alarm metric, or $0.40 a month. The free tier covers 10 alarm metrics. Delete an EC2 instance and its alarms stay behind. They cannot evaluate anything, so they drop into INSUFFICIENT_DATA, which the alarm evaluation guide defines as the state for an alarm that has just started, whose metric is not available, or that lacks enough data to decide.

Each orphan is cheap. Accounts with years of instance churn can hold hundreds.

Listing alarms that have gone quiet

aws cloudwatch describe-alarms --state-value INSUFFICIENT_DATA \
  --query 'MetricAlarms[].[AlarmName,Namespace,Dimensions[0].Value,StateUpdatedTimestamp]' \
  --output table

For each AWS/EC2 alarm, check whether the instance in its dimensions still exists with aws ec2 describe-instances --instance-ids. An error saying the instance is not found, or a terminated state, confirms the alarm is orphaned.

Proof that the source is gone

INSUFFICIENT_DATA alone is never enough, because an alarm can be in that state for many harmless reasons. ZopNight requires a confirmed signal that the monitored resource no longer exists, established by matching the alarm against the resources it has discovered. When ZopNight knows when the alarm entered INSUFFICIENT_DATA, it must have been there for at least 30 days. Finally, the alarm needs a price.

Alarms that are always left alone

Alarms whose names contain scaleup, scaledown or alarmscale are treated as Auto Scaling policy alarms and skipped, since they legitimately report no data while a group is scaled to zero. Alarms without a confirmed missing source get no finding however long they have been quiet. Alarms in OK or ALARM are never considered.

What deleting it recovers

saving = the alarm's monthly charge
cost after fix = 0

Removing orphaned alarms

1. Confirm the monitored resource is really gone and was not replaced under a new ID.
2. If the resource moved, fix the alarm’s namespace or dimensions instead of deleting it.
3. Delete it with aws cloudwatch delete-alarms --alarm-names and the alarm name.
4. Add alarm cleanup to the teardown step of whatever tooling deletes instances.