EMR clusters sitting idle between jobs
What does ZopNight detect here?
EMR reports an IsIdle metric directly, and ZopNight uses it alongside AppsRunning over 30 days. This is categorised as a scheduling finding rather than deletion: EMR clusters are usually idle between batch runs, and the fix is a schedule or auto-termination, not removal.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-051 |
| Category | schedule |
| Severity | medium |
| Metric | none — pure configuration read |
| Source | emr_idle.go |
Where it applies
AWS tells you this one directly
EMR emits an IsIdle metric: a cluster reporting no active applications and no work in
progress. Combined with AppsRunning, the signal needs no inference: EMR itself is stating
whether the cluster is doing anything.
That makes this one of the more reliable idle findings in the catalogue, and it is why the threshold logic is simpler than for compute resources where idleness has to be inferred from utilisation.
Why it is a schedule finding, not an idle one
The rule’s category is schedule, deliberately. An EMR cluster idle between batch runs is
working as designed. The question is whether it needed to exist during the gap, not whether it
should exist at all.
That distinction changes the remediation completely. Deleting the cluster is usually wrong; turning off the hours it spends waiting is usually right.
Auto-termination is the first thing to check
EMR supports an auto-termination policy that shuts a cluster down after a configurable idle period. A cluster without one, running a job that finishes in two hours, keeps billing per instance-hour until somebody notices, which on a large instance fleet is expensive fast.
For clusters running scheduled batch work, a start/stop schedule aligned to the job window is the alternative. For genuinely intermittent work, EMR Serverless removes the question entirely by billing only for job execution.
Reading the IsIdle metric hour by hour
aws cloudwatch get-metric-statistics \ --namespace AWS/ElasticMapReduce --metric-name IsIdle \ --dimensions Name=JobFlowId,Value=j-0123456789ABC \ --start-time "$(date -u -v-7d +%Y-%m-%dT%H:%M:%SZ)" \ --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ --period 3600 --statistics MaximumAn hourly period shows the shape of the idle windows, which is what you need to size a schedule.