Amazon EMR Costs: Idle Clusters, Serverless Apps and Spot

Two different bills

On EMR on EC2, the EMR price is added to the EC2 and EBS price of the cluster’s instances, charged per second with a one-minute minimum. A cluster that finished its steps and sat waiting keeps paying for every instance. On EMR Serverless you pay for the vCPU, memory and storage consumed from the moment workers are ready until they stop.

The right lever for each

EMR on EC2 clusters. AWS’s answer to the waiting cluster is an auto-termination policy: an idle timeout, one minute to seven days, after which the cluster shuts down. On recent releases a cluster counts as idle only when there are no active YARN applications, HDFS use is under 10%, and no notebooks, Studio connections, application UIs or pending steps are active. Auto-termination is not supported for non-YARN applications such as Presto, Trino or HBase.

EMR Serverless applications. An application moves through Started, Stopping and Stopped states, and a stopped application has no resources running. EMR Serverless is on ZopNight’s Start/Stop supported list, so an application can follow the same schedules, groups and overrides as a VM.

ZopNight’s EMR rules

• Idle EMR Cluster fires on clusters in the WAITING state and is filed under schedule recommendations. The fix it recommends is enabling an auto-termination idle timeout, and the saving is the measured idle fraction of the cluster’s cost. When a cluster is idle almost all of the time it abstains, because that cluster is a shutdown candidate, not a scheduling one.
• EMR Spot Task Nodes Opportunity reads the market and instance type of the task nodes and prices moving them to Spot from live rates.

Both run on a read-only connection. Stopping an EMR Serverless application needs read + write.

Related

Fact sheets for EMR clusters and EMR Serverless, and how EMR Serverless schedules run. The Spot trade-offs are on Spot instances.