SageMaker HyperPod clusters under 5% CPU and 10% memory for 7 days that could scale to zero off-hours

An accelerator cluster that is switched on but doing nothing

HyperPod clusters are built for large training jobs, usually on GPU or Trainium instances, and SageMaker AI pricing lists them by instance hour. Between jobs, a cluster can sit for days with its nodes almost untouched. Because each instance group can be set to zero nodes, per the instance group specification, the cluster definition can stay in place while the expensive hardware is released.

Checking what the cluster is running

aws sagemaker list-clusters --query 'ClusterSummaries[].[ClusterName,ClusterStatus]'

aws sagemaker list-cluster-nodes --cluster-name my-cluster \
  --query 'ClusterNodeSummaries[].[InstanceGroupName,InstanceId,InstanceType,LaunchTime]' \
  --output table

Compare the node list with your job scheduler: if Slurm has had nothing queued for a week, the nodes are idle.

The near-zero test

1. The cluster is InService. A cluster already at 0 nodes costs nothing and is not flagged.
2. Node CPU and memory series both exist for the cluster, each with at least 7 days of data.
3. Over the 7-day window, average CPU is below 5%, average memory below 10%, and CPU never peaks above 20%, so there are no hidden bursts.
4. The cluster has a positive monthly cost and is not covered by a Savings Plan or reservation.
5. ZopNight has a measured weekly usage pattern for the cluster with an idle share above zero.

Clusters left alone

A cluster covered by a Savings Plan or reservation produces no finding, because scaling it down would not stop the committed charge. Clusters orchestrated by Amazon EKS report node metrics to a different place and are not evaluated. With missing metrics, a short data window or no measured idle share, the rule stays silent. Clusters between 5% and 10% CPU are sized, not idle, and are covered by Over-Provisioned SageMaker HyperPod Cluster.

Measured idle share of the cluster bill

saving = monthly cluster cost x measured off-hours idle share

There is no flat percentage behind this number. If ZopNight has no idle measurement, there is no figure and no finding.

Scaling the cluster down when nobody is training

1. Confirm with the ML team that no training or inference work is scheduled on the cluster.
2. Review the suggested off-hours window and time zone, then apply it so every instance group scales to 0 nodes off-hours and back before working hours.
3. To do it by hand, set each group’s InstanceCount to 0 with aws sagemaker update-cluster, keeping a note of the original counts.
4. If the cluster is not needed at all, delete it instead to release the capacity and quota.