# Idle SageMaker HyperPod Cluster, Near-Zero Utilization

> HyperPod clusters in service with node CPU averaging under 5%, memory under 10%, and CPU peaks under 20% over 7 days get a scale-to-zero off-hours schedule. ZopNight prices savings as cost times the heatmap-measured idle fraction, abstaining on reservation-covered clusters and on EKS-orchestrated ones, whose node metrics live elsewhere.

Source: https://zop.dev/integrations/aws/recommendations/idle-sagemaker-hyperpod-cluster-near-zero-utilization
Updated: 2026-08-19

---

## Node CPU and memory under the burst floors

- resource.Type: sagemaker-cluster, Status inservice (a 0-node "stopped" cluster has cost 0 and is owned by RC-045)
- metrics: node_cpu_utilization, node_memory_utilization, [the cluster node series SageMaker sends to CloudWatch](https://docs.aws.amazon.com/sagemaker/latest/dg/monitoring-cloudwatch.html), keyed by cluster UID (fetched at the CloudWatch ClusterId dimension, stored under the cluster name)
- ScheduleData[uid].IdlePercentage for sagemaker-cluster (heatmap-derived): node_cpu_utilization/node_memory_utilization are now members of the aggregator heatmap generator's cpuMetricNames/memoryMetricNames (service.go), so IdlePercentage + cost both flow and this path fires concretely for Slurm-orchestrated clusters. Remaining gap: EKS-orchestrated HyperPod clusters publish node metrics to Container Insights / Managed Prometheus, not /aws/sagemaker/Clusters, so they still abstain.
both node series present with ≥ MinMaxCoverageDays coverage (telemetry guard) AND CPU average \< 5% AND memory average \< 10% AND CPU peak \< 20% (no bursts) AND a positive monthly cost is available AND the cluster is NOT Savings-Plan/reservation covered AND a heatmap-derived ScheduleData[uid].IdlePercentage > 0 exists.

## Scaling instance groups to zero off-hours

Concrete or abstain (cost-driven schedule). The lever is a recurring scale-to-zero off-hours schedule (the HyperPod "off" path scales every instance group to 0 nodes off-hours), not a one-shot stop and not a $0 advisory. savings = cost × ScheduleData[uid].IdlePercentage (the measured off-hours idle fraction, same RC-093/RC-110/RC-1611 path); optimized = cost − savings. When no idle measurement exists the rule abstains and returns nil, rather than falling back on a flat fraction or a $0 rec. Commitment guard: a Savings-Plan/reservation-covered cluster now abstains entirely (scaling it down frees the commitment but does not stop the committed charge, so there is no immediate cash saving to quantify, #96). It no longer flags at medium with zeroed dollars.

## Applying the off-hours schedule

1. Confirm no training/inference workload is scheduled on this cluster
2. Review the recommended off-hours schedule (start/stop cron, timezone)
3. Apply it to scale every instance group to 0 nodes during off-hours (the HyperPod off state) and back up before working hours
4. If the cluster is no longer needed at all, delete it instead to free the reservation/quota

## The two node utilization series

node_cpu_utilization + node_memory_utilization, 7-day lookback