Idle SageMaker HyperPod Cluster, Near-Zero Utilization
What does ZopNight detect here?
HyperPod clusters in service with node CPU averaging under 5%, memory under 10%, and CPU peaks under 20% over 7 days get a scale-to-zero off-hours schedule. ZopNight prices savings as cost times the heatmap-measured idle fraction, abstaining on reservation-covered clusters and on EKS-orchestrated ones, whose node metrics live elsewhere.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-1625 |
| Category | schedule |
| Severity | high |
| Metric | none — pure configuration read |
| Source | sagemaker_cluster_idle.go |
Where it applies
Node CPU and memory under the burst floors
- resource.Type: sagemaker-cluster, Status inservice (a 0-node “stopped” cluster has cost 0 and is owned by RC-045)
- metrics: node_cpu_utilization, node_memory_utilization, the cluster node series SageMaker sends to CloudWatch, keyed by cluster UID (fetched at the CloudWatch ClusterId dimension, stored under the cluster name)
- ScheduleData[uid].IdlePercentage for sagemaker-cluster (heatmap-derived): node_cpu_utilization/node_memory_utilization are now members of the aggregator heatmap generator’s cpuMetricNames/memoryMetricNames (service.go), so IdlePercentage + cost both flow and this path fires concretely for Slurm-orchestrated clusters. Remaining gap: EKS-orchestrated HyperPod clusters publish node metrics to Container Insights / Managed Prometheus, not /aws/sagemaker/Clusters, so they still abstain. both node series present with ≥ MinMaxCoverageDays coverage (telemetry guard) AND CPU average < 5% AND memory average < 10% AND CPU peak < 20% (no bursts) AND a positive monthly cost is available AND the cluster is NOT Savings-Plan/reservation covered AND a heatmap-derived ScheduleData[uid].IdlePercentage > 0 exists.
Scaling instance groups to zero off-hours
Concrete or abstain (cost-driven schedule). The lever is a recurring scale-to-zero off-hours schedule (the HyperPod “off” path scales every instance group to 0 nodes off-hours), not a one-shot stop and not a $0 advisory. savings = cost × ScheduleData[uid].IdlePercentage (the measured off-hours idle fraction, same RC-093/RC-110/RC-1611 path); optimized = cost − savings. When no idle measurement exists the rule abstains and returns nil, rather than falling back on a flat fraction or a $0 rec. Commitment guard: a Savings-Plan/reservation-covered cluster now abstains entirely (scaling it down frees the commitment but does not stop the committed charge, so there is no immediate cash saving to quantify, #96). It no longer flags at medium with zeroed dollars.
Applying the off-hours schedule
- Confirm no training/inference workload is scheduled on this cluster
- Review the recommended off-hours schedule (start/stop cron, timezone)
- Apply it to scale every instance group to 0 nodes during off-hours (the HyperPod off state) and back up before working hours
- If the cluster is no longer needed at all, delete it instead to free the reservation/quota
The two node utilization series
node_cpu_utilization + node_memory_utilization, 7-day lookback