Amazon SageMaker HyperPod Cluster
Does ZopNight manage Amazon SageMaker HyperPod Cluster?
SageMaker HyperPod clusters bill per instance-hour across every instance group, and the instances are typically large GPU types, making idle windows extremely expensive. ZopNight discovers clusters on the 6-hour cycle, attributes cost from Cost Explorer or CUR 2.0, and scales all instance groups to zero on schedule via sagemaker:UpdateCluster, restoring saved sizes on start.
Rules that fire on Amazon SageMaker HyperPod Cluster
At a glance
| Field | Value |
|---|---|
| Stop verb | sagemaker:UpdateCluster |
| Scheduling notes | all instance groups scaled to zero via sagemaker:UpdateCluster on stop; saved group sizes are restored on start. |
SageMaker HyperPod provides persistent, resilient clusters for large-scale model training, billed per instance-hour across every instance group. HyperPod clusters typically run large GPU fleets, so idle windows are extremely expensive.
GPU fleets on a persistent meter
HyperPod inverts the usual SageMaker training economics. Ordinary training jobs provision compute per job and release it at completion; a HyperPod cluster is standing infrastructure: instance groups of (typically) high-end GPU instances that bill per instance-hour continuously, job or no job. The persistence is the feature: resilient long-running training with checkpointing, node replacement, and a stable environment. The persistence is also the exposure: at large-scale GPU rates, the gap between training runs is the most expensive idleness in the account, and a cluster held “ready for the next run” bills like the run itself.
Scale-to-zero on a schedule
ZopNight discovers HyperPod clusters automatically on the 6-hour cycle, with per-cluster cost from Cost Explorer or CUR 2.0 and utilization recommendations over its metrics. Clusters are schedulable, and the mechanism matters: on stop, every instance group is scaled to zero via sagemaker:UpdateCluster, with the previous group sizes saved; on start, those sizes are restored. The cluster object, its configuration, and its orchestration survive. What disappears overnight is the GPU instance-hours. For research groups whose training happens in office-hours bursts, this converts the cluster’s idle floor to zero.
Where HyperPod money leaks
Between-run idleness dominates: a fleet provisioned for a two-week pretraining effort held warm for a month of analysis before the next run. Overprovisioned groups follow, with controller and auxiliary instance groups sized like worker groups, billing GPU rates for coordination work. And abandoned clusters close the list: proofs of concept whose teams moved on, still holding accelerated capacity that quota-constrained teams elsewhere are queueing for.
Cluster inventory and group sizes
The SageMaker console’s HyperPod clusters view lists each cluster with its instance groups, types, and counts. Group sizes against the training calendar tell the story. Any hour where the groups are nonzero and no job is running is the number to multiply by the GPU rate.