Scheduling Amazon SageMaker HyperPod Cluster
sagemaker:UpdateClusterCan ZopNight schedule Amazon SageMaker HyperPod Cluster?
HyperPod schedules scale every instance group to zero through sagemaker:UpdateCluster and restore the saved group sizes at start. Scale-in terminates the instances, so local NVMe contents vanish while FSx for Lustre volumes persist and keep billing. Released GPU capacity, especially P5-class, is not guaranteed back at 8am.
How the stop works
| Field | Value |
|---|---|
| Behaviour | all instance groups scaled to zero via sagemaker:UpdateCluster on stop; saved group sizes are restored on start. |
Example schedules
- 0 8 * * 1-5 — Business Hours Start: Start at 8:00 AM on weekdays
- 0 18 * * 1-5 — Business Hours Stop: Stop at 6:00 PM on weekdays
- 0 22 * * * — Night Shutdown: Stop at 10:00 PM every day
- 0 6 * * 1-5 — Morning Startup: Start at 6:00 AM on weekdays
- 0 20 * * 5 — Weekend Shutdown: Stop at 8:00 PM on Friday
- 0 7 * * 1 — Weekend Startup: Start at 7:00 AM on Monday
Instance groups to zero, sizes remembered
The stop issues sagemaker:UpdateCluster and walks the cluster’s instance groups down to zero,
recording each group’s size for the start-time restore. Groups scale independently, so a
cluster mixing a GPU training group with a small CPU controller group has both zeroed and both
restored to their own numbers.
The capacity gamble is the real risk
Scaling a GPU group to zero releases the accelerators back to the general pool, and nothing promises they will be available again in the morning. For plentiful instance families this is theoretical; for the H100-class instances HyperPod clusters typically run, capacity errors at start time are a live possibility. Teams holding hard-won P5 allocations often decide the overnight saving is not worth re-entering the capacity queue. That judgment call belongs in the open before the first scheduled stop, because at several dollars per GPU-hour the stakes run both directions.
Checkpoints are mandatory, not best practice
Scale-in terminates instances. Training processes die, and everything on local NVMe dies with them. A run that checkpoints to FSx or S3 every N steps loses minutes of progress; a run that kept its state on the local drives loses days. The evening cron effectively audits your checkpointing discipline, nightly.
Shared storage persists, and keeps charging
The FSx for Lustre filesystem that HyperPod clusters mount survives the scale-down untouched, which is exactly what makes morning resumption possible. It also bills for its full provisioned size and throughput all night. On large filesystems that residual is significant enough to quote alongside the GPU saving rather than as a footnote.
Mornings are staged, not instant
Restore brings groups back to size, then instances must provision, run lifecycle scripts, rejoin the scheduler and remount storage before jobs resume. With scarce capacity the provisioning step dominates. Give the start schedule a generous head start on the first training job of the day.