# Scheduling Amazon SageMaker HyperPod Cluster

> HyperPod schedules scale every instance group to zero through sagemaker:UpdateCluster and restore the saved group sizes at start. Scale-in terminates the instances, so local NVMe contents vanish while FSx for Lustre volumes persist and keep billing. Released GPU capacity, especially P5-class, is not guaranteed back at 8am.

Source: https://zop.dev/integrations/aws/sagemaker-cluster/schedule
Updated: 2026-08-19

---

## Instance groups to zero, sizes remembered

The stop issues `sagemaker:UpdateCluster` and walks the cluster's instance groups down to zero,
recording each group's size for the start-time restore. Groups scale independently, so a
cluster mixing a GPU training group with a small CPU controller group has both zeroed and both
restored to their own numbers.

## The capacity gamble is the real risk

Scaling a GPU group to zero releases the accelerators back to the general pool, and nothing
promises they will be available again in the morning. For plentiful instance families this is
theoretical; for the H100-class instances HyperPod clusters typically run, capacity errors at
start time are a live possibility. Teams holding hard-won P5 allocations often decide the
overnight saving is not worth re-entering the capacity queue. That judgment call belongs in
the open before the first scheduled stop, because at several dollars per GPU-hour the stakes
run both directions.

## Checkpoints are mandatory, not best practice

Scale-in terminates instances. Training processes die, and everything on local NVMe dies with
them. A run that checkpoints to FSx or S3 every N steps loses minutes of progress; a run that
kept its state on the local drives loses days. The evening cron effectively audits your
checkpointing discipline, nightly.

## Shared storage persists, and keeps charging

The FSx for Lustre filesystem that HyperPod clusters mount survives the scale-down untouched,
which is exactly what makes morning resumption possible. It also bills for its full provisioned
size and throughput all night. On large filesystems that residual is significant enough to
quote alongside the GPU saving rather than as a footnote.

## Mornings are staged, not instant

Restore brings groups back to size, then instances must provision, run lifecycle scripts,
rejoin the scheduler and remount storage before jobs resume. With scarce capacity the
provisioning step dominates. Give the start schedule a generous head start on the first
training job of the day.