Scheduling Azure ML Compute Cluster
Can ZopNight schedule Azure ML Compute Cluster?
Azure ML compute cluster schedules set the minimum node count to 0 at stop and restore the saved minimum at start. Idle nodes then deallocate on the cluster's own idle timeout. Running jobs are never killed; the schedule only changes where the autoscaler is allowed to settle.
How the stop works
| Field | Value |
|---|---|
| Behaviour | minimum node count is set to 0 on stop (idle nodes deallocate) and the saved minimum is restored on start. |
Example schedules
- 0 8 * * 1-5 — Business Hours Start: Start at 8:00 AM on weekdays
- 0 18 * * 1-5 — Business Hours Stop: Stop at 6:00 PM on weekdays
- 0 22 * * * — Night Shutdown: Stop at 10:00 PM every day
- 0 6 * * 1-5 — Morning Startup: Start at 6:00 AM on weekdays
- 0 20 * * 5 — Weekend Shutdown: Stop at 8:00 PM on Friday
- 0 7 * * 1 — Weekend Startup: Start at 7:00 AM on Monday
A floor change, not a stop
Nothing here issues a stop verb. The schedule edits one number: the cluster’s minimum node count, set to 0 in the evening and back to its saved value in the morning. The AmlCompute autoscaler does the actual work, deallocating nodes once they have sat idle for the cluster’s configured timeout. Of every mechanism in this directory, this is among the least invasive: the schedule adjusts policy and the platform enforces it.
Jobs are never interrupted
Because the mechanism is a floor and not a kill, a training job running at the stop time simply keeps running; its node deallocates only after the job finishes and the idle timeout passes. Teams nervous about evening cutoffs killing long experiments can relax here. The worst case is a node that bills a few extra hours to finish real work, which is the correct outcome.
The idle timeout sets the lag
The stop takes effect gradually: min drops to 0 at 19:00, but nodes leave only after the idle window (say 120 seconds to 30 minutes, depending on configuration) expires per node. A long timeout delays the saving every single night. If the cluster’s timeout was set generously to keep nodes warm between rapid experiment iterations, consider whether that generosity should apply at midnight.
The restored minimum buys warm mornings
Restoring a nonzero minimum at 7am pre-provisions nodes before anyone submits, converting the first job’s cold-start wait into zero. Clusters whose minimum was already 0 around the clock get nothing from the stop half of this schedule; for them the start-time pre-warm is the entire value, and it is a real one on GPU SKUs with slow allocation.
What deallocation leaves behind
Deallocated nodes stop compute billing. The workspace’s storage account, container registry and any attached datastores bill independently. Nothing about a compute floor touches those meters.