Stop SageMaker Notebook Instances Outside Working Hours

A SageMaker notebook instance is an ML compute instance plus an attached ML storage volume. StopNotebookInstance terminates the compute instance and preserves the volume; StartNotebookInstance launches a new compute instance and reattaches that same volume. This page covers classic notebook instances, not SageMaker Studio spaces or apps.

Find the running notebooks

aws sagemaker list-notebook-instances --status-equals InService \
  --query "NotebookInstances[].{name:NotebookInstanceName,type:InstanceType,lcc:NotebookInstanceLifecycleConfigName}" \
  --output table

The lcc column matters later: a lifecycle configuration runs again on every start.

What survives a stop, and what does not

• Files on the ML storage volume survive the stop. Keep notebooks, data and custom conda environments under /home/ec2-user/SageMaker: AWS states that data outside that folder is not persisted during maintenance updates, so treat everything else on the instance as disposable.
• The built-in environments and the sample-notebooks folder are refreshed on every stop and start. Packages you pip install into a built-in environment are gone after the next start. AWS’s guidance is to install custom environments and kernels on the instance’s EBS volume, typically from a lifecycle configuration, so they persist.
• Kernel state and anything in memory are lost, as with any shutdown.

What bills while it is stopped

SageMaker stops charging for the ML compute instance when you call StopNotebookInstance. The ML storage volume is preserved, and SageMaker prices ML storage per GB-month of provisioned storage, so a large volume on a notebook that is stopped most of the time is still a line on the bill. Volumes range from 5 GB to 16,384 GB (default 5 GB). Because they are encrypted, SageMaker cannot see how much of a volume is free, so you can increase a volume’s size but not decrease it; to shrink one, create a new notebook instance. Check the size with:

aws sagemaker describe-notebook-instance --notebook-instance-name ds-sandbox-anna \
  --query "{status:NotebookInstanceStatus,volumeGb:VolumeSizeInGB,type:InstanceType}"

Permissions

sagemaker:ListNotebookInstances, sagemaker:StopNotebookInstance and sagemaker:StartNotebookInstance, plus sagemaker:DescribeNotebookInstance for the checks.

Stop and start from the CLI

aws sagemaker stop-notebook-instance --notebook-instance-name ds-sandbox-anna
aws sagemaker wait notebook-instance-stopped --notebook-instance-name ds-sandbox-anna

aws sagemaker start-notebook-instance --notebook-instance-name ds-sandbox-anna
aws sagemaker wait notebook-instance-in-service --notebook-instance-name ds-sandbox-anna

A stop kills anything running in the kernels, including a training loop someone left going in a cell. Agree a cut-off time with the notebook’s owner, or run long jobs as SageMaker training jobs rather than inside the notebook.

Lifecycle configurations can block the morning start

A lifecycle configuration script runs when the notebook is created and whenever it starts. Scripts cannot run longer than 5 minutes; if one does, it fails and the notebook instance does not start. A script that installs large packages on every start is the usual cause of a scheduled start ending in Failed. Logs are in CloudWatch Logs under the log group /aws/sagemaker/NotebookInstances, stream [notebook-instance-name]/[LifecycleConfigHook].

Verify

After the evening stop, describe-notebook-instance should show Stopped; after the morning start, InService. If the morning status is Failed, read the lifecycle configuration log before anything else.

AWS also recommends restarting notebook instances at least once every 30 days to pick up patches, which a daily schedule does as a side effect.

ZopNight and notebook schedules

SageMaker notebook instances are on ZopNight’s start and stop list. Each notebook can carry its own schedule in the owner’s IANA timezone, and a force-on override with an expiry covers the evening someone needs it running late without editing the schedule. Start and stop require a read and write account connection. Resource-specific notes are on the SageMaker notebook scheduling page, and the notebook resource page lists what ZopNight discovers about each instance.