SageMaker HyperPod clusters whose EBS volumes use only the AWS owned key

Default HyperPod volume encryption and why policies ask for more

HyperPod nodes store the operating system, container images, datasets staged for training and local checkpoints on EBS. The HyperPod customer managed key guide says the root EBS volume is encrypted by default with a KMS key owned by AWS, and that you can instead encrypt both the root and secondary volumes with your own customer managed key.

An AWS owned key keeps the data encrypted, but you cannot see its policy, audit its use or revoke it. Regulated training data, and model weights that are themselves valuable, often require a key the organization controls.

Reading instance group storage settings

aws sagemaker describe-cluster --cluster-name my-cluster \
  --query 'InstanceGroups[].[InstanceGroupName, InstanceStorageConfigs]'

Each EBS entry in InstanceStorageConfigs may carry a VolumeKmsKeyId. If no group shows one, every volume uses the default key. NodeProvisioningMode in the same output tells you whether the cluster uses continuous provisioning.

What is required for the finding

The cluster must be in service or scaled down to zero nodes. ZopNight looks through the EBS storage settings of every instance group for a customer managed key and records the first one it finds. When that record is confirmed empty, meaning no group uses its own key, the cluster is flagged.

Situations that produce no finding

A cluster where any instance group sets a customer managed key passes, even if other groups do not; review the query output if you need every group covered. If the cluster description could not be read, nothing is recorded and no finding is raised.

AWS also limits where keys can be used. Customer managed key encryption is supported only for clusters using continuous node provisioning, restricted instance groups do not support it, and keys cannot yet be set through the console. A cluster outside those limits may be flagged with no in-place remedy.

Key control has a cost, but no saving

ZopNight reports $0. A customer managed key adds KMS key and request charges. The cluster’s execution role also needs kms:CreateGrant and kms:DescribeKey on the key so scaling, node replacement and patching keep working.

Adding a customer managed key

1. Create a symmetric KMS key and allow the cluster execution role to use it, without encryption context conditions, which HyperPod does not pass.
2. KMS key transition is not supported, so the key cannot be changed on an existing group. Add a new instance group whose EBS configuration sets VolumeKmsKeyId through aws sagemaker update-cluster --instance-groups.
3. Move work to the new group, then delete the old group.
4. For clusters not on continuous provisioning, recreate the cluster with the key set from the start.