# SageMaker HyperPod Cluster Volume Not Encrypted With CMK

> Node EBS volumes on a HyperPod cluster hold cached training data and checkpoints for the cluster's life, and an empty kmsKeyId means only an AWS-owned key (no audit trail, no revocation) covers them. Because the setting is fixed at creation, ZopNight flags stopped clusters as well as running ones, so 0 running nodes does not clear the finding.

Source: https://zop.dev/integrations/aws/recommendations/sagemaker-hyperpod-cluster-volume-not-encrypted-with-cmk
Updated: 2026-08-19

---

## Checkpoints are the data that matters here

A training cluster's node volumes are not scratch space. They cache shards of the training
corpus and, more importantly, hold model checkpoints, the intermediate weights that embody
everything the training data taught. Whoever can read a checkpoint effectively holds the
model. Under an AWS-owned key that data is encrypted, but no key you control governs it: no
CloudTrail line per decrypt naming your key, no key policy scoping access, no disable switch.
A customer-managed key in the instance groups' storage config is what converts "encrypted" into
"governed".

## The firing condition and its edges

Discovery stamps the cluster's volume-key state; a present-but-empty `kmsKeyId` is the
confirmed no-CMK signal. The status gate is the distinctive part: in-service *and* stopped
clusters both fire, because EBS encryption is settled at creation. A cluster scaled to zero
is exactly as non-compliant as a running one and cheaper to fix right now. Clusters mid-create,
mid-delete, or failed are excluded, and a failed describe produces no finding at all.

## Inspect the storage configs

```bash
aws sagemaker describe-cluster --cluster-name my-cluster \
  --query 'InstanceGroups[].[InstanceGroupName,InstanceStorageConfigs]'
```

Storage configs without a `VolumeKmsKeyId` reproduce the condition per instance group.

## The remediation trade

The fix is a recreate: new cluster (or replacement instance groups) with `VolumeKmsKeyId` set
in every InstanceStorageConfigs entry, workloads migrated, old cluster deleted. That is real
work on training infrastructure, which pushes toward the honest sequencing: fix it while the
cluster is stopped, and fix the IaC module that creates clusters so the next one is born
compliant. Verify the key policy admits the cluster's execution role and the operators who
restore checkpoints; a key nobody can use fails jobs instead of protecting them.

## What this finding does not claim

No dollar value: a CMK changes the KMS line by roughly $1/month plus request charges, and
encryption posture is not a cost lever. And no claim about data-in-transit; the fabric's
network posture is the VPC rule's subject. This one is narrowly about who controls the key
under the data at rest on the nodes.