HyperPod node volumes encrypted only with an AWS-owned key
What does ZopNight detect here?
Node EBS volumes on a HyperPod cluster hold cached training data and checkpoints for the cluster's life, and an empty kmsKeyId means only an AWS-owned key (no audit trail, no revocation) covers them. Because the setting is fixed at creation, ZopNight flags stopped clusters as well as running ones, so 0 running nodes does not clear the finding.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-1628 |
| Category | compliance |
| Severity | high |
| Metric | none — pure configuration read |
| Source | sagemaker_compliance.go |
Where it applies
Checkpoints are the data that matters here
A training cluster’s node volumes are not scratch space. They cache shards of the training corpus and, more importantly, hold model checkpoints, the intermediate weights that embody everything the training data taught. Whoever can read a checkpoint effectively holds the model. Under an AWS-owned key that data is encrypted, but no key you control governs it: no CloudTrail line per decrypt naming your key, no key policy scoping access, no disable switch. A customer-managed key in the instance groups’ storage config is what converts “encrypted” into “governed”.
The firing condition and its edges
Discovery stamps the cluster’s volume-key state; a present-but-empty kmsKeyId is the
confirmed no-CMK signal. The status gate is the distinctive part: in-service and stopped
clusters both fire, because EBS encryption is settled at creation. A cluster scaled to zero
is exactly as non-compliant as a running one and cheaper to fix right now. Clusters mid-create,
mid-delete, or failed are excluded, and a failed describe produces no finding at all.
Inspect the storage configs
aws sagemaker describe-cluster --cluster-name my-cluster \ --query 'InstanceGroups[].[InstanceGroupName,InstanceStorageConfigs]'Storage configs without a VolumeKmsKeyId reproduce the condition per instance group.
The remediation trade
The fix is a recreate: new cluster (or replacement instance groups) with VolumeKmsKeyId set
in every InstanceStorageConfigs entry, workloads migrated, old cluster deleted. That is real
work on training infrastructure, which pushes toward the honest sequencing: fix it while the
cluster is stopped, and fix the IaC module that creates clusters so the next one is born
compliant. Verify the key policy admits the cluster’s execution role and the operators who
restore checkpoints; a key nobody can use fails jobs instead of protecting them.
What this finding does not claim
No dollar value: a CMK changes the KMS line by roughly $1/month plus request charges, and encryption posture is not a cost lever. And no claim about data-in-transit; the fabric’s network posture is the VPC rule’s subject. This one is narrowly about who controls the key under the data at rest on the nodes.