SageMaker endpoint instance volumes lacking a customer-managed key, with NVMe families exempted
What does ZopNight detect here?
Real-time and async SageMaker endpoints cache model artefacts on instance storage volumes, and an empty kmsKeyId means no customer-managed key governs that data. ZopNight deliberately abstains on GPU and NVMe-backed families like g5 and p5, where AWS rejects a KMS key outright. Flagging those would demand an impossible fix on every 1 of them.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-1620 |
| Category | compliance |
| Severity | high |
| Metric | none — pure configuration read |
| Source | sagemaker_compliance.go |
Where it applies
The abstention is the interesting part
Most CMK findings are simple: key empty, flag it. This one carries a carve-out that prevents a whole class of false positives. On instance families with local NVMe storage (g5, g6, p5, trn1, and every family with a “d” attribute marker like m5d or g4dn), the ML storage volume is the instance store, which is hardware-encrypted and cannot accept a KMS key at all: CreateEndpointConfig documents that its key parameter does not cover local storage. A rule that flagged those endpoints would be demanding a remediation AWS rejects. So the rule checks the instance family first and abstains where no fix can exist.
What fires
For EBS-backed instance families, discovery stamps the endpoint config’s kmsKeyId; a
present-but-empty value on an in-service endpoint is the finding. Async inference endpoints
are covered too, since they run the same instance-based storage and carry the identical
config field. Serverless endpoints are excluded because there is no instance volume to
encrypt. Absent metadata abstains, as everywhere in this rule family.
Why the key matters on an inference box
Endpoint volumes hold the decompressed model artefacts and whatever the container caches alongside them. Models are increasingly the crown jewels, because fine-tuned weights embody training data. A customer-managed key is what gives you a per-decrypt audit trail and a revocation lever over that material, rather than encryption that exists but answers to nobody in your account.
Check an endpoint’s config
aws sagemaker describe-endpoint-config \ --endpoint-config-name "$(aws sagemaker describe-endpoint \ --endpoint-name my-endpoint --query EndpointConfigName --output text)" \ --query '[KmsKeyId,ProductionVariants[].InstanceType]'Empty key plus a non-NVMe instance type reproduces the rule’s condition.
Remediating without downtime
Volume encryption is fixed per endpoint config, so the path is a new config with the key set and an endpoint update. SageMaker rolls variants over blue/green, with no client-visible interruption. Confirm the key policy admits the endpoint’s execution role before the update; a key the role cannot use fails the rollout, which is a louder way to discover the same governance gap.