HyperPod training clusters whose nodes run outside your VPC
What does ZopNight detect here?
HyperPod clusters move training datasets and checkpoints across their fabric continuously, and without a customer VPC none of that traffic answers to your security groups or endpoints. ZopNight checks both the cluster-level VpcConfig and every instance group's override before firing, and evaluates stopped clusters too. VPC attachment is fixed at creation, so a scaled-to-0 cluster is still non-compliant.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-1627 |
| Category | compliance |
| Severity | medium |
| Metric | none — pure configuration read |
| Source | sagemaker_compliance.go |
Where it applies
Two attachment points, both checked
A HyperPod cluster can be wired into your network at the cluster level (VpcConfig) or per instance group (OverrideVpcConfig). An earlier reading that looked only at the cluster level false-flagged clusters that were correctly attached group by group. The discoverer now considers both: only a cluster with no subnet at either level produces the confirmed-empty signal this rule fires on. That history is worth knowing when auditing a fleet that attaches networking per group.
Why “stopped” clusters still fire
Most compliance rules here skip non-running resources. This one includes clusters scaled to zero, deliberately: VPC attachment is immutable after creation, so a stopped cluster is not “maybe fine when it starts”; it is provably non-compliant capacity that will run outside the VPC the moment someone scales it up. Creating, deleting, and failed clusters are excluded; their lifecycle is the finding there, not their network.
What the training fabric carries
HyperPod nodes stream datasets in, exchange gradients, and write checkpoints out: sustained, high-volume movement of exactly the data an ML organisation most wants governed. Inside a VPC, that flows under your security groups and through your S3 endpoints; outside one, it traverses a SageMaker-managed network your controls never see. For long-lived training infrastructure (as opposed to a disposable notebook), that gap compounds for the cluster’s whole lifetime.
Verify a cluster’s attachment
aws sagemaker describe-cluster --cluster-name my-cluster \ --query '[VpcConfig,InstanceGroups[].[InstanceGroupName,OverrideVpcConfig]]'Null at the cluster level and null in every group reproduces the finding.
Remediation is a rebuild, so plan it as one
No in-place fix exists: the path is a new cluster created with VpcConfig pointing at private subnets and a restrictive security group, VPC endpoints for S3 and the SageMaker control plane, then workload migration and deletion of the old cluster. For a fleet mid-training run that is a scheduling exercise, which is one more reason the finding fires on stopped clusters. The cheapest moment to rebuild is precisely while the cluster is scaled to zero.