HyperPod clusters running more nodes than their workload uses

Why the CPU band starts at 5%

A HyperPod cluster whose nodes average under 5% CPU is not over-provisioned. It is idle, and a separate rule owns that case with a teardown recommendation. This rule begins where that one stops: node CPU averaging at least 5% but under 10%, with node memory under 50%, sustained over a 30-day lookback. The floor makes the two rules disjoint by construction, so the same monthly reading can never produce both a “delete it” card and a “shrink it” card for one cluster.

Two nodes minimum, by definition

Shedding a node only makes sense if a node remains. A single-node cluster cannot get smaller and keep running, so the rule never fires below 2 nodes. A node count of zero or unset means discovery could not size the cluster; that also lands below the floor and skips, because the rule refuses to claim a saving on a fleet whose real size it does not know.

The cost/N figure is deliberately approximate

HyperPod bills per node instance-hour, so removing one of N nodes returns roughly one Nth of the monthly cost. Roughly, because clusters can mix instance groups with different instance types: cost/N prices the removal of one average node, and the exact figure depends on which group you actually scale down. The recommendation states this caveat rather than presenting the estimate as exact.

Peak, coverage, and pricing abstains

Both of the cluster’s node utilisation series (node_cpu_utilization and node_memory_utilization) must exist for the cluster with enough coverage days; a missing or thin series produces no finding. The trusted CPU peak must also sit below the safe ceiling, because a fleet that bursts near saturation would be starved if shrunk, however low its average. And when no positive monthly cost is available from pricing data, the rule emits nothing rather than a zero-dollar card.

Sizing the instance groups yourself

aws sagemaker describe-cluster --cluster-name my-hyperpod \
  --query 'InstanceGroups[].[InstanceGroupName,InstanceType,CurrentCount,TargetCount]' \
  --output table

Compare each group’s node count against its per-group utilisation in CloudWatch before deciding where the surplus lives.

Scaling down without hurting training

HyperPod has no in-place resize, so the finding is advisory: you apply it by lowering an instance group’s count through UpdateCluster. Pick the group whose nodes are actually idle, not the most expensive one (they are often different groups), and watch training throughput and queue wait on the smaller fleet before shrinking again.