SageMaker HyperPod Cluster Non-Production Scheduling Opportunity
What does ZopNight detect here?
Non-production HyperPod clusters without a schedule tag, whose node CPU averages under 50%, waste round-the-clock billing on part-time work. ZopNight prices the off-hours scale-to-zero schedule as cost times the heatmap-measured idle fraction, and the assumed flat 0.67 pattern is removed. EKS-orchestrated clusters abstain, since their metrics live in Container Insights.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-1612 |
| Category | schedule |
| Severity | medium |
| Metric | none — pure configuration read |
| Source | scheduling_sagemaker_cluster.go |
Where it applies
What marks a HyperPod cluster non-production
- resource.Type: sagemaker-cluster
- resource.Status: inservice (discoverer lowercases ClusterStatus)
- resource.Name / tags: dev/test classification: prefers the authoritative env tag (isDevTestEnvTag: env/environment/stage = dev/test/staging) before falling back to a name substring (isDevTestName); a prod env tag / prod name vetoes. Fail-safe: abstain when neither dev/test signal is present.
- tags: absence of any schedule tag key (scheduleTagKeys)
- metric: node_cpu_utilization, reported per cluster node in CloudWatch: the dev/test + schedule-tag-absence gate is corroborated with the authoritative node CPU axis. The rule requires the node CPU series to exist with >= MinMaxCoverageDays coverage AND average below the round-the-clock-busy line (50%) before surfacing the schedule opportunity; a busy or unmonitored cluster abstains rather than firing off the name alone (item PAP004).
- ScheduleData[uid].IdlePercentage (heatmap-derived, RC-093/RC-1611 path) > 0 is required; abstains otherwise. RESOLVED producer gap (same fix as sibling RC-1625): node_cpu_utilization/node_memory_utilization are now members of the aggregator heatmap generator’s cpuMetricNames/memoryMetricNames (service.go), so IdlePercentage flows for sagemaker-cluster and this path fires concretely for Slurm-orchestrated clusters. Remaining gap: EKS-orchestrated HyperPod clusters publish node metrics to Container Insights / Managed Prometheus, not /aws/sagemaker/Clusters, so they still abstain.
Why the measured idle fraction replaced 0.67
concrete-or-abstain (cost-driven schedule). savings = cost × ScheduleData[uid].IdlePercentage (the measured off-hours idle fraction), optimized = cost − savings. The prior flat cost × 0.67 (which assumed a fixed 16h/day + weekend pattern) is removed; the rule abstains (returns nil) when no heatmap-measured idle fraction exists, never emitting a vague advisory or fabricated fraction. Clusters bill for every instance in every instance group continuously; the executor’s aws-sagemaker-cluster provider scales instance groups to zero and restores the saved counts on start. (RemediationKind:“advisory”.)
Scaling instance groups to zero off-hours
- Recommended off-hours window (start/stop cron, timezone) from the heatmap
- Create a schedule in ZopNight to scale all instance groups to zero during off-hours; the executor’s aws-sagemaker-cluster provider restores the saved counts on start
- Automate the scale-down/scale-up on the cron windows above
The node CPU axis used for corroboration
node_cpu_utilization (corroboration axis, harvested for sagemaker-cluster by the aggregator’s sageMakerClusterSpecs; MultiMetricsAware via SetMultiMetrics, 30-day lookback)