# SageMaker HyperPod Cluster Non-Production Scheduling Opportunity

> Non-production HyperPod clusters without a schedule tag, whose node CPU averages under 50%, waste round-the-clock billing on part-time work. ZopNight prices the off-hours scale-to-zero schedule as cost times the heatmap-measured idle fraction, and the assumed flat 0.67 pattern is removed. EKS-orchestrated clusters abstain, since their metrics live in Container Insights.

Source: https://zop.dev/integrations/aws/recommendations/sagemaker-hyperpod-cluster-non-production-scheduling-opportunity
Updated: 2026-08-19

---

## What marks a HyperPod cluster non-production

- resource.Type: sagemaker-cluster
- resource.Status: inservice (discoverer lowercases ClusterStatus)
- resource.Name / tags: dev/test classification: prefers the authoritative env tag (isDevTestEnvTag: env/environment/stage = dev/test/staging) before falling back to a name substring (isDevTestName); a prod env tag / prod name vetoes. Fail-safe: abstain when neither dev/test signal is present.
- tags: absence of any schedule tag key (scheduleTagKeys)
- metric: node_cpu_utilization, [reported per cluster node in CloudWatch](https://docs.aws.amazon.com/sagemaker/latest/dg/monitoring-cloudwatch.html): the dev/test + schedule-tag-absence gate is corroborated with the authoritative node CPU axis. The rule requires the node CPU series to exist with >= MinMaxCoverageDays coverage AND average below the round-the-clock-busy line (50%) before surfacing the schedule opportunity; a busy or unmonitored cluster abstains rather than firing off the name alone (item PAP004).
- ScheduleData[uid].IdlePercentage (heatmap-derived, RC-093/RC-1611 path) > 0 is required; abstains otherwise. RESOLVED producer gap (same fix as sibling RC-1625): node_cpu_utilization/node_memory_utilization are now members of the aggregator heatmap generator's cpuMetricNames/memoryMetricNames (service.go), so IdlePercentage flows for sagemaker-cluster and this path fires concretely for Slurm-orchestrated clusters. Remaining gap: EKS-orchestrated HyperPod clusters publish node metrics to Container Insights / Managed Prometheus, not /aws/sagemaker/Clusters, so they still abstain.

## Why the measured idle fraction replaced 0.67

concrete-or-abstain (cost-driven schedule). savings = cost × ScheduleData[uid].IdlePercentage (the measured off-hours idle fraction), optimized = cost − savings. The prior flat cost × 0.67 (which assumed a fixed 16h/day + weekend pattern) is removed; the rule abstains (returns nil) when no heatmap-measured idle fraction exists, never emitting a vague advisory or fabricated fraction. Clusters [bill for every instance in every instance group continuously](https://aws.amazon.com/sagemaker/pricing/); the executor's aws-sagemaker-cluster provider scales instance groups to zero and restores the saved counts on start. (RemediationKind:"advisory".)

## Scaling instance groups to zero off-hours

1. Recommended off-hours window (start/stop cron, timezone) from the heatmap
2. Create a schedule in ZopNight to scale all instance groups to zero during off-hours; the executor's aws-sagemaker-cluster provider restores the saved counts on start
3. Automate the scale-down/scale-up on the cron windows above

## The node CPU axis used for corroboration

node_cpu_utilization (corroboration axis, harvested for sagemaker-cluster by the aggregator's sageMakerClusterSpecs; MultiMetricsAware via SetMultiMetrics, 30-day lookback)