Idle Azure ML Compute
What does ZopNight detect here?
AmlCompute clusters configured with min_nodes above 0 are flagged when 30 days of Active Nodes data prove no job ran (average and max below 1.0) while Total Nodes confirms the warm floor is allocated. Savings temper to the per- node cost times min_nodes, capped at the cluster's cost.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-223 |
| Category | idle |
| Severity | medium |
| Metric | none — pure configuration read |
| Source | ml_idle.go |
Where it applies
Why job telemetry cannot prove a cluster idle
- resource.Metadata[“min_nodes”]: the AmlCompute cluster’s configured minimum node count (written by the discoverer from the ARM property). Rule fires only when min_nodes > 0.
- Aggregator-harvested Active Nodes / Total Nodes (Azure Monitor Category: Quota, dims Scenario+ClusterName only; continuous cluster-state gauges, already wired for ml-compute-cluster). CpuUtilization (Category: Resource, dims Scenario+runId+NodeId+ClusterName) is per-run/per-node job telemetry, emitted only while a job executes on a node, so a genuinely idle cluster emits ~0 datapoints for it. It can therefore only be used as optional corroboration, never as the required idle proof.
Node state proof, and when it fails closed
fires when min_nodes > 0 AND a trustworthy (≥30d coverage) Active Nodes series proves no node ran a job (windowed avg AND max < 1.0) AND a trustworthy Total Nodes series proves the warm floor is actually allocated/billing (windowed max ≥ 1.0) AND monthly cost > 0. Either series absent or shallow (<30d) => ABSTAIN (fail-closed). A present, trustworthy CpuUtilization series showing a busy hour/baseline (≥5% avg or max) vetoes the fire even when the node-state proof holds; its absence never blocks the fire.Pricing the warm min nodes slice
PricingAware, tempered to the idle-node delta: savings = min((cost / max(CurrentNodeCount,1)) × min_nodes, cost) (temperedMLSavings), i.e. the warm min_nodes slice of the cluster’s per-node rate, capped at current cost.
Setting minimum nodes to zero
- Check Azure Machine Learning studio (ml.azure.com) for active training jobs; note this is the current product, not the retired ML Studio (classic) which sunset 2024-08-31
- If no jobs are running, set minimum nodes to 0
- Go to Azure Machine Learning studio → Compute → Compute clusters → Edit → set Min nodes to 0
- This allows the cluster to scale to zero when idle
Node gauges required, CPU optional
Active Nodes + Total Nodes (both required, <30d coverage = abstain) + CpuUtilization (optional corroborating veto). Gate: min_nodes>0 AND Active Nodes idle (<1.0 avg+max over 30d) AND Total Nodes allocated (≥1.0 max over 30d); min_nodes>0 alone no longer fires; run-rate min_nodes × per-node rate