SageMaker Training Job, Enable Managed Spot Training
What does ZopNight detect here?
Completed SageMaker training jobs that ran without managed spot get a forward-looking recommendation worth 70% of the run's actual billed cost, the exact complement of the 0.3 multiplier the aggregator bills managed-spot runs at. AWS publishes no ml.* spot SKU, so the constant pair stays in lockstep by design.
Signal and threshold
| Field | Value |
|---|---|
| Rule IDs | RC-1621 |
| Category | discount |
| Severity | low |
| Metric | none — pure configuration read |
| Source | sagemaker_managed_spot.go |
Where it applies
Completed training runs that skipped spot
- resource.Type: sagemaker-training-job, Status completed (only a successfully-finished run is a representative savings basis; failed/stopped/in-progress runs reflect a non-representative partial run and are skipped, #1701)
- metadata spotUsed == false (via metadataBool; absent or unparseable fails safe, and only a confirmed false fires)
Where the managed-spot discount comes from
Concrete, from the system’s own authoritative managed-spot discount (concrete-or-abstain). savings = cost × managedSpotDiscountFraction where managedSpotDiscountFraction = 0.70 is the exact complement of the aggregator’s billing multiplier sageMakerManagedSpotRateMultiplier = 0.3 (calc_training_job re-bills a managed-spot run at 0.3 of on-demand → 1 − 0.3 = 0.70 saving); the two must stay in lockstep so the projection equals the realized post-change bill. Why a shared constant rather than a derived SKU rate: managed spot is a runtime billing discount, not a published spot SKU. AWS publishes no SageMaker spot price list and deriveSpotEntries is intentionally EC2-only, so no producer ever writes a Spot-tier rate for ml.* instances; deriving from TierRates (the prior DeriveSpotFraction path) therefore always abstained (Spot==0 sentinel) and the rule was dead. The cost arrives via EvaluationContext.Pricing (the aggregator’s actual billable-duration run cost, not a 730h projection). The rule abstains (returns nil) when the run cost is absent or non-positive, rather than fabricating a $0 under a cost-driven category. It is advisory, and applies to future runs.
Turning on managed spot training
- Set EnableManagedSpotTraining=true in the training job / estimator configuration
- Set MaxWaitTimeInSeconds >= MaxRuntimeInSeconds to allow time for spot capacity
- Configure a checkpoint S3 location (CheckpointConfig) so interrupted runs resume instead of restarting
- Re-run the job; SageMaker reports the realized spot savings in the training job summary
Why this rule reads metadata, not metrics
None (metadata-driven)