# SageMaker Training Job, Enable Managed Spot Training

> Completed SageMaker training jobs that ran without managed spot get a forward-looking recommendation worth 70% of the run's actual billed cost, the exact complement of the 0.3 multiplier the aggregator bills managed-spot runs at. AWS publishes no ml.* spot SKU, so the constant pair stays in lockstep by design.

Source: https://zop.dev/integrations/aws/recommendations/sagemaker-training-job-enable-managed-spot-training
Updated: 2026-08-19

---

## Completed training runs that skipped spot

- resource.Type: sagemaker-training-job, Status completed (only a successfully-finished run is a representative savings basis; failed/stopped/in-progress runs reflect a non-representative partial run and are skipped, #1701)
- metadata spotUsed == false (via metadataBool; absent or unparseable fails safe, and only a confirmed false fires)

## Where the managed-spot discount comes from

Concrete, from the system's own authoritative managed-spot discount (concrete-or-abstain). savings = cost × managedSpotDiscountFraction where managedSpotDiscountFraction = 0.70 is the exact complement of the aggregator's billing multiplier sageMakerManagedSpotRateMultiplier = 0.3 (calc_training_job re-bills a managed-spot run at 0.3 of on-demand → 1 − 0.3 = 0.70 saving); the two must stay in lockstep so the projection equals the realized post-change bill. Why a shared constant rather than a derived SKU rate: managed spot is a runtime billing discount, not a published spot SKU. [AWS publishes no SageMaker spot price list](https://aws.amazon.com/sagemaker/pricing/) and deriveSpotEntries is intentionally EC2-only, so no producer ever writes a Spot-tier rate for ml.* instances; deriving from TierRates (the prior DeriveSpotFraction path) therefore always abstained (Spot==0 sentinel) and the rule was dead. The cost arrives via EvaluationContext.Pricing (the aggregator's actual billable-duration run cost, not a 730h projection). The rule abstains (returns nil) when the run cost is absent or non-positive, rather than fabricating a $0 under a cost-driven category. It is advisory, and applies to future runs.

## Turning on managed spot training

1. Set EnableManagedSpotTraining=true in the training job / estimator configuration
2. Set MaxWaitTimeInSeconds >= MaxRuntimeInSeconds to allow time for spot capacity
3. Configure a checkpoint S3 location (CheckpointConfig) so interrupted runs resume instead of restarting
4. Re-run the job; SageMaker reports the realized spot savings in the training job summary

## Why this rule reads metadata, not metrics

None (metadata-driven)