# SageMaker Endpoint Instance Count Over-Provisioned

> SageMaker bills every instance behind an endpoint per hour, so a 4-instance fleet at 3% CPU is mostly paid-for silence. ZopNight recommends shedding exactly one instance, saving cost/N precisely, after normalizing CPU by vCPU count, since the endpoint namespace sums utilization across cores.

Source: https://zop.dev/integrations/aws/recommendations/sagemaker-endpoint-instance-count-over-provisioned
Updated: 2026-08-19

---

## The normalization most dashboards miss

[CPU utilization in the SageMaker endpoint namespace](https://docs.aws.amazon.com/sagemaker/latest/dg/monitoring-cloudwatch.html) is summed across vCPUs: a 4-vCPU
instance reads 0–400%, not 0–100%. Read naively, a fleet running at a tenth of capacity
looks four times busier than it is, and an earlier reading of this data under-fired on
every multi-core family for exactly that reason. The rule resolves the instance type's
published vCPU count and divides before judging. It abstains entirely when the count
cannot be resolved, because an un-normalizable number cannot be compared to a 10%
threshold safely. Memory needs no such treatment; it is genuinely 0–100 in this namespace.

## Who qualifies

In-service endpoints running at least 2 instances on a single production variant.
One-instance endpoints cannot shed capacity without going dark; that is the idle rule's
territory. Multi-variant endpoints (A/B, blue-green, shadow) abstain because one
utilization series cannot describe several independent fleets. The trigger is normalized
CPU under 10%; memory under 50% acts as a veto when its series exists, and since memory
here is an agent-dependent metric, its absence falls back to the CPU-plus-peak gates
rather than silently exempting the fleet. A normalized peak near saturation suppresses
everything: bursty fleets keep their headroom.

## Arithmetic with no estimation in it

Endpoint cost is [the per-instance hourly rate](https://aws.amazon.com/sagemaker/pricing/) × instance count, so dropping one of N saves exactly
cost/N: a fifth of a five-instance fleet's bill, half of a two-instance fleet's. No
utilization-to-dollars model, no fabricated fraction. That is also why the rule abstains
when the endpoint's measured cost is missing.

## Observe a fleet yourself

```bash
aws cloudwatch get-metric-statistics \
  --namespace /aws/sagemaker/Endpoints --metric-name CPUUtilization \
  --dimensions Name=EndpointName,Value=my-endpoint Name=VariantName,Value=AllTraffic \
  --start-time "$(date -u -v-30d +%Y-%m-%dT%H:%M:%SZ)" \
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --period 86400 --statistics Average Maximum
```

Divide the values by the instance's vCPU count before comparing to any threshold.

## Applied by you, not by automation

Capacity on an inference endpoint is an SLA decision involving traffic peaks, latency
budgets, and autoscaling floors, so this recommendation is advisory by design: no automated resize is
offered. Lower the variant's desired instance count (or the autoscaling minimum), watch
latency against SLA on the smaller fleet, and repeat next month if utilization still says
so.