AWS X-Ray sampling rules with a fixed rate above 10% and no reservoir, recording far more traces than the 5% default

Trace volume is the X-Ray bill

X-Ray charges for traces recorded. AWS’s price list for US East (N. Virginia) shows $0.000005 per trace stored, which is $5 per million, after a free tier of 100,000 traces a month, with traces retrieved or scanned charged separately. See X-Ray pricing.

Sampling decides how many requests become traces. The sampling documentation describes the default: the SDK records the first request each second, the reservoir, and five percent of any additional requests. A rule raised to 50% or 100% during an incident and never lowered multiplies the bill by ten or twenty for a busy service.

Reviewing your sampling rules

aws xray get-sampling-rules \
  --query 'SamplingRuleRecords[].SamplingRule.[RuleName,FixedRate,ReservoirSize,Priority]' \
  --output table

aws xray get-sampling-statistic-summaries

FixedRate is a fraction, so 0.5 means 50%. The statistic summaries show how many requests each rule has sampled recently.

Conditions for the finding

• ZopNight has marked the Region’s sampling as high: a sampling rule has a fixed rate above 10% and a reservoir size of 0. A rule with any reservoir is not flagged, whatever its fixed rate. The rate must also be above the 5% target, or there is nothing to reduce.
• ZopNight has a recorded-trace series for the Region covering at least 30 days, with positive volume.

When the rule stays quiet

Without trace-volume data there is no finding, because ZopNight will not guess at traffic. That data comes from the account’s default X-Ray group; if group metrics are not available, the series is empty. A rate at or below 10%, or a rule with a reservoir above 0, is left alone. The rule never estimates the saving as a flat share of spend.

Savings from a 5% rate

reduced traces = current monthly traces x (0.05 / current fixed rate)
saving         = (current traces - reduced traces) x $5 / 1,000,000

The current traces are the Region’s whole recorded volume and the current fixed rate is that of the highest flagged rule, so the formula assumes every trace comes from that rule. With several rules, or services already sampling at 5%, the real saving is lower. The figure is also before the 100,000-trace free tier, so small accounts will save slightly less than shown.

Lowering the sampling rate

1. Keep a separate high-priority rule that samples errors or a critical path at a higher rate, if the team depends on it.
2. Lower the broad rule to 5% with a reservoir of 1 request a second, using the rule name from the listing above: aws xray update-sampling-rule --sampling-rule-update RuleName=my-broad-rule,FixedRate=0.05,ReservoirSize=1
3. Check recorded-trace volume over the next day and adjust.