Outcome
By the end of this lesson, you will be able to explain how the complexity router picks a tier, dry-run a prompt against it, and read the AI Spend surface including its stale-over-zero behaviour.
| Tier | Architect |
| JTBD | ”Cut LLM spend without asking every team to rewrite their prompts, and report on what happened.” |
| Personas | FinOps Lead · Engineering Leader · Platform Engineer |
| Prerequisites | M3.10.L3 |
| Time | 9 minutes |
| Bloom verb | Explain (Understand), Dry-run (Apply), Read (Apply) |
1. Concept
Two things decide what the gateway costs you: which model each request reaches, and how clearly you can see where the money went afterwards.
The complexity router
Most LLM spend goes on prompts that did not need the model they were sent to. The router addresses that without the application changing.
IT CLASSIFIES A PROMPT TO A TIER
CHEAP trivial prompts STRONG long prompts, or ones matching strong-tier keyword rules
INPUTS length max-tokens keyword rulesIt is a classifier, not a model. That is deliberate and it is the property that makes it defensible: the routing decision is inspectable, explainable and stable, where an LLM-based router would be none of those and would itself cost money.
Dry-run before you trust it
ROUTING EVALUATION IS A STATELESS DRY-RUN.
Send a prompt, see which tier and model it WOULD pick.Nothing is called, nothing is billed.This is the mechanism that makes the router adoptable. You can take fifty real prompts from a team’s logs, run them through, and see the split before any traffic moves. A router you cannot inspect is one nobody will enable on production traffic.
Reading AI Spend
Cost Reports > AI Spend
spend by PROVIDER spend by MODEL spend by TIER spend by TEAM % of requests served by the CHEAP tier average LATENCY FAILED-REQUEST counts a per-request logThe two numbers that drive decisions:
% SERVED BY CHEAP The router's effectiveness in one number. A low percentage on a workload you believe is mostly simple means your keyword rules are miscalibrated.
FAILED-REQUEST COUNTS Frequently the first sign of a model allow-list that is too narrow: an application asking for a model its key rejects.Remember from L1 what the per-request log contains: which key, model, team, cost, latency and outcome. Not the prompt and not the completion.
Stale over zero
ON A BRIEF UPSTREAM OUTAGE, THE SURFACE DEGRADES TO A"STALE" PAYLOAD RATHER THAN REPORTING $0.This is a small design decision with a large trust consequence, and it is worth generalising.
A COST SURFACE THAT REPORTS $0 DURING AN OUTAGE ISACTIVELY DANGEROUS. Somebody reads it as "spendstopped", and either relaxes or panics, and both arewrong.
Reporting stale data with a stale label is honest: itsays "this is what we last knew, and we do not knownow".You have met this principle before under different names: concrete-or-abstain in the rules engine, abstention in verification, and no-fire on missing data in watch policies. Same idea, four surfaces: never let missing data render as a confident zero.
What the router does not do
IT DOES NOT ROUTE BY QUALITY OUTCOME. It has nofeedback loop from whether the cheap tier's answer wasgood enough.
So the cheap-tier percentage is a COST metric, not aquality one, and raising it is not automatically animprovement.The honest framing for a team: the router saves money on prompts that were over-served, and it is your evaluation, not the router’s, that establishes which prompts those are. Dry-running real prompts is how you find out before rather than after.
2. Demo
Adopting the router on one team’s traffic:
BASELINE, before routing team payments-api monthly spend $1,140 all traffic strong tier requests ~84,000/mo
STEP 1: DRY-RUN 50 REAL PROMPTS Pulled from the team's own logs.
Router would send: cheap 38 of 50 (76%) strong 12 of 50 (24%)
The team reviewed the 38 by hand. This is the step that matters and the one people skip.
Of the 38: 34 were genuinely trivial (status lookups, classification, short extraction) 4 were NOT: short prompts whose ANSWERS needed reasoning. Length is a proxy for complexity and it is an imperfect one.
STEP 2: TUNE THE KEYWORD RULES Added strong-tier keywords covering the 4 misroutes. Re-ran the dry-run: cheap 34, strong 16. All 34 reviewed as correct.
STEP 3: ENABLE ON REAL TRAFFIC After one month: % served by cheap 71% monthly spend $1,140 -> $412 average latency down (cheap tier is faster) failed requests 0
Saving: $728/mo, ~64%, no application change.
WHAT THE FAILED-REQUEST COUNT CAUGHT LATER Month 2, failed requests jumped to 340.
Cause: the team added a new feature calling a model that was not on their key's allow-list. The key rejected it, exactly as designed.
The AI Spend failed-request count surfaced it within a day. Without that number it would have been a user-reported bug.
THE OUTAGE Month 3, a brief upstream outage. The surface showed STALE rather than $0.
Their FinOps lead noted afterwards that a $0 would have gone into a weekly report as "AI spend dropped to zero", which would have been read as an adoption problem and investigated for a week.3. Hands-on (6 min)
1. Dry-run 10 real prompts from a team's logs. cheap ______ strong ______
2. Review the cheap ones BY HAND. How many would have been under-served? ______ Those are your keyword-rule gaps. Note one: ______________________________________________
3. Open Cost Reports > AI Spend. % served by cheap ______% failed requests this month ______ Is the failure count near zero? Y / N If not, check for a model outside a key's allow-list.
4. Split spend by team. Which team is largest? ______________________________________________ Does that match where you thought the spend was? Y / N
5. Confirm what the per-request log contains, and write the sentence you would give your security team: ______________________________________________4. Knowledge check
Q1
The complexity router classifies prompts using:
A. A small language model that judges difficulty
B. Historical latency per model
C. Length, max-tokens and keyword rules
D. The team’s declared tier preference
Show answer
Correct: C. It is a classifier rather than a model, which makes the routing decision inspectable, explainable and stable, where an LLM-based router would be none of those and would itself cost money. The stateless dry-run follows from this: because the decision is a deterministic classification, you can run real prompts through it and see the split before moving any traffic.
Q2
During a brief upstream outage, the AI Spend surface shows stale data with a stale label rather than $0. Why does this matter?
A. A zero value would break the chart rendering on the AI Spend surface entirely, leaving the viewer with a blank empty panel to puzzle over
B. Because the gateway retries and the data arrives late anyway, so a zero would only ever have been shown briefly in any case
C. Because a $0 figure would trigger the budget alerts on every single key in the organisation all at once, quite needlessly
D. A cost surface reporting $0 during an outage is actively dangerous: somebody reads it as “spend stopped” and either relaxes or panics, and both are wrong
Show answer
Correct: D. Stale data with a stale label says “this is what we last knew, and we do not know now”, which is honest. This is the same principle as concrete-or-abstain in the rules engine, abstention in verification and no-fire on missing data in watch policies: never let missing data render as a confident zero.
Q3
A team’s ”% served by cheap tier” rises from 60% to 85%. This means:
A. Cost went down
B. Quality improved
C. The keyword rules are now correct
D. Latency increased
Show answer
Correct: A. The percentage is a cost metric, not a quality one: the router has no feedback loop from whether the cheap tier’s answer was good enough, so raising it is not automatically an improvement. Establishing which prompts were genuinely over-served is your evaluation, not the router’s. This is why hand-reviewing the dry-run output matters before enabling: in a typical first pass a handful of short prompts turn out to need reasoning, and length is only a proxy for complexity.
5. Apply
Dry-run fifty real prompts from one team before enabling routing on their traffic, and hand-review the cheap-tier results. The misroutes you find become keyword rules, and finding them before the traffic moves is the difference between a cost win and a quality incident.
Watch the failed-request count monthly. It is the cheapest early warning that a key’s allow-list and an application’s model choice have diverged.
Related lessons
- L3: Virtual keys and budgets
- L5: AI RBAC and provider scoping (next)
- L1: What the AI Gateway governs
- T2.M2.11.L3: Model selection trade-offs
Glossary terms touched
Complexity router · Cheap tier · Dry-run evaluation · Stale payload