LLM Cost Governance: Batch, Caching, Routing and Budgets

LLM cost governance is the practice of making spend on language models visible, attributable, and bounded, the same way FinOps treats cloud infrastructure. It covers two different kinds of AI spend: usage-priced API calls, billed per token by providers such as OpenAI, Anthropic, and AWS Bedrock, and seat-priced assistants, billed per licensed user whether or not that user touches it. The levers for each are different.

The price levers providers publish

Token prices are only the starting point. Several providers publish mechanisms that change the effective rate for the same model:

• Batch processing. OpenAI’s Batch API offers a 50 percent discount compared with its synchronous APIs, with a 24-hour turnaround. Anthropic’s Message Batches API charges all usage at 50 percent of standard API prices. Amazon Bedrock offers batch inference for select foundation models at a 50 percent lower price than on-demand inference. Any workload that does not need an answer immediately, such as evaluations, classification backfills, or document enrichment, belongs here.
• Prompt caching. Anthropic prices 5-minute cache writes at 1.25 times the base input token price and cache reads at 0.1 times on most models (a few newer models price reads lower still), with a default 5-minute cache lifetime that refreshes each time the cached content is used. Bedrock also supports prompt caching to reduce input token cost and latency. Long, stable system prompts and shared document context are the candidates.
• Committed capacity. Bedrock Provisioned Throughput is billed hourly per model unit, with no-commitment, 1-month, and 6-month terms; longer terms are cheaper per hour, but a committed Provisioned Throughput cannot be deleted before its term ends, and billing continues until it is deleted.
• Routing to a smaller model. Bedrock’s intelligent prompt routing sends each request to a model within one model family, based on predicted response quality, to balance quality and cost.

Attribution comes before optimization

None of these levers can be prioritized without knowing which team or feature generates the tokens. A provider bill aggregates by account. The practical fix is a gateway that issues each consumer its own key, so spend is recorded per key, and a budget per key so that one runaway job cannot consume the month.

ZopNight’s AI Gateway capability, released behind a feature flag, is built for this. Per-team virtual keys carry hard USD budgets and model allow-lists, a complexity router serves simple prompts from a cheap tier and longer or keyword-matched prompts from a strong tier, and Cost Reports shows AI spend by provider, model, tier, and team, the share of requests served by the cheap tier, average latency, and failed-request counts. Per-provider AI budgets sit on the same Budgets page as cloud budgets.

Seat licences are a separate problem

Seat-priced AI never passes through a gateway, so its waste is unused seats rather than expensive tokens. ZopNight’s AI seat licences feature starts with Microsoft 365 Copilot: it compares purchased, assigned, and active seats and shows what each gap costs, using read-only Microsoft Graph permissions the customer’s admin consents to. Microsoft states its Copilot usage report typically becomes available within 48 hours of the end of a day, so “inactive” should be judged over weeks, not days. The strongest outcome is a reclaim candidate; nothing in the product revokes a licence.

A governance checklist

1. Issue a key per team and environment; no shared provider keys.
2. Move non-interactive workloads to batch pricing.
3. Cache stable prompt prefixes.
4. Buy committed capacity only for measured, steady throughput, and remember the term locks it in.
5. Review seat utilization monthly against the vendor’s own usage report.

Related: AI gateways and the AWS Bedrock cost guide.