LLM Cost Governance: Explained

LLM cost governance is the practice of controlling spend on large language model providers the same way FinOps controls cloud spend: give each team a budgeted key, route requests to the right-cost model, and make spend visible by provider, model, tier, and team. ZopNight AI Gateway does this as a thin facade over a hosted LiteLLM gateway and never sits in the request path.

This guide keeps the theory short and spends most of its length on what you can actually do. Every recommendation here is one ZopNight can help you execute, starting from a read-only connection.

Connect providers, keep control

Connect OpenAI, Anthropic, OpenRouter, or AWS Bedrock in Settings then Integrations. Connecting one provider auto-registers a curated cheap and strong model set, and you can add any model a connected provider offers. ZopNight stores only a connection record in the vault, so the whole surface is removable with no migrations.

Virtual keys and budgets

Issue per-team virtual keys in Developer Settings, each with a hard USD budget and a model allow-list. Keys are org-stamped so two organizations sharing a fleet never see each other keys, and rotation mints a replacement before revoking the old key. A per-key budget participates in the same alerting as cloud and AI-provider budgets, firing warning and exceeded notifications that name the key.

Routing by complexity

A complexity router serves trivial prompts from a cheap tier and harder or longer prompts from a strong tier. You can dry-run any prompt to see which tier and model it would pick before wiring it into an application.

Spend visibility

Cost Reports then AI Spend shows spend by provider, model, tier, and team, the share of requests served by the cheap tier, average latency, and failed-request counts. On a brief upstream outage it degrades to a stale banner rather than reporting a zero.

Key takeaways

• Treat LLM spend like cloud spend: budgeted keys and clear attribution.
• Per-team virtual keys carry hard USD budgets and model allow-lists.
• A complexity router sends cheap work to the cheap tier.
• ZopNight never sits in the request path and is removable with no migrations.

Where ZopNight fits

ZopNight turns this from reading into doing. It ships 490 built-in audit rules across AWS (216), GCP (127), and Azure (147), 124 of those recommendations are wired to act end to end, 28 one-click and 96 guided, and it starts read-only so you can see the opportunity before you act on any of it. The most direct place to begin is scheduling non-production resources to your working hours, which is covered in the FinOps guide and shown concretely for AWS EC2.

How ZopNight schedules non-production resources

The loop that does this is deliberately mechanical, and it starts read-only. You connect your cloud provider with a read-only role, and ZopNight discovers every non-production resources across your regions and accounts. It records a per-action permission verdict for each one, so you can see where it can list a resource but not yet stop it, and you review that inventory, filter it by status or type, and search for the specific resources you care about before anything is scheduled.

Scheduling itself is a cron you write once in plain terms, stop at 7 PM, start at 8 AM on weekdays, pinned to your timezone so the jobs fire at local business hours rather than UTC. A weekly 24-hour grid shows the schedule visually so you catch gaps and overlaps before you save, and an estimate of active versus inactive hours appears before you commit. Resources attach individually or bundle into groups like “dev-cluster” or “staging-db” so a whole environment follows one cadence.

Actions run in dependency order, so a database comes up before the app server that depends on it. When something needs to stay up, an override forces a non-production resources ON or OFF for a defined window, carries a reason so teammates understand why it exists, and expires automatically so nothing is left running by accident. If a start or stop fails, ZopNight retries up to three times and falls back to a dead-letter queue rather than silently dropping the action, and every state change lands in an audit trail that records whether a schedule, an override, or a specific user triggered it.

Getting started

Getting started is intentionally low-stakes:

• Connect your cloud provider with a read-only role. Nothing is scheduled or changed at this stage.
• Let ZopNight discover your non-production resources and review exactly what it found, filtered by account, region, and status.
• Create a schedule in your timezone and attach the non-production resources or groups you want it to cover.
• Watch the first cycle run, with Slack, Teams, or Google Chat notifications on every start, stop, and failure, then layer in idle cleanup and guided rightsizing.

Production stays excluded by default throughout, and because discovery and recommendations are read-only, you can prove the value before you enable a single action.