AI Gateway

LLM spend is the newest line item on the cloud bill and the hardest to attribute; one shared API key, no per-team budget, no idea which model is burning the money. The AI Gateway brings the same governance ZopNight applies to cloud spend to your LLM usage: per-team keys with hard budgets, complexity-based routing to cheaper models, and spend broken down by provider, model, tier, and team.

Cost Reports → AI Spend; month-to-date spend, requests, latency, and failures, with spend split by routing tier.

Where the surfaces live

Task	Where
Connect providers, manage models	Settings → Integrations
Issue and manage virtual keys	Developer Settings (beside personal access tokens)
See AI spend	Cost Reports → AI Spend
Set per-provider AI budgets	Budgets page (the ai scope)

Connect a provider

Connect OpenAI, Anthropic, OpenRouter, or AWS Bedrock from Settings → Integrations. Connecting one auto-registers a curated cheap + strong model set so it’s usable immediately.

You can add any model a connected provider offers; the catalog discovers models live (OpenRouter’s model list; Bedrock’s foundation models and inference profiles via STS from your connected AWS account, no extra key), or add one by id. Each provider has a Manage models panel to list, tier, and remove models.

The customer-supplied LiteLLM base URL is SSRF-guarded, and the master key never leaves the Config vault.

Virtual keys & budgets

• Per-team virtual keys with hard USD budgets and per-key model allow-lists. Apps use one endpoint and one key, and switch providers just by changing the model name.
• Rotate a key (mints a replacement and revokes the old one, shown once), edit its budget or model list, or revoke it. Every key is org-stamped, so two orgs sharing a fleet never see each other’s keys (a foreign key id returns 404).
• Per-provider AI budgets on the Budgets page (the ai scope) track against live spend, exactly like a cloud budget.

Routing

A complexity router classifies each prompt to a cheap or strong tier based on length, max-tokens, and keyword rules; trivial prompts go cheap, harder ones go strong. You can dry-run any prompt to see which tier and model it would pick before wiring it in.

Cost & usage

Cost Reports → AI Spend shows month-to-date spend by provider, model, tier, and team; the percentage of requests served by the cheap tier; average latency; and failed-request counts, plus a per-request log. During a brief gateway outage it degrades to a “stale” banner rather than reporting $0.

Permissions

Prefix-grained policies, each matching its resource:

Policy	Grants
ai-usage:view	Read AI spend and usage
virtual-key:create / virtual-key:delete	Manage virtual keys
ai-model:manage	Manage models and provider connections

Availability

The AI Gateway is gated behind a fail-closed flag and rolls out per environment. If you don’t see the AI Spend tab or the model-management panels, the feature isn’t enabled for your workspace yet; contact support.