# AI Gateway

> Govern LLM spend the same way you govern cloud spend; per-team virtual keys with hard budgets, complexity-based routing, and spend by provider/model/team.

Source: https://zop.dev/docs/zopnight/integrations/ai-gateway

---

LLM spend is the newest line item on the cloud bill and the hardest to attribute; one shared API key, no per-team budget, no idea which model is burning the money. The AI Gateway brings the same governance ZopNight applies to cloud spend to your LLM usage: per-team keys with hard budgets, complexity-based routing to cheaper models, and spend broken down by provider, model, tier, and team.

![Cost Reports AI Spend tab showing spend, request count, average latency, and failed requests, plus a spend-over-time chart split by strong and cheap routing tiers](https://storage.googleapis.com/zopdev-blog-resources/1/files/originals/20260707/e0a3d76b-ca25-4b54-bde3-42d917b0a695-aispend.png)

*Cost Reports → AI Spend; month-to-date spend, requests, latency, and failures, with spend split by routing tier.*

**Note**

ZopNight **never sits in the LLM request path**. It stores only a connection record (base URL + keys, encrypted in the Config vault) and reads usage back. The gateway itself is a hosted LiteLLM instance that is the store of record; the whole feature is removable cleanly. There is no standalone "AI Gateway" page; its surfaces live inside the tools that already own them.

## Where the surfaces live

| Task | Where |
|---|---|
| Connect providers, manage models | **Settings → Integrations** |
| Issue and manage virtual keys | **Developer Settings** (beside personal access tokens) |
| See AI spend | **Cost Reports → AI Spend** |
| Set per-provider AI budgets | **Budgets** page (the `ai` scope) |

## Connect a provider

Connect **OpenAI**, **Anthropic**, **OpenRouter**, or **AWS Bedrock** from Settings → Integrations. Connecting one auto-registers a curated cheap + strong model set so it's usable immediately.

You can add **any** model a connected provider offers; the catalog discovers models live (OpenRouter's model list; Bedrock's foundation models and inference profiles via STS from your connected AWS account, no extra key), or add one by id. Each provider has a **Manage models** panel to list, tier, and remove models.

The customer-supplied LiteLLM base URL is SSRF-guarded, and the master key never leaves the Config vault.

## Virtual keys & budgets

- **Per-team virtual keys** with hard USD budgets and per-key model allow-lists. Apps use one endpoint and one key, and switch providers just by changing the model name.
- **Rotate** a key (mints a replacement and revokes the old one, shown once), edit its budget or model list, or revoke it. Every key is org-stamped, so two orgs sharing a fleet never see each other's keys (a foreign key id returns 404).
- **Per-provider AI budgets** on the Budgets page (the `ai` scope) track against live spend, exactly like a cloud budget.

## Routing

A complexity router classifies each prompt to a **cheap** or **strong** tier based on length, max-tokens, and keyword rules; trivial prompts go cheap, harder ones go strong. You can dry-run any prompt to see which tier and model it would pick before wiring it in.

## Cost & usage

**Cost Reports → AI Spend** shows month-to-date spend by provider, model, tier, and team; the percentage of requests served by the cheap tier; average latency; and failed-request counts, plus a per-request log. During a brief gateway outage it degrades to a "stale" banner rather than reporting `$0`.

## Permissions

Prefix-grained policies, each matching its resource:

| Policy | Grants |
|---|---|
| `ai-usage:view` | Read AI spend and usage |
| `virtual-key:create` / `virtual-key:delete` | Manage virtual keys |
| `ai-model:manage` | Manage models and provider connections |

## Availability

The AI Gateway is gated behind a fail-closed flag and rolls out per environment. If you don't see the AI Spend tab or the model-management panels, the feature isn't enabled for your workspace yet; contact support.