AI Gateway
Connect LLM providers, give each team an inference key with a hard budget, route prompts by complexity, and track AI spend by provider, model, tier, and team.
LLM spend is hard to attribute when every app shares one API key and no team has a budget. The AI Gateway puts one URL in front of OpenAI, Anthropic, OpenRouter, and AWS Bedrock, gives each team its own inference key with a hard budget, routes simple prompts to cheaper models, and breaks spend down by provider, model, tier, and team. For Copilot seats billed per person, see Microsoft 365 Copilot seat licences.
Costs → Reports → AI Spend: total AI cost across inference and seat licences, with inference spend, requests, latency, and failures.
Before you start
- To connect a provider: a role that can create AI models (
ai-model:create). Only the built-in Admin role can; Editor is view-only on models. - To issue inference keys:
virtual-key:create. The built-in Admin and Editor roles have it. - To see spend:
ai-usage:view. - On the provider side: an API key for OpenAI, Anthropic, or OpenRouter. For AWS Bedrock you need no separate key: ZopNight uses a connected AWS cloud account, which must have access to the Bedrock models you plan to use.
How it works
Keys and budgets
- Per-team keys carry a hard USD budget and a per-key model allow-list.
- Budget alerts fire on a key’s own budget the same way cloud budgets do: a warning at 80% and an exceeded alert at 100%, naming the key. They follow whatever budget notification channels you already have configured. Past 100%, the gateway refuses the call, so a runaway job cannot outspend its budget.
- Rotate a key (mints a replacement, shown once with the same use-this-key guide), edit its budget or model list, or revoke it.
- Org isolation. Every key is org-stamped, so two orgs sharing a fleet never see each other’s keys (a foreign key ID returns 404).
Routing
Send a request to the model chat-auto and a complexity router classifies it to the Efficient or Frontier tier based on length, max-tokens, and keyword rules. Trivial prompts go to Efficient and harder ones to Frontier, and an X-ZN-Tier header forces a tier. Dry-run any prompt to see which tier and model it would pick before you wire it in. See AI routing policies for the exact rules.
Cost and usage
AI Estate → AI Gateway (and the same view on Costs → Reports → AI Spend) shows:
- Month-to-date spend by provider, model, tier, and team
- The percentage of requests served by the Efficient tier
- Average latency and failed-request counts
- A per-request log
During a brief gateway outage, the view shows a “stale” banner instead of reporting $0.
The gateway sees only AI traffic you route through it. For Microsoft 365 Copilot, which is billed per seat, see Microsoft 365 Copilot seat licences.
Connect a provider
Open Integrations
Go to Settings → Integrations and choose OpenAI, Anthropic, OpenRouter, or AWS Bedrock.
Enter the credential
Paste the provider API key. Bedrock needs no key: ZopNight uses your connected AWS cloud account. ZopNight verifies the credential against the vendor before storing it, and re-verifies on every update, so a mistyped or revoked key fails at connect rather than turning the tile green and failing on your first inference.
Check the models
Connecting auto-registers a curated set of models for the Efficient and Frontier tiers, so you can use the provider straight away. Open the provider’s Manage models panel to list, tier, and remove models, or add any other model the provider offers.
The catalogue discovers models live (OpenRouter’s model list, or Bedrock’s foundation models and inference profiles via STS from your connected AWS account, with no extra key), or you can add a model by ID. The customer-supplied LiteLLM base URL is SSRF-guarded, and the master key never leaves the vault.
Create an inference key
Inference keys are the virtual keys your apps use to call the gateway. They live on AI Estate → Inference Keys.
AI Estate → Inference Keys: every key in the organisation with its allowed models, budget, spend, and creator.
Create the key
On AI Estate → Inference Keys, choose Create virtual key. Give it an alias, a hard USD budget, and the models it may call.
Copy it once
The one-time reveal, shown when you create a key and again when you rotate one, pairs the key with a use-this-key guide: copy-paste snippets for Claude Code, the OpenAI SDK, and cURL, with the key filled in, and the gateway URL too wherever one is configured. Each snippet pins a model the key is allowed to call, because a key rejects any model outside its allow-list.
Point your app at the gateway
The gateway speaks the OpenAI SDK and the Anthropic Messages API, so most apps need only a new base URL and key. Apps use one endpoint and one key, and switch providers just by changing the model name.
To cap total spend, set an Org budget ceiling from the button on the same page. Per-key budgets still apply underneath it, so a single team cannot spend the whole ceiling. For per-provider budgets, use Costs → Budgets (the ai scope); they track against live spend, exactly like a cloud budget.
Reference
Where the surfaces live
AI has its own section in the sidebar, AI Estate, with two pages: AI Gateway (spend and seat licences) and Inference Keys. The rest lives inside the tools that already own it.
| Task | Where |
|---|---|
| Connect providers, manage models | Settings → Integrations |
| Issue and manage inference keys, set the org budget ceiling | AI Estate → Inference Keys |
| See AI spend and seat licences | AI Estate → AI Gateway (also Costs → Reports → AI Spend) |
| Plan a routing strategy (see AI routing) | Governance → Policies → AI Routing |
| Set per-provider AI budgets | Costs → Budgets (the ai scope) |
| Connect Microsoft 365 Copilot seats | Settings → Integrations → AI Seat Licences (see Microsoft 365) |
Permissions
Prefix-grained policies, each matching its resource:
| Policy | Grants |
|---|---|
ai-usage:view | Read AI spend and usage |
virtual-key:view / create / update / delete | View, create, edit or rotate, and revoke inference keys. Rotating needs update, so a create-only role cannot rotate a key. |
ai-model:view / create / update / delete | View and manage models and provider connections |
| Built-in role | AI access |
|---|---|
| Admin | Full access |
| Editor | Full access to inference keys; view-only on models |
| Viewer | View-only on all AI |
The virtual-key and ai-model policies accept a provider as a third segment (for example virtual-key:create:bedrock), so a custom role can manage only one provider’s keys or models.
Troubleshooting
Connecting a provider fails
ZopNight checks the credential with the vendor before saving it. A mistyped or revoked key fails here by design. For Bedrock, confirm the connected AWS cloud account has access to the models you want.
A request is rejected for its model
A key can call only the models on its allow-list. Add the model to the key on AI Estate → Inference Keys, or send a model name the key already allows.
Calls stop once a key reaches its budget
Past 100% of a key’s budget, the gateway refuses the call. Raise the key’s budget, or check the Org budget ceiling, which caps total spend across all keys.
Someone cannot rotate a key
Rotating needs virtual-key:update. A role with only virtual-key:create can issue keys but not rotate them.
AI spend shows a stale banner
The gateway was briefly unreachable. The view keeps the last figures and marks them stale rather than showing $0.