Outcome
By the end of this lesson, you will be able to state what the AI Gateway controls and what it never touches, locate its surfaces in the product, and explain why it is fully removable.
| Prerequisites | M4.6 (budget governance) |
| Time | 9 minutes |
1. Concept
LLM spend has the shape cloud spend had five years ago: it grows fast, nobody knows whose it is, and the only control is asking people to be careful.
The AI Gateway applies the governance you already have to it: virtual keys, budgets, provider and model management, and spend reporting, built by reusing existing subsystems rather than as a parallel product.
IT REUSES Config integrations provider connections Budgets the `ai` scope RBAC two new capabilities, same machinery Aggregator spend the same reporting pipelineThe data question, answered first
Every conversation about this feature starts here, so state it before anything else.
ZOPNIGHT NEVER PROCESSES OR STORES LLM REQUEST ORRESPONSE CONTENT.
Customer completions go to the public gateway endpoint,an ingress managed outside this repo. ZopNight is not inthe request path for the content.
There is also NO ZopNight-managed prompt logging and nocontent inspection. The hosted LiteLLM fleet ownsrequest and response data.So the governance is over spend and access, not over prompts. If your security team’s concern is “will our prompts be stored by a cost vendor”, the answer is no, and it is architectural rather than a policy promise.
Where it lives
AI has its own section in the navigation, AI Estate, and a few surfaces stay with the tools that already own them. Knowing the split tells you where to look.
AI ESTATE SECTION: AI spend -> AI Estate > AI Gateway (/ai-agents/usage) virtual keys -> AI Estate > Inference Keys (/ai-agents/keys)
KEPT WITH THEIR EXISTING HOMES: AI spend (again) -> Cost Reports > AI Spend providers + models -> Settings > Integrations routing policy -> Governance > Policies > AI Routing per-provider budget -> the Budgets page, `ai` scopeThe reasoning: AI spend is cost, so it also stays on the cost report, where every cost category lives. A provider connection is an integration, so it stays with the integrations. Keys moved out of Developer Settings into Inference Keys. The whole section sits behind the same flag as the rest of the AI capability, and the old /ai-gateway/* paths now redirect into it.
It is fully removable
NO AI GATEWAY TABLES TO DROP.
The hosted fleet is the store of record, so thecapability can be removed cleanly. Per-org connectionrecords and AI budgets are ordinary rows you candelete. ZopNight stores no per-org LLM secret bydefault: the shared fleetconnection comes from config env, and a per-orgconnection record overrides it.For an architect this is the property that makes adoption low-risk. A feature that writes its own schema is a commitment; one whose state of record lives elsewhere is a trial.
The fail-closed kill switch
THE ENTIRE SURFACE IS HIDDEN UNLESS ENABLED.
UI a build-time flag, in both the customer and internal apps BACKEND a routes flag, in Config and the Gateway
Both are OFF IN PROD and on in stage, and they fliptogether.Two consequences to plan around: you may not see this in your production tenant yet, and enabling it is a coordinated flip rather than a per-org setting.
The internal admin console does not surface AI today either. Its build leaves the flag unset, so the section is hidden; the read-only proxy and UI gating exist, so enabling the flag is all that would be needed, and writes stay blocked either way.
Who serves it
Config the admin surface: connections, models, keysAggregator spend
NO SEPARATE SERVICE. This is the same "no new service"discipline you met with ITSM and IAM import: a capabilityis folded into the services whose nature it matchesrather than getting its own.2. Demo
A security review of the feature, in the order the questions came:
Q1: "Do our prompts go to ZopNight?" No. Completions go to the public gateway endpoint, which is an ingress managed by DevOps outside this product. ZopNight is not in the content path, does no prompt logging and does no content inspection.
Q2: "What does ZopNight store, then?" Spend and access metadata: which key, which model, which team, how much, how long it took, whether it failed. Not what was asked or answered.
Q3: "Where is our provider API key?" By default, nowhere in ZopNight: the shared fleet connection comes from config env. If you supply a per-org connection it is stored in the Config vault, and the master key never leaves Config.
Q4: "If we adopt this and change our minds?" Removable cleanly. There are no AI Gateway tables to drop: the hosted fleet is the store of record, and per-org connection records and AI budgets are ordinary rows you can delete.
Q5: "Can we see it in prod?" Not yet. The kill switch is off in prod, on in stage. It is a coordinated UI and backend flip, not a per-org toggle.
Q6: "Can your internal support staff change our AI config?" No. The internal console does not surface AI at all today, and even with the flag enabled it is read-only: writes are blocked either way.
OUTCOME Approved for stage evaluation. The two answers that carried it were Q1 (architectural, not a policy promise) and Q6 (read-only regardless of flag state).3. Hands-on (6 min)
1. Is the AI Gateway visible in your environment? prod: Y / N stage: Y / N If not visible in prod, that is expected.
2. Find its four surfaces. Note the path for each: AI spend ____________________________ providers + models ____________________________ virtual keys ____________________________ per-provider budget ____________________________
3. Answer the data question in one sentence, as you would to your own security team: ______________________________________________
4. Where does your provider API key live, in your configuration? [ ] config env (shared fleet) [ ] a per-org connection in the Config vault
5. Note what would have to be unwound if you adopted this and later removed it: ______________________________________________ (The answer should be "only ordinary rows: the per-org connection record and any AI budgets".)4. Knowledge check
Q1
A security team asks whether ZopNight stores their LLM prompts. The accurate answer:
A. Only for requests that fail
B. No. What ZopNight stores is spend and access metadata
C. Yes, for the per-request log on the AI Spend page
D. Only when the complexity router is enabled
Show answer
Correct: B. Completions go to the public gateway endpoint, an ingress managed outside this product, so ZopNight is not in the content path. There is no ZopNight-managed prompt logging and no content inspection; the hosted LiteLLM fleet owns request and response data. C is the tempting wrong answer: there is a per-request log, and it carries which key, model, team, cost, latency and success, not the prompt or the completion. The distinction between metadata and content is the whole answer.
Q2
Where do the AI Gateway’s surfaces live in the product?
A. An AI Estate section for spend and keys, plus Cost Reports, Settings → Integrations and the Budgets page
B. Only on a playground demo page that is fenced behind a flag and never reachable from a customer’s own production tenant at all
C. Only in Developer Settings, beside personal access tokens, with no AI spend shown on the cost reports or budget pages either
D. Only in the internal admin console run by ZopNight staff, so a customer has to raise a ticket to see their own AI spend
Show answer
Correct: A. AI Estate holds AI Gateway (spend) and Inference Keys, which moved out of Developer Settings. AI spend also stays on Cost Reports, because no cost category may be missing from Cost. The old /ai-gateway/* paths now redirect into the AI Estate section.
Q3
What makes adopting the AI Gateway a low-risk trial from an architecture standpoint?
A. It can be enabled per-organisation, so that a single team is able to trial it without affecting anyone else at all in the meantime
B. It is removable cleanly with no AI Gateway tables to drop, because the hosted fleet is the store of record and ZopNight stores no per-org LLM secret by default
C. It runs in a completely separate service that can simply be stopped, taking the whole feature offline instantly if needed
D. It only ever reads data and never writes anything at all, so there is nothing that a rollback would ever have to undo afterwards
Show answer
Correct: B. A feature that writes its own schema is a commitment; one whose state of record lives elsewhere is not. A is specifically wrong: the kill switch is a coordinated UI and backend flip, off in prod and on in stage, rather than a per-org setting. C is also wrong: there is no separate service, it is Config plus Aggregator.
5. Apply
Answer the data question before anything else when you introduce this internally. “ZopNight is not in the content path” is architectural and it is the sentence that unblocks the conversation.
Then locate the four folded surfaces so you are not looking for a page that does not exist.
Related lessons
- L2: Providers and model registration (next)
- L3: Virtual keys and budgets
- T4.M4.6.L1: Budget vs forecast vs alert
- T6.M6.1.L2: Read by default: the write-tier contract
Glossary terms touched
AI Gateway · Virtual key · Kill switch