What the AI Gateway governs

Outcome

By the end of this lesson, you will be able to state what the AI Gateway controls and what it never touches, locate its surfaces in the product, and explain why it is fully removable.

1. Concept

LLM spend has the shape cloud spend had five years ago: it grows fast, nobody knows whose it is, and the only control is asking people to be careful.

The AI Gateway applies the governance you already have to it: virtual keys, budgets, provider and model management, and spend reporting, built by reusing existing subsystems rather than as a parallel product.

IT REUSES
  Config integrations   provider connections
  Budgets               the `ai` scope
  RBAC                  two new capabilities, same
                        machinery
  Aggregator spend      the same reporting pipeline

The data question, answered first

Every conversation about this feature starts here, so state it before anything else.

ZOPNIGHT NEVER PROCESSES OR STORES LLM REQUEST OR
RESPONSE CONTENT.

Customer completions go to the public gateway endpoint,
an ingress managed outside this repo. ZopNight is not in
the request path for the content.

There is also NO ZopNight-managed prompt logging and no
content inspection. The hosted LiteLLM fleet owns
request and response data.

So the governance is over spend and access, not over prompts. If your security team’s concern is “will our prompts be stored by a cost vendor”, the answer is no, and it is architectural rather than a policy promise.

There is no AI Gateway page

This is a design decision worth understanding, because it tells you where to look.

ITS SURFACES ARE FOLDED INTO THE TOOLS THAT ALREADY OWN
THEM:

  AI spend            -> Cost Reports > AI Spend
  providers + models  -> Settings > Integrations
  virtual keys        -> Developer Settings, beside PATs
  per-provider budget -> the Budgets page, `ai` scope

The reasoning: AI spend is cost, so it belongs on the cost report. A provider connection is an integration, so it belongs with the integrations. A key is a developer credential, so it sits beside the other developer credentials. Giving it its own page would have created a second place to look for things that already have homes.

There are /ai-gateway/* pages, and they exist only to drive a playground demo, fenced behind a flag in both the router and the nav, with a drift test pinning that so the side-tab cannot return by accident.

It is fully removable

ZERO MIGRATIONS.

The hosted fleet is the store of record, so the
capability can be removed cleanly. ZopNight stores no
per-org LLM secret by default: the shared fleet
connection comes from config env, and a per-org
connection record overrides it.

For an architect this is the property that makes adoption low-risk. A feature that writes its own schema is a commitment; one whose state of record lives elsewhere is a trial.

The fail-closed kill switch

THE ENTIRE SURFACE IS HIDDEN UNLESS ENABLED.

  UI       a build-time flag, in both the customer and
           internal apps
  BACKEND  a routes flag, in Config and the Gateway

Both are OFF IN PROD and on in stage, and they flip
together.

Two consequences to plan around: you may not see this in your production tenant yet, and enabling it is a coordinated flip rather than a per-org setting.

The internal admin console does not surface AI today either. Its build leaves the flag unset, so the section is hidden; the read-only proxy and UI gating exist, so enabling the flag is all that would be needed, and writes stay blocked either way.

Who serves it

Config      the admin surface: connections, models, keys
Aggregator  spend

NO SEPARATE SERVICE. This is the same "no new service"
discipline you met with ITSM and IAM import: a capability
is folded into the services whose nature it matches
rather than getting its own.

2. Demo

A security review of the feature, in the order the questions came:

Q1: "Do our prompts go to ZopNight?"
  No. Completions go to the public gateway endpoint,
  which is an ingress managed by DevOps outside this
  product. ZopNight is not in the content path, does no
  prompt logging and does no content inspection.

Q2: "What does ZopNight store, then?"
  Spend and access metadata: which key, which model,
  which team, how much, how long it took, whether it
  failed. Not what was asked or answered.

Q3: "Where is our provider API key?"
  By default, nowhere in ZopNight: the shared fleet
  connection comes from config env. If you supply a
  per-org connection it is stored in the Config vault,
  and the master key never leaves Config.

Q4: "If we adopt this and change our minds?"
  Removable with zero migrations. The hosted fleet is
  the store of record, so there is no ZopNight schema
  to unwind.

Q5: "Can we see it in prod?"
  Not yet. The kill switch is off in prod, on in stage.
  It is a coordinated UI and backend flip, not a per-org
  toggle.

Q6: "Can your internal support staff change our AI
     config?"
  No. The internal console does not surface AI at all
  today, and even with the flag enabled it is read-only:
  writes are blocked either way.

OUTCOME
  Approved for stage evaluation. The two answers that
  carried it were Q1 (architectural, not a policy
  promise) and Q6 (read-only regardless of flag state).

3. Hands-on (6 min)

1. Is the AI Gateway visible in your environment?
   prod:  Y / N        stage:  Y / N
   If not visible in prod, that is expected.

2. Find its four surfaces. Note the path for each:
   AI spend            ____________________________
   providers + models  ____________________________
   virtual keys        ____________________________
   per-provider budget ____________________________

3. Answer the data question in one sentence, as you
   would to your own security team:
   ______________________________________________

4. Where does your provider API key live, in your
   configuration?
   [ ] config env (shared fleet)
   [ ] a per-org connection in the Config vault

5. Note what would have to be unwound if you adopted
   this and later removed it:
   ______________________________________________
   (The answer should be "nothing".)

4. Knowledge check

Q1

A security team asks whether ZopNight stores their LLM prompts. The accurate answer:

A. Only for requests that fail
B. No. What ZopNight stores is spend and access metadata
C. Yes, for the per-request log on the AI Spend page
D. Only when the complexity router is enabled

Q2

Why is there no dedicated AI Gateway page in the product?

A. Its surfaces are folded into the tools that already own them: AI spend into Cost Reports, providers and models into Settings → Integrations, virtual keys into Developer Settings beside PATs, and per-provider budgets onto the Budgets page
B. It is still in development, and the dedicated page is planned for a later release once the individual surfaces have settled down and the navigation has been agreed with the design team internally
C. Because RBAC cannot gate a standalone page, so there would be no way at all to keep the AI settings away from the users who should not be able to see them in the first place, which is why they are spread out
D. Because it is served by two entirely different services, and a single page could not aggregate across both of them without an entirely new aggregation API being built first for that purpose alone

Q3

What makes adopting the AI Gateway a low-risk trial from an architecture standpoint?

A. It can be enabled per-organisation, so that a single team is able to trial it without affecting anyone else at all in the meantime
B. It is fully removable with zero migrations, because the hosted fleet is the store of record and ZopNight stores no per-org LLM secret by default
C. It runs in a completely separate service that can simply be stopped, taking the whole feature offline instantly if needed
D. It only ever reads data and never writes anything at all, so there is nothing that a rollback would ever have to undo afterwards

5. Apply

Answer the data question before anything else when you introduce this internally. “ZopNight is not in the content path” is architectural and it is the sentence that unblocks the conversation.

Then locate the four folded surfaces so you are not looking for a page that does not exist.

Related lessons

• L2: Providers and model registration (next)
• L3: Virtual keys and budgets
• T3.M3.6.L1: Budget vs forecast vs alert
• T6.M6.1.L2: Read by default: the write-tier contract

Glossary terms touched

AI Gateway · Virtual key · Content path · Kill switch