Providers and model registration

Outcome

By the end of this lesson, you will be able to connect an AI provider and register models, explain why credentials are verified at connect time, and describe how tenant isolation is enforced on a shared fleet.

1. Concept

A provider is the company whose models you are paying for: OpenAI, Anthropic, Google, AWS Bedrock. You connect each one once, register the models you want teams to reach, and the gateway handles everything after that.

Connecting a provider

SUPPORTED
  OpenAI
  Anthropic
  OpenRouter
  AWS Bedrock

Connecting one AUTO-REGISTERS a curated cheap + strong
set, so the provider is usable immediately rather than
requiring a model-registration step before anything
works.

That auto-registration is a small thing with a real effect: the most common failure of a gateway feature is a connected provider with no models registered, which looks connected and serves nothing.

The credential is verified against the vendor

CONNECTING VERIFIES THE CREDENTIAL AGAINST THE VENDOR
BEFORE STORING IT.

So a revoked or mistyped key fails AT CONNECT, rather
than turning the tile green and failing on the
customer's first inference.

Re-verified on every credential update.

This is the same discipline as cloud-account connection (T1.M1.1): a successful connect is itself the proof. The alternative is a tile that says Connected and a team discovering otherwise in production.

Adding any model the provider offers

Beyond the curated set, you can add any model a connected provider offers, through a provider-agnostic catalog that discovers models live.

OpenRouter   its /models endpoint
Bedrock      ListFoundationModels plus inference
             profiles, via STS minted from the org's
             CONNECTED AWS ACCOUNT
             -> no extra key needed
Others       interface-ready

Or add one by id, if you know it.

The Bedrock path is worth noticing as an architect: it reuses the AWS account you already connected for cost management, so enabling Bedrock models does not mean handing over another credential.

Each connected model-supporting provider gets a Manage models panel: list, tier, remove. A deployment is removed by its opaque id, never by a customer-controlled name, which closes an obvious class of mistake where a rename makes a delete target the wrong thing.

Tenant isolation, and why it works the way it does

This is the part an architect should understand properly, because the mechanism is non-obvious and the reasoning is empirical.

PHASE 1a SHARES ONE HOSTED FLEET ACROSS ORGS.

So "how do I know another tenant's traffic cannot reach
our vendor credential" is the right question to ask.

The naive answer would be: each virtual key has a model allow-list, so a key restricted to one model can only use that model. That answer is wrong, and it was measured to be wrong:

MEASURED ON LITELLM OSS

  Two deployments registered under the SAME model_name
  with DIFFERENT vendor keys form a LOAD-BALANCING POOL.

  A virtual key restricted to models:["gpt-4o"] had
  5 OF 14 REQUESTS served by the OTHER org's vendor
  credential.

  A key's `models` list gates which NAME may be called.
  It never gates WHICH DEPLOYMENT ANSWERS.
  The org stamp on the deployment is descriptive only.

Team-scoped deployments would fix it, and that is an Enterprise-only feature on LiteLLM, so it was not available.

THE MECHANISM THAT DOES WORK

  Each deployment is registered as {orgID}/{logicalName},
  so no two orgs share a model_name and no pool can form.

  Each virtual key carries an ALIAS mapping the friendly
  name onto that org's deployment.

  LiteLLM applies the alias AFTER the key's model check,
  so the customer still sends model="gpt-4o" and reaches
  only their own deployment.

  RE-MEASURED with this shape: 16/16 correct for one org,
  10/10 for a mirror org, and a key WITHOUT an alias
  resolves nothing.

So tenant isolation is enforced at the model name, because on LiteLLM OSS that is the only boundary that actually holds. Reads stay filtered on the org stamp.

The lesson generalises beyond this feature: a boundary that looks like it should work is not a boundary until someone measures it. The 5-of-14 number is why this design is the shape it is.

The customer-supplied base URL

If you supply your own LiteLLM base URL, it is SSRF-guarded, and the master key never leaves Config, the vault-owning service.

Phase 1a does registration-time validation only, which is honest about what it is: not DNS-rebinding-proof, and acceptable while the URL is ZopNight-owned. Worth knowing if you plan to point it at your own fleet.

What Phase 1a is not

NOT per-org isolated LiteLLM fleets. One shared
ZN-hosted fleet, provisioned from config env, with a
per-org connection record able to override it. Per-org
customer-hosted fleets are Phase 1b.

2. Demo

Connecting two providers, and the isolation question in a review:

CONNECT 1: Anthropic
  Pasted an API key. Verification called the vendor.
  First attempt FAILED: the key had been rotated the
  week before and nobody had updated the shared doc.

  This is the verification earning its place. Under a
  store-then-hope design the tile would have shown
  Connected and the first team to try would have found
  out.

  Second attempt succeeded. Auto-registered a curated
  cheap + strong pair, usable immediately.

CONNECT 2: AWS Bedrock
  No new credential requested. It minted STS credentials
  off the AWS account already connected for cost
  management, then listed foundation models and
  inference profiles live.

  The security reviewer's note: "no additional standing
  credential" was the reason this one passed quickly.

ADDING A MODEL BEYOND THE CURATED SET
  A team needed a specific long-context model.
  Settings > Integrations > provider > Manage models
  Listed live from the provider, selected, assigned a
  tier.

  Removal is by opaque id, so a later rename cannot
  cause a delete to hit the wrong deployment.

THE ISOLATION QUESTION
  Reviewer: "You share one fleet across customers. What
  stops our key reaching another customer's vendor
  credential?"

  The wrong answer, and the one they expected to hear:
  "each key has a model allow-list."

  The actual answer: that was measured and it does not
  hold. Two deployments under the same model_name form a
  load-balancing pool, and a restricted key had 5 of 14
  requests served by another org's credential. So
  deployments are registered per-org as
  {orgID}/{logicalName}, no pool can form, and each key
  carries an alias applied after its model check.
  Re-measured 16/16 and 10/10.

  OUTCOME: this answer is what closed the review. A
  vendor who says "we measured our own boundary failing
  and changed the design" is more credible than one who
  asserts the boundary works.

3. Hands-on (6 min)

1. Connect one provider. Did verification pass on the
   first attempt?                             Y / N
   If N, what was wrong with the credential?
   ______________________________________________

2. After connecting, are models already registered?
                                              Y / N
   How many?  ______
   (Auto-registration means a connected provider is
   never a provider that serves nothing.)

3. If you use Bedrock: did it ask for a new credential?
                                              Y / N
   Which existing connection did it use?
   ______________________________________________

4. Open Manage models. Add one model beyond the curated
   set, and note how removal is keyed:
   [ ] by name    [ ] by opaque id

5. Write the tenant-isolation answer in your own words,
   including the mechanism rather than the intent:
   ______________________________________________

4. Knowledge check

Q1

Why is a provider credential verified against the vendor at connect time?

A. To discover which models that credential can actually reach and then register all of them automatically for the org
B. So a revoked or mistyped key fails at connect rather than turning the tile green and failing on the customer’s first inference
C. To mint the STS credentials that Bedrock needs before any inference request can be forwarded upstream to AWS
D. Because the vendor requires registration before any key may be used against their own endpoint at all in production

Q2

On a shared LiteLLM fleet, what actually enforces tenant isolation?

A. Each virtual key’s model allow-list, which gates which model names a key may call and therefore also which org’s deployment ends up answering that request in the end
B. Registering each deployment as {orgID}/{logicalName} so no two orgs share a model name and no load-balancing pool can form, with each key carrying an alias applied after its model check
C. The org stamp on the deployment metadata, which the gateway checks on every single request before it forwards anything upstream to the vendor’s own endpoint
D. Team-scoped deployments, so that a model registered by one team is invisible to every other team and therefore also to every other organisation on the fleet

Q3

Enabling AWS Bedrock models requires:

A. A separate Bedrock API key stored in the Config vault
B. A per-org LiteLLM fleet
C. An OpenRouter connection as a proxy
D. No additional credential

5. Apply

Connect a provider and confirm the verification behaviour by trying a deliberately wrong key first. Knowing that a bad credential fails at connect is what lets you trust a green tile.

Have the tenant-isolation answer ready as a mechanism rather than an intent, because on a shared fleet it is the question that decides the review.

Related lessons

• L1: What the AI Gateway governs
• L3: Virtual keys and budgets (next)
• L5: AI RBAC and provider scoping
• T1.M1.1.L1: Vault credentials

Glossary terms touched

Deployment · Model alias · Inference profile · SSRF guard