Skip to content

Chapter 2.5 — AI gateways, model gateways and call governance

🎯 Objective

Treat the AI gateway as an operational governance point between AI applications and model providers. The chapter covers what a gateway solves, what it does not solve, how to evaluate it, and the risks of adopting it blindly.

🧠 What an AI Gateway / Model Gateway is

An intermediary layer between applications consuming LLMs and providers (OpenAI, Anthropic, Vertex, Bedrock, Azure OpenAI, local models via vLLM, TGI, Ollama). It is usually a stateless or stateful HTTP/gRPC service with a backing store for budgets and logs.

🔁 Traditional API gateway vs AI gateway

Function Traditional API Gateway AI Gateway
AuthN / AuthZ ✅ ✅
Rate limiting ✅ (req/s) ✅ (req/s + tokens/s + cost/s)
Caching By key/URL Prompt cache + semantic cache
Routing By path/host By model, task, cost, jurisdiction
Fallback Backup endpoint Backup model + retry with a different model
Observability Latency, status Tokens, cost, quality, citations
DLP / Redaction Usually external Inline in prompt/output
Audit log Requests/responses Requests + tokens + policy decision
Budget No Yes, per tenant/product/agent
Versioning API version Model + prompt + schema

Summary: a traditional API gateway does not understand tokens, prompt caching, models or semantic risk — so, in an LLM portfolio, it is common to have both layers (an API gateway at the network perimeter and an AI-specific gateway).

🧠 Common capabilities

Capability What it does Limits
Model routing Chooses a model by task, cost, latency, jurisdiction Wrong routing silently degrades quality
Fallback Tries provider B if A fails/exceeds SLA Can mask incidents; beware of duplicate charges
Retry with backoff Handles 429/5xx Can amplify cost if poorly calibrated
Budget enforcement Blocks when exceeding a cap per user/tenant Needs fine classification; a poorly defined "tenant" leaks budget
Rate limiting Limits RPM/TPM/CPM (cost-per-min) Use in layers: client, tenant, agent
Tenant-level quotas Allocates capacity per customer Requires real isolation, not just a label
Prompt caching Reuses stable prefixes Needs a deterministic prefix + keying by version
Semantic caching Reuses a response for semantically similar queries Risk of a stale or incorrect response; use with TTL and revalidation
Observability Per-call trace, cost/token/latency metrics OpenTelemetry GenAI conventions apply
DLP / Redaction Masks PII/secrets in prompts and logs Not a defense against prompt injection
Audit log Immutable record Depends on retention and encryption
Guardrails Runtime classifiers and rules Mitigations, not controls (ch. 0.6)
Policy enforcement Integration with OPA/Cedar Policy depends on the contextual data exposed
MCP / tools integration Routes calls to MCP servers Retains the tool-misuse risk — real authorization is still the executor's responsibility

🏗️ Architecture patterns

application ──► AI gateway ──► provider A
                     │  ├──► provider B
                     │  └──► self-hosted (vLLM/TGI)
                     ├──► cache (prompt + semantic)
                     ├──► policy engine (OPA/Cedar)
                     ├──► observability (OTel -> backend)
                     └──► budget store (Redis/SQL)

In mature environments, agents do not talk directly to providers; they talk to the gateway. The gateway fails fast when there is a budget or policy violation.

🧪 Neutral comparison between widely used projects

Editorial stance. The book does not recommend a product. The comparison is only to guide evaluation. Check the official documentation and the current license before adopting.

Project License model Declared focus Common deployment form
LiteLLM Open-source (MIT) + enterprise edition Unified proxy for 100+ providers, budgets, logs, MCP integration Container/binary; Python SDK
Kong AI Gateway Plugins over Kong (open-source + enterprise edition) AI proxy plugins (rate limit, prompt template, semantic cache) integrated into Kong Kong Gateway with plugins enabled
Portkey SaaS + self-hosted option AI gateway + observability + guardrails Managed SaaS or self-hosted (paid plans)

Other projects exist (Cloudflare AI Gateway, Envoy AI extensions, etc.). Evaluation criteria:

  1. Support for the providers you use, with feature parity (streaming, tool calling, structured outputs).
  2. License model and total cost.
  3. Compatibility with your stack (observability, secrets, OPA, MCP).
  4. How rate limiting/budget work when the provider is down.
  5. Cache behavior and consistency model.
  6. Real multi-tenant support (not just tags).
  7. The possibility of self-hosting with data under your control.
  8. Frequency and quality of releases.

⚠️ What AI gateways do NOT guarantee

  • They do not replace authorization. What a user/agent can do is decided by IAM + policy engine, not the gateway.
  • They do not prevent prompt injection. DLP and heuristic filters are mitigations.
  • They do not eliminate hallucination. Caching can even amplify it if it serves an old response.
  • They do not replace evals. Quality metrics are the product's responsibility.
  • They do not guarantee multi-tenant isolation. Real isolation is storage, indexes, credentials; the gateway only observes.

🚨 Failure modes

  • Operational lock-in: the application only works with that gateway.
  • A single point of failure when all AI traffic goes through it.
  • Budgets computed from estimated tokens without reconciliation against the provider's bill.
  • A semantic cache returning another tenant's response due to a poorly isolated similarity search.
  • Automatic fallback masking a primary provider incident.
  • Logs with unredacted prompts leaking PII.
  • A false sense of governance because "it goes through the gateway".

🛡️ Controls and mitigations

  • Stateless and horizontally scalable; transparent failover.
  • Redaction before logging.
  • Budget with keys per tenant/agent and periodic reconciliation against the bill.
  • Semantic cache always with a tenant filter in the key.
  • Fallback metrics as an operational signal, not as a rug to sweep dirt under.
  • A retention policy for prompts and responses.
  • A portability plan: how to migrate to another gateway without rewriting the product.

🧠 Token budget

Budgets without an AI gateway become a spreadsheet. With a gateway, they become an executable policy. Define them per:

  • request;
  • step;
  • task;
  • user;
  • tenant;
  • product.

Without a budget, agents can enter a loop and consume the monthly budget in hours. See also Chapter 6.4 (FinOps).

📈 Metrics

  • Cost per task, per tenant, per product.
  • Hit rate of prompt cache and semantic cache.
  • Fallback rate per provider.
  • Rejections by budget and by policy.
  • Latency distribution per model.

📌 Checklist

  • [ ] Is there a gateway centralizing calls?
  • [ ] Is there explicit routing by task type?
  • [ ] Is there a per-tenant budget, reconciled against the bill?
  • [ ] Is the semantic cache isolated per tenant?
  • [ ] Are logs redacted before being persisted?
  • [ ] Is there a portability plan to another gateway?
  • [ ] Do fallback metrics trigger an operational alert?
  • EX-LLM-04 — a simplified gateway with routing, budget and redaction.
  • EX-COST-02 — a gateway with a tenant-controlled semantic cache.

📚 References