The AI gateway pattern for enterprise applications

The first AI feature in an enterprise is always a direct call. Someone drops an SDK into a service, pastes an API key into an environment variable, and ships. It works. So the next team does the same thing. And the next. Eighteen months later there are nineteen services calling four providers with a dozen keys, nobody can answer "what did we spend on AI last month and which team spent it," and the security team has just discovered that customer PII has been flowing to a provider that was never in the data processing agreement.

None of that is a modeling problem. It's an architecture problem, and it has a well-understood shape: you put a gateway in front of the providers. An AI gateway is a single internal service — a seam — that every application calls instead of calling OpenAI, Anthropic, Bedrock or Google directly. One doorway in, many providers out, and everything you need to control lives at the doorway.

The sprawl an AI gateway is fixing

Direct provider calls scattered across a codebase produce a predictable set of problems, and they compound.

Scattered keys. Every service that calls a provider needs a key. Keys end up in a dozen environment configs, some in secret managers, some — inevitably — hardcoded in a repo somewhere. Rotating a leaked key means finding every place it lives. Nobody can tell you how many valid keys exist.

No cost visibility. The provider bill arrives as one number. You cannot attribute it to a team, a feature, a customer or a model. When finance asks "why did AI spend triple this quarter," the honest answer is "we don't know, but we'll try to find out," and finding out takes a week of grepping logs.

No central policy. Want to ban a deprecated model? Enforce a per-request token cap? Require that summarization runs on a cheap model and only legal review touches the frontier one? With direct calls, "policy" is a code review comment repeated across nineteen services, enforced by nobody.

No audit trail. For a regulated buyer, "show me every prompt that touched customer data in March" is a real question with a real deadline. If each service logs however its author felt like, the answer doesn't exist.

Provider lock-in. When every service imports the vendor's SDK and speaks its exact request shape, switching providers — or even failing over during an outage — is a migration project, not a config change.

Each of these is solvable in isolation. The reason they don't get solved is that there's no place to solve them. The gateway is that place.

What the gateway is responsible for

The gateway is worth building because a whole stack of concerns collapses onto a single seam. Put them anywhere else and you're implementing each one N times.

Provider routing and fallback. Applications ask for a capability ("summarize this," "a fast cheap model," "the best reasoning model") and the gateway maps that to a concrete provider and model. If the primary fails or rate-limits, the gateway retries elsewhere. No app knows or cares which provider served the response.

Authentication and authorization. Apps authenticate to the gateway with their own internal identity — a service token, mTLS, an OIDC subject — not with a provider key. The gateway holds the real provider keys and never hands them out. Which team is allowed to use which model is an authorization decision made here, once.

Per-team budgets and rate limits. Every request carries a caller identity, so the gateway can enforce a monthly dollar budget per team, throttle a runaway batch job, and stop a single feature from exhausting a shared rate limit for everyone else.

Request/response logging and tracing. One structured log line per call — caller, model, token counts, latency, cost, a trace ID that stitches into your existing distributed tracing. This single feature pays for the gateway on its own.

Prompt and PII guardrails. Inbound prompts can be scanned and redacted before they ever leave your network; outbound responses can be checked for leaked secrets or policy violations. A blocklist, a PII redactor, a jailbreak classifier — all live at the seam, applied uniformly.

Caching. Identical or semantically similar requests can be served from a cache instead of a paid provider call. Even a modest hit rate on repetitive workloads (classification, tagging, boilerplate generation) is a direct line off the bill.

Model-version pinning. Providers deprecate and silently update models. The gateway pins summarizer-v3 to an exact model version so a provider-side change doesn't quietly alter your outputs. Upgrades become a deliberate, tested config change.

Compliance and audit. Because every call passes through one service, the audit trail is complete by construction. Data residency rules, retention policies and "who touched what" queries all have one place to live.

Routing and fallback, concretely

Routing is the responsibility teams underestimate, so it's worth making concrete.

A request arrives asking for the reasoning capability. The gateway's routing table says: primary is Provider A's frontier model, fallback is Provider B's equivalent, and the tier ceiling for this caller is "frontier allowed." The gateway calls A. If A returns a 429 or a 5xx, or breaches a latency budget, the gateway transparently retries against B. The application sees one successful response and never learns a failover happened.

Model tiering rides on the same machinery. Cheap, high-volume work — classification, extraction, tagging — routes to a small fast model. Only requests that genuinely need it reach the expensive frontier model, and whether a caller is even allowed to reach it is a policy the gateway enforces. This is usually the single largest cost lever in the whole system, and it's invisible to application code: the app asks for a capability, the gateway decides the tier.

Two rules keep this sane. First, applications must never name a raw provider model — they name a capability or a named route (summarizer, fast-cheap, reasoning). The mapping is the gateway's job, and that indirection is exactly what makes provider switches a config change. Second, fallback must be bounded: a retry budget and a circuit breaker, so a provider-wide outage degrades gracefully instead of turning every request into three slow failures in a row.

Governance is the enterprise reason it exists

For a startup, the gateway is a convenience. For an enterprise, the governance layer is the entire justification, and it's where the sprawl stories turn into audit findings.

Cost attribution and budgets. Every request is tagged with a team, and increasingly a tenant — which of your customers this call was made on behalf of. That turns the one big provider bill into a per-team, per-tenant, per-feature breakdown you can actually chargeback or optimize. Budgets become enforceable: when a team hits its monthly ceiling, the gateway can alert, throttle or hard-stop by policy rather than by surprise.

Data governance and PII. The gateway is the one place you can guarantee that PII is redacted before it reaches a provider, that requests from an EU tenant only route to region-pinned models, and that a provider not covered by your DPA is simply never called. These are the controls that get written into enterprise contracts, and a gateway is how you actually satisfy them instead of hoping every developer remembered.

Tenancy. In a multi-tenant product, the gateway enforces isolation — one tenant's cache never serves another's data, one tenant's budget can't be spent by another, and per-tenant routing rules (this customer requires on-prem models, that one forbids training-data retention) are applied without touching application code.

Audit and policy. An append-only log of every call, plus a policy engine that decides allow/deny/redact/route before the provider is ever touched, is what lets you answer a compliance question with a query instead of an apology. Policy lives as configuration, versioned and reviewable, not as tribal knowledge spread across teams.

Build vs. buy

You don't have to write this from scratch, and for most teams you shouldn't start there.

Buy (or adopt open source). Tools like LiteLLM, Portkey, Kong's AI gateway and cloud-native equivalents give you provider routing, a unified API, key management, budgets, caching and logging out of the box. LiteLLM in particular has become a common default: a proxy that speaks one API and fans out to a hundred providers, with virtual keys and per-key budgets built in. Adopt one of these when your needs are the standard ones — and they usually are at the start.

Build. An in-house gateway is worth it when your governance requirements are specific enough that a generic tool fights you: bespoke PII redaction tied to your data catalog, routing rules driven by your tenancy model, audit output that has to land in your compliance pipeline in your schema. The honest middle path most enterprises land on is buy the data plane, build the policy plane — run LiteLLM or Portkey for the routing and provider abstraction, wrap it in a thin internal service that owns your auth, your tenancy tags and your audit format.

The one thing that is not optional is the seam itself. Whether the box is bought or built, applications must call one internal endpoint, never a provider directly. That discipline is what makes everything else possible; the implementation behind it can evolve.

How it composes with the rest of the stack

The gateway is a network hop, so it inherits and extends what you already run. It authenticates callers with your existing identity system. It emits traces into your existing observability stack with a trace ID that stitches an LLM call into the surrounding request. It reads secrets from your existing secret manager and holds the provider keys there. Its budget and audit data flow to the same dashboards and SIEM everything else uses.

Latency-wise it's one extra internal hop — single-digit milliseconds if you run it in-region and keep it stateless behind a load balancer. Deploy it like any other tier-one internal service: horizontally scaled, health-checked, with the provider keys as its only real secret. It should be boring infrastructure, because everything interesting it does is policy, not compute.

Mistakes

Letting one team keep a direct line. The gateway's value is that it's the only path. The moment one "special" service is allowed to call a provider directly, your audit trail has a hole, your cost numbers are wrong, and the exception becomes precedent. Enforce it at the network layer — providers are only reachable from the gateway's egress, nowhere else.

Making it a smart monolith. The gateway routes, authorizes, logs and guards. It does not do your prompt engineering, own your business logic, or become the place every feature dumps its orchestration. A gateway that grows application logic becomes a bottleneck every team has to queue behind. Keep it thin.

Ignoring the latency and failure budget. A synchronous hop in front of every AI call is a new dependency on the critical path. If the gateway is down, everything AI is down. Treat it like the tier-one service it is: redundant, circuit-breakered, with sane timeouts and bounded retries so a slow provider doesn't cascade.

Shipping without cost attribution from day one. Retrofitting per-team, per-tenant tagging after the fact means backfilling identity into calls that never carried it. Make the caller identity and tenant tag mandatory fields on the very first request the gateway ever serves.

Caching without thinking about tenancy and correctness. A cache that serves one tenant's response to another is a data breach, not a saving. A cache that serves a stale answer for a request that needed freshness is a correctness bug. Scope cache keys by tenant and be deliberate about what's cacheable.

TL;DR

An AI gateway is a single internal service every application calls instead of calling model providers directly. It exists because direct calls sprawl: scattered keys, no cost visibility, no central policy, no audit trail, provider lock-in.

Onto that one seam collapses a whole stack of concerns — provider routing and fallback, authentication and authorization, per-team budgets and rate limits, logging and tracing, PII guardrails and redaction, caching, model-version pinning, and compliance audit. Routing and model tiering are the biggest cost levers; the governance layer — cost attribution, data residency, tenancy, policy enforcement, audit — is the reason enterprises need it at all.

Don't start by building it. Adopt LiteLLM, Portkey or a cloud-native gateway for the data plane, and if your governance needs are specific, wrap it in a thin service that owns auth, tenancy and audit. The one rule that isn't negotiable: applications call the gateway, never a provider directly.

If your AI usage is sprawling across teams, let's talk.