How AI changes backend architecture — the parts that are genuinely different

Most of what we know about building backends assumes a few things quietly hold true. A request completes in milliseconds. The same input produces the same output. A call to a downstream service costs nothing you'd put on a spreadsheet. And a request handler is stateless — it takes an input, returns an output, and forgets everything.

Put an LLM in the request path and all four of those assumptions break at once. Not slowly, not at scale — immediately, on the first call. This is why teams who bolt an LLM onto a conventional backend keep getting surprised: the architecture was designed around guarantees the model doesn't provide.

This post is about the parts that are genuinely different, and the patterns that hold up once you accept them.

Latency: seconds, not milliseconds

A typical API call to a database or internal service returns in single-digit to low-tens of milliseconds. An LLM call returns in seconds — one to ten, sometimes more for long outputs or reasoning models. That's two to three orders of magnitude slower than everything else in your stack.

You cannot hide a five-second call inside a synchronous request/response cycle and pretend it's fine. It blocks a worker thread, it holds a connection open, it trips load-balancer and gateway timeouts (often defaulted to 30 or 60 seconds), and it makes your p99 latency meaningless because the AI path dominates it.

The fixes are the ones you'd reach for with any slow operation, applied deliberately:

• Stream the response. For anything a human reads, stream tokens as they're generated (SSE or WebSockets). Time-to-first-token is what the user perceives, and it's usually under a second even when total generation takes eight. Streaming turns "slow" into "responsive."
• Go async for anything non-interactive. Batch summarisation, document processing, agent workflows — push these onto a job queue, return a job ID immediately, and let the client poll or receive a webhook. The request that kicks off the work should return in milliseconds; the work itself runs on a worker.
• Set timeouts that match reality. A 3-second timeout that's correct for your database is wrong for an LLM. Set per-call timeouts explicitly, and set them low enough that a hung provider doesn't pin your workers indefinitely.

Non-determinism: same input, different output

Send the same prompt twice and you can get two different answers. Even at temperature zero, providers don't guarantee bit-identical output across calls — model updates, load-balanced backends, and floating-point nondeterminism all leak through. This quietly invalidates a lot of standard backend hygiene.

Retries stop being safe by default. In a normal system, retrying an idempotent GET is free. Retry an LLM call and you may get a different result, so a retry after a partial success can double-charge a user, send two emails, or produce two conflicting records. If a model call has side effects, you need an idempotency key and a way to detect "this work already happened" — the same discipline you'd use for payment processing.

You must validate the output, every time. The model will occasionally return malformed JSON, an extra field, prose wrapped around your JSON, or a value outside the enum you asked for. Treat every model response as untrusted input: parse it against a schema, and have a defined path for when parsing fails (repair prompt, retry with a stricter instruction, or fall back). Structured-output / tool-calling modes reduce this but do not eliminate it.

Guardrails are part of the request path, not an afterthought. Input validation (prompt-injection screening, PII detection) and output validation (toxicity, policy, format) are architectural components with their own latency and failure modes. Budget for them.

Cost per request: a call can cost real money

This is the assumption whose breaking surprises finance, not just engineering. A conventional request's marginal cost rounds to zero. An LLM request has a real, variable, per-call cost driven by tokens in and tokens out — fractions of a cent for a small call, but several cents for a long-context or agent call that loops. Multiply by traffic and a single careless feature becomes a five-figure monthly line item.

Cost becomes a first-class design constraint:

• Cache aggressively — including semantically. Exact-match caching catches identical prompts. Semantic caching — embedding the request and returning a cached answer when a near-identical question was asked before — catches the long tail of "same question, different wording." For FAQ-style and retrieval workloads this can remove a large fraction of calls outright.
• Tier your models. Don't send every request to your most expensive model. Route simple classification and extraction to a small, cheap model; reserve the frontier model for the hard cases. A cheap model plus a confidence check that escalates to an expensive one is often the right shape.
• Budget per tenant. Track spend per user, per feature, per tenant, and enforce limits. Without this, one customer's runaway agent loop is your bill. Rate limits protect availability; spend limits protect the P&L.
• Cap tokens. Set max_tokens deliberately. An unbounded output is an unbounded cost.

Statefulness: the conversation has to live somewhere

A REST handler is stateless by design — that's what let us scale horizontally without thinking. But a conversation or an agent has memory: the prior turns, the retrieved context, the tools it has already called, the intermediate results. That state has to live somewhere, and "in the request" is no longer an option because the meaningful unit of work now spans many requests.

This pushes real design decisions back onto you:

• Where does conversation state live? Usually a fast store (Redis, or a database with a session table) keyed by conversation ID, holding the message history and any working context. It has to be durable enough to survive a worker restart mid-conversation and fast enough to load on every turn.
• How much history do you replay? The full transcript grows without bound and every token is paid for on every turn. You need a windowing or summarisation strategy — keep the last N turns verbatim, summarise the rest — which is itself a model call with its own cost and latency.
• Agents make this sharper. A multi-step agent has a plan, a scratchpad, and tool-call results that must persist across steps, plus the ability to resume after a failure three steps in. That's closer to a durable workflow engine (think Temporal-style checkpointing) than to a request handler. If you're building agents, model the state explicitly and persist it at every step — don't hold it in memory and hope the process survives.

New failure modes

Your backend already handles downstream failures. The AI boundary adds failure modes that are shaped differently and happen more often.

• Rate limits are normal, not exceptional. Providers enforce requests-per-minute and tokens-per-minute limits, and you will hit them under load. Handle 429s with backoff and a queue, not a 500 to the user.
• Provider outages happen and are outside your control. The major providers have real incidents. If your product hard-depends on one, its bad day is your bad day.
• Hallucination is a failure mode without an exception. The call succeeds, returns 200, and the content is confidently wrong. Nothing throws. Grounding (retrieval), output validation, and — for high-stakes actions — a human or a deterministic check in the loop are how you contain it.

The responses are familiar patterns pointed at a new target: circuit breakers so a struggling provider fails fast instead of dragging your latency down; fallbacks to a cached answer, a smaller model, or a degraded-but-honest "try again shortly"; and multi-provider abstraction so you can fail over from one vendor to another. Which leads to the seam that makes all of this manageable.

The AI gateway is the architectural seam

Everything above — retries, timeouts, caching, budgeting, model tiering, guardrails, provider failover, observability — wants to live in one place. Scatter it across every feature that calls a model and you'll implement it five times, inconsistently, and be unable to change providers without a migration.

The pattern that holds up is an AI gateway: a single internal boundary that every model call goes through. Application services never call a provider SDK directly; they call your gateway, which owns:

• Provider abstraction — a stable internal interface so openai vs anthropic vs a self-hosted model is a config change, not a code change. This is also what makes failover and A/B model testing possible.
• Cross-cutting concerns — caching, rate-limit handling, retries, timeouts, budgeting, and guardrails, implemented once.
• Observability — the single point where you can log, trace, and measure every call.

This is a well-understood move — it's the anti-corruption layer / port-and-adapter idea applied to a volatile external dependency. LLM providers are the most volatile dependency most backends have ever had: models get deprecated, prices change, better options appear monthly. The gateway is what lets you absorb that churn without it rippling through your codebase.

Observing a probabilistic system

You cannot debug what you can't see, and an AI system fails in ways your existing dashboards don't show. A p99 latency chart tells you nothing about whether the answers got worse after a model update.

You need to capture, per call: the full prompt and response, token counts and cost, latency, model and version, cache hit/miss, and whether output validation passed. Traces should span the whole chain — retrieval, the model call, tool calls, validation — because in an agent the interesting failure is usually the interaction between steps, not one step. And because "correct" is fuzzy, quality itself becomes something you measure with evaluation sets and (carefully) LLM-as-judge scoring, run continuously rather than at release. Treat the model as a dependency whose behaviour drifts, and instrument accordingly.

mistakes

Calling the model synchronously from the request handler. It works in the demo with one user, then falls over the moment real traffic or a slow provider shows up. Decide sync-with-streaming vs. async-job up front — it's expensive to retrofit.

Trusting the output. Treating model responses as valid data instead of untrusted input. Parse against a schema, always, with a defined failure path.

Retrying calls with side effects like they're idempotent. Non-determinism means a retry can duplicate work. Use idempotency keys for any call that writes, sends, or charges.

No cost ceiling. Shipping without per-tenant budgets and max_tokens caps. One agent loop or one abusive user, and you find out from the invoice.

Scattering provider calls across the codebase. Every feature calling the SDK directly means you implement retries and caching N times and can't change providers. Route everything through one gateway from day one.

Depending on a single provider for a critical path. Their outage becomes your outage. Abstract the provider even if you only wire up one at first.

TL;DR

An LLM in the request path breaks four assumptions conventional backends rely on: millisecond latency, deterministic output, near-zero cost per call, and stateless handlers.

Design for it: stream or go async because calls take seconds; validate every output and use idempotency keys because output is non-deterministic; cache semantically, tier models, and budget per tenant because calls cost real money; persist conversation and agent state because the unit of work now spans requests. Put retries, timeouts, caching, budgeting, guardrails, provider abstraction, and observability behind a single AI gateway so the most volatile dependency your backend has ever had stays a config change, not a rewrite.

The individual patterns are familiar. What's new is that you need all of them, at once, for a dependency that's slow, probabilistic, metered, and stateful.

If AI is reshaping your backend, let's talk.