AI rate limiting and cost control — before the bill surprises you

A normal API bill is boring. You provisioned some servers, you pay for them whether they're busy or idle, and the invoice is roughly the same every month. You can be careless with request counts for a long time before it costs you anything.

An LLM bill is not boring. Every request costs real money in near-linear proportion to how much text goes in and comes out. A user who pastes a 200-page PDF into your summariser costs a hundred times more than a user asking "what's the weather." A retry loop with no backoff can fire ten thousand calls before anyone notices. We have watched a single misconfigured cron job turn a healthy AI feature into a five-figure invoice over one weekend — and nobody was even using the product.

Rate limiting and cost control for AI are not the same problem as for a normal API. Here's why, and here's how to cap spend without quietly breaking the thing people came for.

Why AI cost control is different

Three properties make LLM spend behave unlike anything in a traditional backend.

Cost scales with tokens, not requests. The unit you pay for is the token, not the call. Counting requests tells you almost nothing about your bill. Two endpoints can serve the same number of requests per second and differ by 50x in cost because one processes tweets and the other processes contracts. If you rate-limit by request count alone, you are throttling the wrong dimension.

One request can be 100x another. In a CRUD API, a GET /orders/123 and a GET /orders/456 cost the same. In an LLM API, request size is unbounded by default. Input context, retrieved documents, conversation history and output length all vary wildly per call. Your worst-case request is not twice your average — it's two orders of magnitude worse, and a single user can trigger it deliberately.

Per-request cost is money, immediately. There's no amortisation. Every token you generate is billed at generation time. A bug doesn't waste spare capacity you already paid for — it spends new money you didn't budget for. That changes the posture: cost control has to be preventive, enforced before the call goes out, not reconciled at the end of the month.

Put those together and the lesson is simple. You need to meter, budget and cap on tokens and dollars, and you need to do it in the request path.

Lever 1 — Rate limit by tokens, not just requests

Keep request-rate limits — they're still your first defence against raw flooding. But add a second bucket that meters token throughput.

The mechanism is the same token bucket you already know, applied to a different quantity. Each user (or team, or API key) has a bucket that refills at a fixed rate — say, 100,000 tokens per minute — and drains by the estimated token cost of each request before it's sent. If the bucket can't cover the request, you throttle. The trick is that you must estimate the token cost up front, because you can't measure output tokens until after generation. Estimate conservatively (prompt tokens are exact; reserve the max output budget), then reconcile the bucket with actual usage once the response returns.

Tier the buckets. A free user gets a small token budget and a small model. A pro user gets more. An enterprise key gets a large budget and priority. The tiers do two jobs at once: they align cost with revenue, and they contain blast radius — one abusive tenant on the free tier can't drain the capacity your paying customers rely on.

Lever 2 — Budgets and quotas with hard and soft caps

Rate limits govern speed. Budgets govern total spend over a window — per day, per month, per customer. These are different guarantees and you want both.

Give every budget two thresholds. The soft cap is where you start degrading gracefully: the feature still works, but cheaper. The hard cap is where you stop, because past it you're losing money or exceeding what the customer has paid for.

The point of the soft cap is that "the product stops working" should almost never be the first thing a user experiences. Between soft and hard, you have room to downgrade the model, shorten the output, disable an expensive optional step (like a second reasoning pass), or queue non-urgent work for off-peak processing. The user gets a slightly leaner experience; you get a bill that stays inside its lane. Only at the hard cap do you actually block or queue — and even then, block with a clear message and a path to upgrade, not a 500.

Lever 3 — Model tiering

Not every query deserves your most expensive model. "Reformat this list" and "reason about this legal clause" are different jobs, and paying frontier-model prices for the first is pure waste.

Route by need. A cheap, fast model handles classification, extraction, short rewrites and simple Q&A. The expensive model is reserved for genuine reasoning, long-context synthesis and anything the cheap model demonstrably fails at. You can decide the route with a lightweight classifier, with heuristics on input length and task type, or by trying the cheap model first and escalating only on low-confidence output.

Done well, model tiering is usually the single biggest line-item reduction available — often more than caching — because most real traffic is mundane. The mistake is treating "which model" as a global config setting instead of a per-request routing decision.

Lever 4 — Caching, exact and semantic

If you paid to answer a question once, don't pay to answer it again.

Exact caching keys on the normalised prompt. Identical input, identical parameters, return the stored response. This is trivial to implement and catches a surprising amount of traffic — repeated system prompts, common FAQs, retried requests, the same document summarised twice.

Semantic caching keys on the meaning of the input via an embedding. "What's your refund policy?" and "how do I get my money back?" are different strings but the same question. You embed the incoming query, search for a near-duplicate above a similarity threshold, and serve the cached answer if you find one. It's more powerful and more dangerous — set the threshold too loose and you'll serve confidently wrong answers — so reserve it for stable, factual content and keep a tight threshold.

Cache the retrieval and embedding steps too, not just the final generation. In a RAG pipeline, the embedding of a repeated query and the retrieved chunks are just as cacheable as the answer.

Lever 5 — Cap the output and the context

The cheapest tokens are the ones you never send.

Set a max_tokens on every generation. An unbounded output is an unbounded bill, and a model that rambles for 4,000 tokens when 400 would do is costing you 10x for no benefit. Pick a ceiling that fits the task and enforce it.

Cap the context on the way in, too. Long conversations grow without limit if you keep replaying the full history on every turn — and you pay for that entire history on every message. Trim it: keep a rolling window of recent turns, summarise older ones into a compact running summary, and drop retrieved documents that didn't contribute. Truncate or reject oversized user input before it reaches the model rather than silently paying to process a pasted novel.

Lever 6 — Abuse protection and runaway-loop guards

The scenarios that produce the scary invoices are rarely normal usage. They're automation gone wrong and users acting in bad faith.

Guard against the runaway loop specifically. Any code path that can call the model in a loop — an agent, a retry handler, a batch job — needs a hard iteration cap and a circuit breaker. If an agent has taken 20 steps without finishing, stop it; something is wrong. Retries need exponential backoff and a maximum count, because a tight retry loop against a transient error is the fastest way to five figures.

For abuse, watch for the signatures: a single key suddenly consuming 100x its normal token volume, a spike in maximum-length requests, the same expensive prompt fired thousands of times. Alert on cost anomalies the way you'd alert on error rates. And enforce per-key hard caps so that even a fully compromised key has a bounded worst case.

Where all this lives — the AI gateway seam

Do not scatter these checks across every feature that touches the model. Put them in one place: a thin gateway that every LLM call passes through.

The gateway is the seam where estimation, budgeting, routing, caching and metering all happen. A request comes in; the gateway estimates its token cost, checks the rate bucket and the budget, decides which model to route to, checks the cache, applies the output and context caps, makes the call, records actual usage, and reconciles the buckets. Every feature calls the gateway instead of the provider SDK directly. That's what makes the controls enforceable — there's exactly one code path to get right, one place to change a limit, and one place that sees your true spend.

The gateway is also where you decide how to fail. The default should be to degrade gracefully — downgrade the model, shorten the output, serve a cached or slightly stale answer, queue the work. Reserve the hard stop for the hard cap and for clear abuse, and when you do stop, return a clear, honest message with a route to upgrade. A user who hits a limit and understands why will upgrade; a user who hits a mysterious 500 will churn.

The mistakes that make AI cost control fail

Metering requests instead of tokens. Request counts don't correlate with your bill. If your dashboards and limits are in requests, you're flying blind on the number that actually matters.

Reconciling monthly instead of enforcing per-request. By the time the invoice arrives, the money is gone. Cost control has to happen before the call, in the request path — not in a spreadsheet at month end.

Only hard caps, no soft caps. If your only tool is "block," every limit is a broken feature. Soft caps and graceful degradation let you protect spend while keeping the product alive.

No max_tokens and no context trimming. These are one-line defences against unbounded output and unbounded history. Leaving them off is leaving the meter running.

No loop guard. The single most expensive incidents come from automation with no iteration cap. Every loop that can call a model needs a hard ceiling and a circuit breaker.

Controls scattered across features. If each feature calls the provider SDK directly, you have no single place to enforce anything and no accurate view of total spend. Route everything through the gateway.

TL;DR

AI cost is different because it scales with tokens, varies 100x per request, and spends real money at generation time. So meter and cap on tokens and dollars, and do it in the request path.

Rate limit by token throughput, tiered per user, team and key. Set budgets with soft caps (degrade gracefully) and hard caps (block or queue). Route cheap queries to cheap models. Cache exact and semantic hits so you never pay twice. Cap output tokens and trim context. Guard every loop with an iteration limit and a circuit breaker. Put all of it behind one AI gateway so there's a single place to enforce limits and see true spend — and fail by degrading, not by 500.

If your AI costs are unpredictable, let's talk.