RAG is not dead — you're just building it wrong

Every few months someone declares RAG dead. The argument is always the same: context windows keep growing — 200K, 1M, 2M tokens — so why bother retrieving the right chunks when you can just stuff the whole corpus into the prompt and let the model sort it out?

It's a tempting story. It's also wrong, and the reason it's wrong tells you almost everything about why most RAG systems underperform. The teams declaring RAG dead usually never built RAG that worked in the first place. They built naive RAG — fixed-size chunks, a single embedding lookup, top-k stuffed into a prompt, no evaluation — watched it return mediocre answers, and concluded the whole approach was a dead end. What actually died was their specific, under-engineered pipeline.

RAG isn't a technique you either use or don't. It's a retrieval quality problem, and retrieval quality is something you engineer.

Why "just use long context" doesn't kill retrieval

Start with the thing that's supposedly the killer. Long context is real and useful, but it complements retrieval — it doesn't replace it. Three reasons.

Cost. Tokens aren't free. If every query stuffs 500K tokens of corpus into the prompt, you pay for 500K tokens of input on every single request. At scale that's the difference between a product with margins and a science project. Retrieval that hands the model 4K high-signal tokens instead of 500K low-signal ones is not a compromise — it's the economically viable version of the same feature.

Latency. Prefill scales with input length. A million-token prompt has to be processed before the model emits a single output token, and users feel every millisecond of it. Retrieving the right 4K tokens keeps time-to-first-token in the range people actually tolerate.

Lost in the middle. This is the one people underestimate. Models don't attend uniformly across a long context — retrieval accuracy sags for information buried in the middle of a large prompt. Dumping your whole knowledge base into context doesn't guarantee the model uses the relevant part; it often buries the signal under noise. Feeding the model a small, ranked set of highly relevant chunks isn't just cheaper and faster — it produces better answers than the giant-context version.

So the choice was never "retrieval or long context." Good systems use long context to be more forgiving about chunk boundaries and to fit more reranked candidates — not to skip retrieval entirely.

Why most RAG underperforms

If RAG is fine, why does so much of it feel broken? Because the default tutorial pipeline skips every step that actually matters.

Naive fixed-size chunking. The most common mistake, and the most damaging. You split documents into 512-token windows with some overlap and call it done. The problem: 512 tokens has nothing to do with where ideas begin and end. You slice a table in half, split a function from its explanation, cut a definition off from the sentence that qualifies it. Now the embedding for that chunk represents half an idea, and retrieval either misses it or returns a fragment the model can't use.

Embedding-only retrieval with no reranker. A single dense-vector lookup gives you approximately semantically similar chunks. "Approximately" is the operative word — embeddings collapse a lot of meaning into a fixed vector, and the top-k by cosine similarity is full of near-misses. Without a reranking step, you're handing the model a noisy candidate set and hoping.

No query rewriting. Users don't write queries that embed well. They write "why is it slow" when the document says "p99 latency regression under connection-pool saturation." They ask follow-ups that only make sense given three turns of prior context ("what about the second one?"). Embedding the raw user string and searching with it throws away most of your retrieval potential.

No evaluation of retrieval quality. This is the silent killer. Most teams evaluate the final answer — vibes, a few test prompts, maybe an LLM judge — and never measure whether retrieval actually surfaced the right chunks. So when answers are bad, they can't tell whether the model reasoned poorly or never received the relevant context at all. You can't fix what you don't measure, and retrieval is the part nobody measures.

Over-stuffed context. The overcorrection to bad retrieval: "recall is low, so let's return top-30 instead of top-5." Now the model wades through 30 chunks, most of them irrelevant, and lost-in-the-middle does the rest. More context is not more signal. Concentration of signal is what matters.

The pipeline that actually works

Here's the shape of a retrieval pipeline that earns its keep. None of these steps is exotic — the discipline is doing all of them and measuring the result.

Chunk on meaning, not on token count. Split along the document's own structure — headings, sections, list items, code blocks, table rows. For prose, semantic chunking (grouping sentences by embedding similarity and breaking where the topic shifts) beats fixed windows. The rule of thumb: a chunk should be a self-contained unit of meaning that would make sense if a human read it alone. Keep the parent-document reference so you can expand context when the model needs it.

Retrieve hybrid — dense plus keyword. Dense embeddings are great at semantic similarity and terrible at exact matches: product SKUs, error codes, function names, rare proper nouns. Keyword search (BM25) is the opposite. Run both and fuse the results (reciprocal rank fusion is the simple, robust default). Hybrid retrieval consistently beats either method alone, because most real queries need a bit of both.

Rerank the candidates. Retrieve a wide candidate set — 50 to 100 chunks — cheaply, then pass them through a cross-encoder reranker that scores each chunk against the query directly. Cross-encoders are far more accurate than bi-encoder embeddings because they look at query and chunk together instead of comparing two independent vectors. This is the single highest-leverage upgrade for most systems: same retriever, add a reranker, keep only the top 5, and answer quality jumps.

Transform the query before you search. Rewrite the raw user input into a retrieval-friendly form. Resolve pronouns and references from conversation history. Expand a terse question into the vocabulary the documents actually use. For hard queries, generate a few paraphrases and retrieve for each (multi-query), or use HyDE — have the model draft a hypothetical answer and embed that, since a fake answer often sits closer in vector space to the real source than the question does.

Evaluate retrieval as its own metric. This is the step that turns RAG from guesswork into engineering. Build a labeled set: representative queries paired with the chunk IDs that should be retrieved. Then measure retrieval directly — precision (of what you returned, how much was relevant), recall (of what was relevant, how much you returned), and rank-aware metrics like MRR or nDCG that reward putting the right chunk near the top. Now every change — new chunking, a different reranker, a query-rewrite tweak — is a number that goes up or down, not a vibe. You'll find that most "the model is dumb" complaints are actually "recall@5 is 0.4" problems.

The mistakes that keep RAG broken

Evaluating the answer but never the retrieval. If you only score the final output, you can't localize the failure. Separate the two: measure retrieval quality against a labeled set, then measure generation quality given good context. Fix them independently.

Chunking once and never revisiting it. Chunking is not a set-and-forget decision. It's the highest-leverage variable in the whole pipeline and the one people touch least. When retrieval eval is flat, chunking is usually the first thing to change.

Treating the reranker as optional. It's the cheapest large win available. Skipping it to save 50ms of latency while shipping worse answers is a bad trade in almost every product.

Confusing "more context" with "better context." Returning top-30 to be safe is a recall band-aid that hurts precision and triggers lost-in-the-middle. Retrieve wide, rerank hard, return narrow.

Ignoring keyword search because embeddings feel modern. Dense-only retrieval silently fails on exact-match queries — the error codes and identifiers users paste in most. Hybrid is not a legacy compromise; it's the correct default.

TL;DR

RAG isn't dead — naive RAG is, and it was never really alive. Long context complements retrieval rather than replacing it: retrieval is what keeps you cheap (fewer tokens), fast (shorter prefill), and accurate (no lost-in-the-middle).

Most RAG underperforms for five fixable reasons: fixed-size chunking that splits ideas, embedding-only retrieval with no reranker, no query rewriting, no retrieval evaluation, and over-stuffed low-signal context.

The pipeline that works: chunk on meaning, retrieve hybrid (dense + keyword), rerank a wide candidate set down to a narrow top-k, transform the query before searching, and — the step that makes all the others improvable — evaluate retrieval precision and recall on a labeled set. Measure retrieval and generation separately so you can tell which one is failing.

Do those things and RAG doesn't just work — it beats the giant-context version on cost, latency, and answer quality at the same time.

If your RAG system isn't pulling its weight, let's talk.