The RAG wave that didn’t survive contact with reality
Two years ago, “RAG” meant a specific architecture: chunk documents, embed the chunks, store embeddings in a vector database, retrieve the top-k by cosine similarity at query time, stuff the chunks into the prompt, let the model answer. There were probably a thousand startups built on this exact stack.
Most of them produced demos that worked impressively well on curated documents and production systems that gave wrong answers about 25% of the time. The failure modes were consistent: the retrieved chunks were near-misses, the top-k cut off the right answer, the chunking destroyed structure, the model confabulated when retrieval failed. Teams responded by tuning embedding models, expanding k, re-ranking — squeezing 2–3 percentage points of improvement at significant cost while the fundamental architecture stayed brittle.
The teams that have moved on from this don’t talk about “RAG” much anymore. They talk about retrieval, which has gotten substantially more sophisticated, and the word “RAG” no longer describes a single architecture. It describes a problem space. Below is what the solution actually looks like in 2026.
The five things that changed
1. Vector search alone is no longer the default
The honest data: pure vector search rarely beats well-tuned BM25 on retrieval quality. It often loses. On most enterprise corpora, lexical search (a 50-year-old technique) is competitive with or better than embedding search, especially for short queries that contain rare terms.
The 2026 default is hybrid retrieval: BM25 + embedding search, combined with a reranker. Lexical handles rare-term queries and exact-match needs; vectors handle semantic similarity. A reranker (often a small specialized model) re-orders the merged candidate set before it’s passed to the LLM.
Pure vector RAG made sense when the rest of the stack didn’t exist. The stack exists now. Use it.
2. Chunking is no longer fixed-window
The original chunking heuristic — 500-token sliding windows with overlap — was a defensible default and a fundamentally lossy operation. It destroys document structure, splits paragraphs mid-thought, and chunks tables across boundaries.
The current best practice is structure-aware chunking:
- Documents parsed into their structural elements (headings, sections, paragraphs, tables, lists, code blocks).
- Chunks aligned to structural boundaries.
- Metadata preserved (parent heading, document path, source URL, last-modified).
- Tables and code blocks treated as atomic when possible.
The cost is a few hours of engineering per document type. The retrieval quality improvement is large enough that it dominates further embedding model tuning.
3. Retrieval is multi-stage, not single-shot
The single retrieve-then-answer pattern works for simple questions. It fails for anything multi-part, or anything that requires reasoning over multiple documents.
The pattern that works is agentic retrieval: the agent decides what to retrieve, evaluates whether it has enough, and retrieves more if it doesn’t. Concretely:
- The agent issues an initial broad query.
- It evaluates the results: does this answer the question, partially answer it, or miss?
- It issues follow-up queries with refined terms or different filters.
- It synthesizes across the gathered evidence.
This sounds expensive. It is. It is also dramatically better for non-trivial queries. The right framing is that retrieval is a workflow, not a function call.
4. Long-context is now a real option
When Claude shipped 1M-token context with prompt caching, the calculus for some retrieval problems shifted. For a corpus that fits in context (which is now a much larger corpus than it used to be), the right answer is sometimes “load the whole thing into context and skip retrieval entirely.”
This is not always cheaper. It often is, especially with caching. It’s almost always more reliable, because the model isn’t depending on the retrieval being right.
The decision tree we use:
- Corpus < 200k tokens? Load it in context. Don’t bother retrieving.
- Corpus 200k–10M tokens? Hybrid retrieval into context.
- Corpus > 10M tokens? Agentic retrieval over a hybrid-indexed store.
- Corpus changes rapidly? Always retrieval, regardless of size.
The “always retrieve everything” pattern from 2023 was an artifact of small context windows. Let it go.
5. Source attribution is mandatory now
Production retrieval systems in 2026 cite their sources. Every claim in the response is traceable back to a specific chunk in the source corpus. This isn’t a UX feature — it’s a correctness control. When the user (or an auditor) asks “where did that come from,” there’s an answer.
Implementations vary: inline citation marks, structured citation objects, source URLs as part of the response. The architectural requirement is that the retrieval layer passes provenance through to the generation layer, and the prompt instructs the model to cite.
Systems without source attribution have a hidden hallucination risk: the model produces plausible-sounding answers when retrieval underdelivers. Source attribution forces the model to either find the support or admit it can’t.
The architecture that holds up
Below is the retrieval architecture we ship by default in 2026.
- Document ingestion pipeline. Structure-aware parser per document type. Outputs structured chunks with metadata.
- Hybrid index. BM25 (or a competent equivalent) + embedding index. Often the same store; sometimes separate stores joined at query time.
- Reranker. A small model that re-orders the top candidates. Specialized rerankers (Cohere Rerank, BGE, etc.) substantially beat raw similarity ranking.
- Retrieval agent. A thin agent that issues queries, evaluates results, and decides whether to query again. Bounded by a maximum step count.
- Source attribution. Provenance flows from chunks to citation in the response.
- Eval harness. A standing set of queries with known correct answers, run continuously, with regression alerts.
This is a more sophisticated stack than “drop everything into Pinecone.” It is also strictly better, and the engineering cost is bounded.
Where simple still wins
Simple retrieval still works for some cases:
- Small, stable corpora. A 50-document handbook doesn’t need agentic retrieval. Load it in context.
- Tightly scoped queries. “What does our refund policy say about returns over 30 days?” is a string-match problem.
- High-recall over high-precision needs. If you want every document that touches a topic for a human reviewer, BM25 alone is often the right answer.
Don’t over-engineer when the simple thing works. The point of the more sophisticated architecture is to handle the cases where simple breaks, not to be sophisticated for its own sake.
The patterns we wouldn’t repeat
Three patterns we’ve seen and would actively warn against:
Pure vector RAG as the only retrieval mode. Brittle, hard to debug, performs poorly on rare-term queries. If your stack is this, plan a hybrid migration.
Re-embedding entire corpora to chase embedding-model improvements. Real-world quality improvement is usually marginal. The compute is large. The right work is almost always in chunking, reranking, or agentic structure — not embedding tuning.
Vector DBs as the central data architecture. Many teams put their vector store at the center and the rest of the data architecture orbits it. This is backwards. The source-of-truth data is in operational stores; the vector index is a derived secondary structure. Treat it as cache, not as canonical.
What this means for buying versus building
The retrieval space has matured to the point where you can buy a lot of this stack. Cohere’s reranker, OpenAI/Anthropic/Voyage embeddings, several hybrid search engines (Vespa, Elasticsearch with reciprocal rank fusion, Weaviate hybrid), and several “RAG as a service” offerings (Glean, Vectara). The buy decisions depend on your data sensitivity and on whether you need to customize the retrieval logic.
For most mid-market companies, a hybrid: buy the components (search engine, reranker, embeddings), build the agentic retrieval and the integration. This minimizes the surface area you have to maintain while keeping control over the parts that matter.
The take
“RAG” as a single architecture is dead. Retrieval as a problem is more important than ever. The shape that works in 2026 is hybrid, multi-stage, structure-aware, and explicitly cites its sources. It’s more engineering than the 2023 default. It’s also dramatically more reliable, and the difference between the two architectures in production is the difference between an AI product customers trust and one they merely tolerate.
Retrieval architecture is one of the most common areas we redesign during Custom AI Builds. If your RAG is brittle and you want a second look, schedule a call.
