Building RAG systems that stay grounded in production

A retrieval-augmented generation system is easy to demo and hard to keep honest. The demo version works because you ask it the three questions you built it around. The production version faces a thousand questions a week, most of them phrased badly, many of them about documents that changed last Tuesday, and a handful about things your corpus simply does not cover.

Grounding is what separates the two. A grounded answer is one you can trace back to a specific passage in a specific document. An ungrounded answer is a guess wearing a confident tone. This is how we build for the first case and design around the second.

Grounding is a system property, not a prompt

The most common mistake we see in inherited RAG projects is treating grounding as a prompt engineering problem. The system prompt says “only answer using the provided context, and say you don’t know if the context is insufficient.” Then everyone is surprised when the model answers anyway.

It answers anyway because the instruction competes with everything else pulling on the model: the user’s phrasing, the model’s own parametric knowledge, and the fact that a partial match in the retrieved context looks close enough. A prompt is a preference, not a constraint. If the wrong passage reaches the model, no amount of instruction reliably saves you.

So the work moves upstream. Most of the accuracy in a RAG system is decided before the model generates a single token.

Retrieval is where accuracy is won or lost

Chunk for meaning, not for token count

Fixed-size chunking — 512 tokens with a 50-token overlap — is the default in every tutorial and it is almost always wrong for real documents. It splits tables from their headers, separates a policy clause from the condition that governs it, and strips the heading that told you which product section you were reading.

We chunk structurally instead. Split on the document’s own boundaries: headings, list groups, table rows, clause numbers. Then attach the ancestry to every chunk as a prefix or as metadata, so a fragment about “the 30-day window” still knows it lives under “Refunds → Digital goods.” A chunk that cannot be understood on its own will be retrieved out of context and answered out of context.

Two rules we hold to: never split a table away from its header row, and keep a chunk small enough that a reranker can judge it in one pass but large enough to contain a complete thought.

Hybrid search beats pure vector search

Embeddings are excellent at paraphrase and terrible at exact tokens. Ask a vector-only index for “error PGRST301” and it will cheerfully return passages about authentication errors in general, because that is what the phrase means semantically. Ask it for a part number, a version string, a person’s surname, or a statute reference and you get the same soft failure.

Running BM25 or another lexical index alongside the vector index and fusing the results fixes the entire category. Reciprocal rank fusion is the cheapest way to combine them and needs no tuning to be a clear improvement over either list alone:

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:            # one list per retriever
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

candidates = rrf([vector_search(q, 50), bm25_search(q, 50)])[:25]
passages   = rerank(q, candidates)[:6]     # cross-encoder, then cut hard

Rerank, then cut hard

Retrieve wide and pass narrow. A bi-encoder embedding search is fast because it compares the query and the document independently, which is also why it is imprecise. A cross-encoder reads the query and the passage together and scores the actual match. It is too slow to run over the whole corpus and perfectly affordable over 25 candidates.

The instinct to stuff twenty passages into the context window to be safe is the wrong one. More marginal context does not make a model more careful — it gives it more material to blend and more opportunity to pull a plausible sentence from an irrelevant document. Six good passages beat twenty mediocre ones on every evaluation we have run.

The other half of reranking is the score threshold. If the best passage scores below your floor, the correct behaviour is to return nothing to the model and tell the user the corpus does not cover the question. A system that can say “I don’t have that” is worth more than one that is right slightly more often but fails silently.

Make the model cite, and then check the citation

Ask for citations and you will get citations. That is the problem: the model will happily attach [2] to a sentence it invented, because the format was requested and the format is easy to produce.

Citations only mean something if something verifies them. The cheap version is a post-generation check: for each cited claim, confirm the passage was actually in the context you supplied, and run an entailment check between the sentence and its cited passage. Claims that fail get flagged, dropped, or sent back for a second pass.

Give each retrieved passage a stable ID at retrieval time, require the model to cite by that ID, and reject IDs that were never sent. It sounds obvious. It catches a surprising amount.

Build the evaluation set before you build the demo

This is the step teams skip, and skipping it is why RAG projects stall at “it seems better.” Without a fixed evaluation set, every change is a vibe check, regressions are invisible, and nobody can say whether last week’s prompt tweak helped.

You do not need thousands of examples. A hundred and fifty real questions, written down with their expected source passages, will tell you more than any amount of manual clicking. Build it from actual support tickets and search logs rather than questions you invented, and deliberately include the hard cases:

• Answerable questions with one clear source passage.
• Multi-hop questions that need two or more documents combined.
• Unanswerable questions the corpus genuinely does not cover — the model should decline, and you should measure how often it does.
• Near-miss questions where a plausible but wrong passage exists, which is where hallucination actually happens.
• Stale questions whose correct answer changed when a document was updated.

Then measure the two stages separately. Retrieval quality is recall at k — was the right passage in what you fetched? Generation quality is faithfulness — did the answer stay inside what was fetched? Blended end-to-end scores hide which half is broken, and the fixes for the two halves are completely different.

Guardrails for the answers you cannot ground

Some questions should never reach the model. Others should reach it with the brakes on. In practice we rely on a short list of hard constraints rather than clever prompting:

• An abstention path. Below the relevance threshold, return a “not covered” response with a route to a human. Track the abstention rate; if it is zero, your threshold is not doing anything.
• Permission filters inside the query, not after it. Filter by the user’s access rights in the retrieval call itself. Retrieving a document and then hiding it is a leak waiting for a bug.
• Freshness metadata on every chunk. Show the source date in the answer. Users forgive an old answer they can see is old, and they do not forgive a confident one that turns out to be from a superseded policy.
• Bounded scope. Off-topic questions get a polite redirect, not a best effort. A support assistant that answers general trivia is a support assistant that will eventually answer a legal question.

What we watch after launch

A RAG system decays quietly. Documents get updated, product names change, and the questions people ask drift away from the ones you evaluated. The signals worth putting on a dashboard from day one:

• Abstention rate over time — a sudden drop usually means an indexing job broke and the retriever is returning noise that clears the threshold.
• Retrieval score distribution — the shape shifting is an early warning that the corpus or the question mix has changed.
• Questions with no good match, clustered weekly. This is the highest-value list in the whole system: it is your documentation backlog, written by your users.
• Answers users copied, retried, or escalated to a human.

A realistic starting point

If you are beginning a RAG build, the order that has served us best is: structural chunking with ancestry metadata, hybrid retrieval fused with RRF, a cross-encoder reranker with a real threshold, verified citations, and an evaluation set of a hundred and fifty real questions written before any of it ships. Everything else — query rewriting, multi-hop agents, fine-tuning, knowledge graphs — is a refinement on that base and should be justified by a number that moves on your evaluation set.

Grounding is not a feature you add at the end. It is the set of decisions you make about what the model is allowed to see, what it is allowed to say, and what happens when it does not know. Get those right and the model has a much easier job.

Working on a retrieval system that is not behaving? Tell us what it is getting wrong and we will tell you where we would look first.