RAG in production: what actually breaks

Retrieval-augmented generation is the standard way to make a model answer questions about your own documents. It is also the thing we are most often called in to repair, because the gap between a convincing demo and a system people trust is much wider than it looks.

Failure one: retrieval is the problem, not the model

When a RAG system gives a wrong answer, the instinct is to blame the model or rewrite the prompt. Usually the model answered perfectly well using the wrong three paragraphs. If the retrieval step did not surface the right source, no amount of prompt engineering will save the answer.

Measure retrieval separately. Build a set of real questions with the documents that should be returned, and score whether they appear in the top results at all. Fix that number before touching anything else.

Failure two: chunking that cuts the meaning in half

Splitting documents every 500 characters is the default in every tutorial, and it routinely severs a table from its heading or a clause from the condition that qualifies it. The retrieved chunk is then technically relevant and practically misleading.

• Split on document structure — headings, sections, list boundaries — before falling back to length
• Keep the heading trail in each chunk so the model knows what it is reading
• Store tables whole, even when that makes them large
• Overlap adjacent chunks so a sentence spanning a boundary is not lost

Failure three: nobody deletes anything

This is the one that bites six months in. Someone updates a policy document, the new version is indexed, and the old one is still sitting there. Now retrieval returns both, and the model confidently cites last year's rules.

Failure four: no honest way to say 'I don't know'

A model asked a question with no supporting document will still produce fluent prose. Unless you explicitly instruct it to refuse, and give it a confidence signal from the retrieval step to refuse on, it will fill the gap. Users lose trust faster from one confident fabrication than from twenty honest refusals.

Failure five: it is never evaluated again

The system was tested carefully at launch and never since. Meanwhile the documents changed, the model provider shipped an update, and the questions people ask drifted. Without a regression suite that runs on a schedule, degradation is invisible until someone complains.

Cite sources in every answer. It is the cheapest trust mechanism you can build, and it turns your users into your evaluation team.

The short version

1. Measure retrieval quality on its own, before end-to-end answer quality
2. Chunk along document structure, not character counts
3. Make deletion a first-class part of the ingestion pipeline
4. Let the system refuse, and show its sources when it does not
5. Keep a regression set and run it on every change to documents, prompts or models