RAG in Production: What Actually Breaks and How to Fix It

Retrieval-augmented generation is the fastest way to put a company's knowledge in front of an LLM, and one of the easiest systems to get subtly wrong. Demos impress in an afternoon; production breaks for reasons that have little to do with the model. These are the five failure modes we meet most often, and what actually fixes them.

1. Retrieval returns plausible, useless chunks

The vector search happily surfaces text that mentions the topic without answering the question. Users see confident answers assembled from the wrong passages.

The fix is retrieval quality work, not a bigger model: chunk by document structure (sections, not fixed character windows), keep headings with their bodies, and add a reranking step so the top ten candidates get re-scored against the actual question. Measure retrieval separately from generation: if the right passage is not in the top five, no prompt will save you.

2. Stale knowledge

Documents change; embeddings do not. Last quarter's pricing document answers this quarter's question with total confidence.

Treat your index as a live system: content should carry an owner and a freshness policy, ingestion should be incremental and event-driven where possible, and every chunk should store its source URL and modified date so answers can prefer recent material. A scheduled re-index is maintenance, not an option.

3. The model answers beyond the evidence

When retrieval misses, a helpful model improvises. In an enterprise setting, improvisation is a liability.

Constrain generation to the retrieved context, instruct the model to say when the answer is not in the documents, and show citations inline. Users forgive "I couldn't find that in the knowledge base"; they do not forgive a fabricated policy. Confidence should come from evidence, not tone.

4. No evaluation beyond vibes

Teams demo changes instead of measuring them, so quality drifts invisibly. The fix is an evaluation harness built in the first sprint, not after the first incident:

• A golden set of real questions with known correct sources and answers.
• Retrieval metrics (is the right chunk retrieved?) scored separately from answer metrics (is the answer correct and grounded?).
• Regression runs on every prompt, model or chunking change, exactly like unit tests.

Cost and latency belong in the harness too. A system that got 5% better answers while tripling spend is not an improvement.

5. Permissions are an afterthought

In a demo, everyone can see every document. In production, a chatbot that leaks a salary spreadsheet through a cleverly phrased question is an incident. Filters must enforce access at retrieval time (per user, per tenant, per document) and be tested with adversarial prompts. Security review is part of the definition of done.

The pattern behind the fixes

RAG is not a model problem; it is an information problem with a model attached. Teams that treat ingestion, retrieval, evaluation and permissions as first-class engineering ship systems that keep working after the demo glow fades. That is the difference between an AI feature and an AI liability.