Engineering· 7 min read
What breaks when RAG goes to production, and how we fix it
Retrieval demos are easy. Keeping answers accurate when documents change every week is the real work. Here is the checklist we run on every knowledge system.
LogicGridLab Engineering
Applied AI team
A retrieval augmented generation demo takes an afternoon. A knowledge system that stays accurate for a year takes discipline. These are the failure modes we see most often, and what we do about each one.
1. Stale documents win
The most common bug is not a hallucination. It is the model faithfully quoting an article that was archived last month. We tag every chunk with its source, version and last edited date, and drop anything that has been archived or replaced before it reaches the model.
2. Chunks without context
A paragraph that says "this applies to all orders" means nothing without its heading. We store the heading path with every chunk and include it in the prompt, which fixes a surprising share of wrong answers on its own.
3. No benchmark, no progress
Before we change anything, we collect 200 to 300 real questions with approved answers. Every release runs against that set, and we publish the score in the client portal. If accuracy drops, the release does not ship.
4. Permissions as an afterthought
Retrieval must respect the same access rules as the source system. We filter by permission at query time, never after the model has already seen the text.
Our production checklist
- Source, version and freshness on every chunk
- Heading path stored with the text
- A benchmark of real questions, run on every release
- Permission filters applied before retrieval
- Citations on every answer, so people can check the source
None of this is glamorous, but it is the difference between a demo and a system your team relies on.
