Retrieval-augmented generation gives a language model passages from your own content before it answers. In production, most failures come from retrieval rather than the model: the wrong passages were found, permissions were applied too late, or the indexed content was out of date. Fixing RAG usually means fixing retrieval.
Retrieval is the weak link, not the model
When a RAG system gives a wrong answer, the instinct is to change the model or the prompt. Far more often the model answered correctly from the passages it was given, and the wrong passages were retrieved.
Diagnose retrieval on its own. Take a set of real questions, record which passages come back, and have someone judge whether the answer was present in them. That number — not end-to-end accuracy — tells you where the effort belongs. If the right passage is not in the top results, no amount of prompt work will fix it.
Chunking decisions have long consequences
Splitting documents into fixed-size pieces is convenient and frequently wrong. A procedure split mid-sequence retrieves as two fragments that each look partial. A table separated from its heading loses the meaning of its columns.
Chunking that follows the structure of the document — sections, clauses, procedure steps — retrieves better and reads better in the answer. It costs more to build because it depends on the document type, which is why it is often skipped and later revisited.
Permissions cannot be an afterthought
An index built over everything will happily answer a question using a document the person asking should never see. Filtering after retrieval is not enough either — the fact that a document exists can itself be sensitive.
Permissions belong in the retrieval query, evaluated against the identity of the person asking at the moment they ask. That means the index has to carry access metadata and keep it current as permissions change, which is ongoing work rather than a one-time import.
Stale content answers confidently
A model has no sense of whether the passage it was handed is current. If a policy was superseded last quarter and both versions are indexed, the answer may be delivered with the same confidence either way.
Freshness needs to be explicit: a re-index cadence matched to how quickly the source changes, deletion that actually removes content from the index, and version or date metadata surfaced in the answer so a reader can judge for themselves.
Instrument before you scale
Log the question, the retrieved passages, the answer and the feedback. Without that record, every report of a bad answer is unreproducible and every improvement is guesswork. With it, patterns appear quickly — a document type that never retrieves well, a question shape that consistently fails — and those patterns are what you fix.
