Where RAG systems actually fail
Almost never in the generation step. They fail in retrieval — the right document was never found, so the model answered from nothing. Most "the AI is inaccurate" complaints are retrieval problems wearing a generation costume.
This is the engineering that separates a RAG demo from a production system. Each stage has specific failure modes, and knowing them tells you where to look when quality is poor.
The pipeline, stage by stage
| Stage | Job | Failure mode |
|---|---|---|
| Ingestion | Get documents in, cleanly | Broken PDFs, lost structure |
| Chunking | Split into retrievable units | Splitting mid-thought; lost context |
| Embedding | Convert to vectors | Model mismatch with domain |
| Retrieval | Find candidates | Missing the right document entirely |
| Reranking | Order by true relevance | Skipped, so noise reaches the model |
| Assembly | Build the prompt | Too much context; instructions buried |
| Generation | Produce the answer | Ignoring context; fabricating |
Ingestion: unglamorous and decisive
Quality here caps everything downstream. Specific problems worth handling deliberately:
- PDF extraction — multi-column layouts, headers repeating into body text, tables becoming word soup.
- Table handling — tables flattened into prose lose their meaning entirely. Preserve structure or convert to a readable text form.
- Document metadata — title, section, date, source. Essential for filtering and for citations.
- Deduplication — the same policy appearing in five documents produces five near-identical chunks crowding out other content.
- Freshness — outdated documents that should have been removed are a common source of confidently wrong answers.
Read your extracted text before building anything on it. Teams routinely spend weeks tuning retrieval on a corpus where the PDF extraction mangled half the documents. Ten minutes of reading raw extracted output reveals problems that no amount of embedding tuning will fix.
Chunking: the highest-leverage decision
Chunking determines what can be retrieved at all. Strategies, roughly in order of sophistication:
- Fixed size with overlap — simple, works acceptably, splits mid-sentence.
- Recursive by separator — splits on paragraphs then sentences, respecting natural boundaries. A good default.
- Structure-aware — splitting on headings and sections, preserving hierarchy. Substantially better for structured documentation.
- Contextual — prefixing each chunk with its document and section context, so an isolated chunk still makes sense.
The cheapest improvement most RAG systems can make: prefix every chunk with its document title and section heading. A chunk reading "the limit is 30 days" is useless in isolation; "Returns Policy → Timeframes: the limit is 30 days" is retrievable and answerable. It costs almost nothing and consistently improves results.
Hybrid retrieval
Semantic search alone reliably fails on a specific and common class of query: exact identifiers. Someone searching for product code "XR-4471B" wants that exact string, and embeddings treat it as approximately similar to every other code.
- Run both semantic and keyword search.
- Merge results, typically with reciprocal rank fusion.
- Rerank the merged set to produce the final ordering.
If your users search by part number, order ID, error code or product name, pure vector search will disappoint them — and it will do so in exactly the cases where being right matters most.
Reranking
Retrieval and precision are different problems. Retrieval scans a large corpus cheaply and optimises for not missing things. Reranking examines a small candidate set expensively and orders it accurately.
Retrieve twenty candidates, rerank, keep the top three to five. This is usually better and cheaper than sending ten raw results to the model — fewer tokens, less noise, better answers.
Prompt assembly
- Instructions before context, so they are not buried under a wall of retrieved text.
- Clear delimiters separating retrieved content from instructions — this also matters for injection resistance.
- Include source identifiers so the model can cite them.
- Explicit grounding instruction — answer only from the provided context, and say when it is insufficient.
- Cap total context. More is not better; relevance density matters more than volume.
Diagnosing quality problems
| Symptom | Likely stage | First thing to check |
|---|---|---|
| Answers unrelated to the question | Retrieval | Is the right chunk in the results at all? |
| Partially right, missing detail | Chunking | Was the answer split across chunks? |
| Fails on codes and IDs | Retrieval | Add keyword search |
| Right context, wrong answer | Generation | Prompt clarity; model capability |
| Fabricates when uncertain | Generation | Grounding instruction; refusal behaviour |
| Cites the wrong source | Assembly | Source identifiers in context |
Measure each stage separately
The discipline that makes all of this tractable: score retrieval independently from generation. If retrieval recall is 60%, no prompt engineering will produce good answers — the information never arrives. Knowing which stage is failing is most of the work of fixing it.
Building or debugging a RAG system? Tell us where the answers go wrong. See our AI agent service, choosing a vector database, and evaluation frameworks.