Diagnosing RAG Failures: Retrieval vs Generation (2026)
When an interviewer says “your RAG app gives a wrong answer — debug it,” they are testing whether you diagnose before you fix. A bad RAG answer has two very different root causes: a retrieval failure (the chunk that holds the answer never made it into the context) or a generation failure (the right chunk was retrieved, and the model still ignored or misread it). The fixes are opposite, so naming which one comes first.
The fastest diagnostic is simple: check whether the gold chunk is in the retrieved context. If it isn’t, it’s a retrieval problem — and retrieval is a recall-under-latency tuning problem. In the puzzle below, the default configuration (512-token chunks, top-5, no rerank) reaches 67.0% recall, while the tuned optimum (256-token chunks, top-5, rerank 5) reaches 84.7% under the same 138ms budget. That recall gap is exactly the retrieval failures you can fix by tuning.
That diagnosis is the interview. If retrieval already surfaces the answer and the model still gets it wrong, no amount of chunk-size tuning helps — you fix the prompt, the reranking, or the model. This guide gives you the retrieval-vs-generation decision tree, and a free interactive that drills the retrieval-tuning half against the exact optimum.
Why this question shows up in AI-engineering interviews
“It sometimes gives wrong answers” is the most common RAG complaint, so debugging it is a staple — disguised as:
- “Your RAG bot hallucinated — walk me through how you’d debug it.”
- “Recall is fine but answers are still wrong — now what?”
- “How do you tell a retrieval problem from a generation problem?”
- “Where would you add an eval to catch this class of failure?”
Each rewards a diagnosis-first habit. The single most useful check is whether the chunk containing the answer is present in the retrieved context. That one question splits every RAG failure into retrieval (fix the retriever) or generation (fix the prompt/model) — and interviewers want to see you make that split before reaching for a fix.
The decision tree
- Is the gold chunk in the retrieved context? If NO → retrieval failure. Tune chunk size, top-k, reranking, or the embedding — raise recall. This is the half the puzzle below drills.
- If YES, but the answer is still wrong → generation failure. The model had the evidence and didn’t use it: tighten the prompt, add citations/grounding, rerank so the right chunk lands where the model attends, or use a stronger model.
- Is it wrong only sometimes? → measure it. Build an eval set that includes the failing case so a fix is verifiable and a regression is caught.
- Don’t tune blind. Cranking retrieval when the failure is generation (or vice-versa) burns time and changes nothing — diagnose first.
The retrieval half is a concrete optimisation: maximise recall (how often the gold chunk is retrieved) under a latency budget, by tuning chunk size, top-k, and rerank depth. That is exactly what the puzzle scores — and where most ‘wrong answer’ bugs actually live.
The trap: fixing the wrong half
The common mistake is to jump straight to “use a better model” when retrieval never surfaced the answer — or to endlessly tune chunking when the model is the problem. Diagnose with the gold-chunk check first. In the retrieval-tuning puzzle below, moving off the default config lifts recall from 67.0% to 84.7% under the same budget — but only if retrieval is actually your bottleneck.
Practice the retrieval half: tune recall against the optimum
Diagnosing is half the skill; the other half is tuning retrieval when it’s the culprit. Below is the live RAG Pipeline puzzle from The Arena. Tune chunk size, top-k, and rerank depth to maximise recall under the latency SLA; The Arena scores you against the best feasible configuration. No login required.
What “good” looks like
When you reflexively check “is the gold chunk retrieved?” before touching anything, you debug RAG in minutes instead of hours — and you only tune retrieval when retrieval is the problem. Pushing recall from the default 67.0% toward 84.7% means you can fix the retrieval half once you’ve correctly diagnosed it.
Frequently asked
How do you tell a retrieval failure from a generation failure in RAG?
Check whether the chunk that contains the answer is present in the retrieved context. If it’s absent, it’s a retrieval failure — fix the retriever (chunking, top-k, reranking, embeddings). If it’s present but the answer is still wrong, it’s a generation failure — fix the prompt, grounding, ordering, or model.
Why does my RAG system hallucinate?
Usually one of two reasons: the relevant chunk was never retrieved (a retrieval/recall failure), or it was retrieved but the model ignored it (a generation/grounding failure). Diagnose with the gold-chunk-present check before choosing a fix, because the two require opposite changes.
How do you improve RAG retrieval recall?
Tune chunk size to its sweet spot, raise top-k to the point of diminishing returns, and add reranking — all under your latency budget. In the worked example, moving off the 512/top-5/no-rerank default lifts recall from the default toward the tuned optimum for the same budget.
What's the first step in debugging a wrong RAG answer?
Determine whether it’s a retrieval or a generation problem by checking if the answer-bearing chunk is in the retrieved context. That single check tells you whether to fix the retriever or the prompt/model, and prevents tuning the wrong half.
New scored, provably-optimal interview drills, now and then — no spam, unsubscribe anytime.
More interview guides
41 puzzles — RAG tuning, tool routing, agent orchestration, prompt/KV caching, memory compaction — each scored against a provably optimal answer.
Explore the puzzles →