A RAG system's wrong answer is a retrieval defect, a generation defect, or a stale index — and "hallucination" is the shared label that hides the difference. Retrieval defects live in ingestion, ranking and packing. Generation defects contradict a passage already in the prompt. A stale index is quoted faithfully and still wrong. Substitute the correct chunk into the retrieved set, replay at three positions, and the branch names itself.
Ross Jones — Founder, The Hopium Lab. Last modified 22 July 2026. Engineering account, not legal advice.
Is a wrong answer with nothing retrieved still a hallucination?
Yes, and the shared name is why teams misdiagnose. The standard taxonomy (Huang and colleagues, ACM TOIS, doi:10.1145/3703155) splits the term: a factuality hallucination contradicts real-world knowledge or cannot be verified against it, a faithfulness hallucination is unfaithful to the context supplied. Inventing an answer over an empty context is the first. Contradicting a passage sitting in the prompt is the second, filed under context inconsistency.
"We have a hallucination problem" is a shrug with a technical term attached.
Which half of the pipeline is actually to blame?
Whichever half you have not fixed yet — the split moves with corpus quality, and any universal claim about it is marketing. In my own builds, on the low-quality corpora most first attempts run on, the errors concentrate on retrieval; repair retrieval and the balance shifts to integration. That is experience, not a measured split, and anyone quoting you a percentage has not conditioned it on corpus quality. Elchafei and colleagues (April 2026, revised May 2026) traced evidence use across medical QA and HotpotQA, and found hallucinations driven less by retrieval accuracy than by how evidence is integrated. Retrieval-dominant and integration-dominant findings sit at different points on one curve.
"Check retrieval first" is not a fresh insight either. LlamaIndex's failure mode checklist says to print the retrieved nodes and confirm the answer is there; Kezhan Shi's Towards Data Science piece (15 July 2026) argued the retrieval-dominates case at length. Nobody covers the ambiguous case: chunk present, answer wrong, three plausible causes, and now what.
How do you run the replay so the result means something?
Substitute, do not insert. Force-injecting the correct chunk at the top changes presence, position and the distractor set at once, so a right answer names no cause. A substitution replay swaps the correct chunk in for the lowest-ranked chunk the retriever returned, holding k and total token count fixed and the surviving chunks in their original relative order. Then move the swapped chunk through three slots: head, middle, tail. Substitution is not free either — you have removed a distractor. It is the cheapest control there is, not a clean room. Benchmarks call the gold-chunk arm the oracle-context setting; the difference here is running it against a production incident, not a leaderboard.
Position is a live variable, not a detail. Lost in the Middle (TACL vol. 12, 2024, pp. 157–173) put the curve on record: accuracy peaks at the start or end of the window and sags in between. NoLiMa (ICML 2025) is sharper — strip literal lexical overlap between question and target, and ten of twelve models advertising 128K context fall below half their short-context baseline at 32K, GPT-4o dropping from 99.3% to 69.7%.
Add the arm everyone skips: the same question, no retrieved context. Answer correct unaided, and the exercise is measuring parametric memory. RAGChecker (NeurIPS 2024 Datasets and Benchmarks) meters the same effect inside a RAG response as Self-Knowledge: correct claims appearing in no retrieved chunk. Correct unaided but wrong on the live set is context poisoning: the retrieved noise did the damage.
Then run every arm at least five times. Temperature zero is not determinism: Thinking Machines Lab reported in September 2025 that 1,000 completions from Qwen3-235B-Instruct at temperature 0 gave 80 distinct outputs, first diverging at token 103, because inference kernels are not batch-invariant. A lab engineering blog, not peer review.
A single replay is an anecdote. One run that comes out right is a coin that landed heads.
No arm runs without capture. Can you reconstruct every input that produced the answer without calling the model again? If not, you do not have a system. You have a demo with a try/except around it. Capture like that is the replay test pointed at your own debugging, and the record-keeping an evidence layer is built on.
Where exactly does the correct chunk stop being useful?
At one of seven points, two of which get fused on every slide I see. Barnett and colleagues (CAIN 2024) separate FP2, missed the top ranked documents, from FP3, not in context: ranked below the cut, versus retrieved, then dropped when the prompt was packed. Reranking fixes the ranking bug and does nothing for the packing bug.
| Branch | What the arms show | Fix | Fix that will not work |
|---|---|---|---|
| FP1 — missing content | No chunk to substitute | Ingestion, coverage, parsing | Reranking. Nothing retrieves what was never indexed |
| FP2 — ranking | In the corpus, ranked worse than k | Embeddings, hybrid search, query rewriting, reranker | Prompt engineering |
| FP3 — consolidation | Retrieved, not in the packed prompt | Top-k, dedup, packing order, truncation policy | Better embeddings. Retrieval already worked |
| Reachability | Right at head, wrong at middle or tail | Shorter context, rerank to head or tail | Raising k. A longer haystack |
| Faithfulness | Wrong at head, middle and tail | Grounding constraints, citation-or-refuse | Reranking. The retriever already worked |
| Context poisoning | Right with no context, wrong with the live set | Precision over recall, lower k, noise filtering | Raising k. More context is the mechanism |
| Index drift | Quoted faithfully, superseded at source | Retrieval over (document_id, version_id) | Anything in the model layer |
| Non-abstention (orthogonal) | Confident answer over thin context | Logged refusal path; selective generation | A prompt asking for humility |
Can a faithfulness score make the call for you?
No, and the formula is the proof. Ragas defines faithfulness as claims in the response supported by the retrieved context, over total claims in the response. Both terms reference retrieved context. Ground truth appears in no part of the formula.
Faithfulness is scored against the context you retrieved, never against the truth. Retrieve the wrong passage, answer it perfectly, score 1.0.
A green faithfulness dashboard beside a queue of wrong-answer tickets is not a mystery: faithfulness is blind to retrieval failure by construction, so the pairing proves the defect sits upstream. RAGChecker's context utilisation is the number for "the chunk was there and the model ignored it". Almost nobody computes it.
Why does a superseded chunk survive every check?
Because retrieval worked and the index lied. A stale chunk is retrieved on merit, quoted faithfully, scores 1.0 on faithfulness, and passes a substitution replay when the substituted chunk is the stale one. Vector similarity carries no temporal dimension — a point Oracle's index-drift write-up makes with the obvious example: amend a clause from thirty days to sixty, and similarity to the query is near-identical for both versions. Vendor engineering, not peer review.
Test drift with two arms — fresh chunk alone, then fresh and stale together. Divergence is the signature. Xie and colleagues (ICLR 2024 Spotlight) showed models accept coherent conflicting evidence, but show strong confirmation bias when the retrieved set also matches parametric memory. Where the superseded value sits in training data, the fresh chunk can lose.
The fix is structural: retrieve over (document_id, version_id) pairs and close the validity window on the superseded version. RAG over regulations is where the bill arrives — an amended clause, retrieved cleanly, cited confidently, wrong since the last consolidation.
If nothing usable was retrieved, is the generator off the hook?
No — a retrieval miss and a failure to refuse are two independent defects on one ticket, and where context is insufficient the correct output is a refusal. Sufficient Context (ICLR 2025) shows frontier models answer anyway, and Google's write-up (14 May 2025) has the figure: Gemma answered incorrectly on 10.2% of questions with no context at all, rising to 66.1% with insufficient context. Retrieving badly is worse than retrieving nothing.
Every retrieval branch needs an abstention check hanging off it. Fix the retriever and the second defect is still live.
Kalai and colleagues (September 2025) argue hallucination persists because grading schemes reward guessing over uncertainty, so their remedy is rescoring benchmarks, not a sprint item. Yours: make refusal a logged outcome and gate generation on a sufficiency signal. Joren and colleagues report that gating lifted the fraction of correct answers among answers actually given by 2–10%.
How does the diagnosis itself fail?
Predictably, and in ways that survive a retrospective. Name them: unnamed failure modes get no budget.
- Insertion instead of substitution. k and token count moved underneath the test.
- Position blindness. One position tested, so a packing bug is filed as a retrieval win.
- Single-shot replay. n = 1, no agreement threshold, a coin flip written up as procedure.
- Rank amnesia. Ranks unlogged, so FP2 and FP3 look identical and both get a reranker.
- Parametric pass. No no-context arm, so memory scores as a retrieval win.
- Corpus-gap denial. Weeks of tuning against content nobody ingested.
- Faithfulness laundering. A metric blind to retrieval failure, promoted to verdict.
- Abstention amnesia. Refusal unlogged, so misses become confident answers.
What does the decision tree look like?
Seven steps, one branch name per incident. Fork it, run it.
RAG WRONG-ANSWER TRIAGE — DECISION TREE v1.0 (22 July 2026)
The Hopium Lab. Not legal advice.
PREREQUISITE, logged per request:
[ ] query as received
[ ] retrieved chunk IDs, in order, with scores AND ranks
[ ] packed prompt verbatim
[ ] model id + version, decoding params, answer
0 Correct answer in the index?
NO -> FP1 MISSING CONTENT. Stop. Fix ingestion.
YES -> call it GOLD. Check GOLD against the live source:
superseded -> INDEX DRIFT (step 4).
1 ARMS — same question, template, model, decoding params.
n >= 5 each, majority verdict.
A NO-CONTEXT no retrieved context at all
B LIVE the context production served
C/D/E SUBSTITUTION REPLAY: swap GOLD in for the lowest-
ranked chunk the retriever returned, holding k and
total token count fixed and the surviving chunks
in their original relative order. GOLD at head (C),
middle (D), tail (E).
2 Compare arm A with arm B.
A right, B right -> parametric memory. INCONCLUSIVE;
re-run on a harder question.
A right, B wrong -> CONTEXT POISONING. The retrieved set
is worse than no context at all.
A wrong -> step 3.
3 In arm B, where did GOLD stop?
never retrieved, rank > k -> FP2 RANKING
retrieved, not in the prompt -> FP3 CONSOLIDATION
in the prompt -> step 4
4 GOLD in the prompt, answer wrong.
C right, D or E wrong -> REACHABILITY / PACKING
wrong at C, D and E -> FAITHFULNESS
INDEX DRIFT — arms F (fresh alone), G (fresh + stale):
F right, G wrong -> DRIFT + KNOWLEDGE CONFLICT
5 ABSTENTION CHECK, at EVERY branch above:
[ ] with insufficient context, did it refuse?
[ ] is the refusal logged, or invisible?
A confident answer over insufficient context is a SECOND defect.
6 Write the branch name on the ticket. One per incident.
A branch name is triage, not taxonomy. The current published taxonomy (TrustNLP 2026) runs to 33 failure modes across seven pipeline stages, and grades 12 as lacking peer-reviewed evidence. Start with two branches, get to seven.
A ticket that does not name a branch is a guess with a number on it.