RAG retrieval debugging: find why the wrong document wins
Inspect the evidence supplied to the model before deciding that a stronger model is the answer.
Quick answer
Debug a RAG answer by tracing retrieval separately from generation. Check whether the expected document is indexed, eligible under the caller’s filters, returned as a candidate, and retained in the final context. Compare lexical, vector and hybrid retrieval on representative questions. Then evaluate whether the answer uses that evidence correctly and completely.
On this page
When a RAG answer cites the wrong document, follow the expected document through indexing, eligibility, retrieval, ranking and final context. Repair the first stage that loses it. A more capable generation model cannot recover evidence that the retrieval pipeline never supplies. Evaluate retrieval and the final answer as separate parts of the system. [1]
A plausible answer can come from the wrong revision#
Consider an illustrative support assistant answering, “How do I rotate the key for SDK AX-17?” The corpus contains an old AX-17 guide, the current AX-17 guide and an AX-71 guide with similar wording. The answer sounds sensible but describes the retired workflow. Increasing the context window might include even more versions. Before changing a model, find out which guide was eligible, retrieved and actually supplied.
{
"questionId": "sdk-key-rotation-01",
"expectedDocument": "ax-17-guide-current",
"indexRevision": "rehearsal-index-a",
"permissionScope": "test-tenant-reader",
"eligible": true,
"candidateIds": ["ax-17-guide-old", "ax-71-guide", "ax-17-guide-current"],
"finalContextIds": ["ax-17-guide-old", "ax-71-guide"],
"failureStage": "selection-after-retrieval"
}These identifiers are invented for the example; the trace is not an Azure API response or a measured result. It makes one diagnosis possible: the desired guide reached the candidate list but disappeared before generation. Re-indexing might still be useful for another case, but this trace does not support it as the first repair. Inspect the ranking, truncation or version-selection step that sits between those two lists.
Conceptual illustration
Follow one document through the pipeline
Index
Current guide exists
Eligibility
Caller may read it
Candidates
Current guide retrieved
Final context
Current guide missing
Now change one fact: suppose the expected guide is absent from the index. The same polished answer needs a different repair—ingestion or extraction—before rank tuning matters. If the guide exists but a stale metadata value excludes it, repair the metadata or permitted filter. A trace earns its value by distinguishing these cases, not by collecting every possible score.
Where did the expected document disappear?#
Record a question and its expected supporting document, then follow that document through ingestion, eligibility, candidate selection and final context. Preserve document and chunk identifiers so the investigation does not depend on visually similar text. Record the index version and the query settings used for the failing attempt.
| Boundary | Evidence to capture | Next investigation |
|---|---|---|
| Index contents | Expected document and current revision | Ingestion, extraction or indexing |
| Eligibility | Applied metadata and permission filters | Incorrect exclusion or stale metadata |
| Candidate retrieval | Returned identifiers before later selection | Query strategy and matching |
| Final context | Chunks actually sent to generation | Ranking, truncation and duplication |
| Response | Claims tied to supplied context | Groundedness, relevance and completeness |
Keep the original question alongside a minimally changed diagnostic query. Our recommendation is to avoid rewriting the prompt so extensively that the known failing case disappears from the test set. A repair should explain the original failure and retain a regression check for it.
Does the question need exact matching as well as similarity?#
Azure AI Search hybrid queries combine text and vector retrieval and merge their results with Reciprocal Rank Fusion. Its documentation notes that product codes and specialized terms can benefit from keyword matching. This supports testing a mixed strategy; it does not prove hybrid search will win for every corpus or question. [2]
Include exact identifiers, paraphrased questions and ambiguous terms. Label the supporting document for each question. Inspect which strategy retrieves the necessary evidence instead of judging only whether the generated answer sounds fluent. Keep the rest of the pipeline fixed while testing the retrieval choice.
Do not use a raw ranking score as though it had the same meaning across retrieval modes. Our review asks which ranking stage produced it and what the final selection actually retained. A low-looking number is not by itself evidence that a document is irrelevant; evaluate the ordered results against the labelled question.
What would a useful retrieval experiment compare?#
Build a small labelled set that represents the kinds of questions the assistant must answer. Include the literal AX-17 identifier, a paraphrase that omits it, and a question that asks about an obsolete version on purpose. Each question needs an expected source and an explanation of acceptable alternatives. Otherwise the evaluator cannot distinguish a useful newer guide from a merely similar one.
| Question type | Expected evidence | Failure worth separating |
|---|---|---|
| Exact identifier | Guide for AX-17 | Similar AX-71 text outranks the exact product |
| Version-sensitive | Current guide unless an old version is requested | Retired instructions enter context |
| Paraphrased task | Relevant rotation procedure | Keyword-only query misses the wording |
| Restricted material | Only permitted documents | Recall improvement crosses a permission boundary |
For an exact identifier, compare a text path, a vector path and a supported hybrid path while keeping the corpus and other settings fixed. Azure AI Search combines text and vector retrieval using rank fusion; that is a reason to test a mixed strategy, not a universal winner. Inspect document identities and the context eventually sent to the model rather than comparing unrelated raw scores. [2]
A simple diagnostic metric can be defined without an automated judge: among questions with a labelled permitted supporting document, how often does that document appear in the selected context? Write down the denominator and the context-selection rule. This is an evaluation design, not a benchmark claimed for Dopstack. It still needs human review when several sources are valid or the expected answer is disputed.
Preserve the original failing wording. Replacing it with a beautifully explicit query can reveal a useful query-rewrite opportunity, but it does not show that the user’s actual question is fixed. Keep both versions and label the transformation. Then inspect whether the improvement helps the other cases or merely overfits the one example.
Are filters excluding the useful candidates?#
Azure AI Search documents different times for applying vector-query filters. Postfiltering narrows results after query execution, and shard-level candidate selection matters to what survives. Check the supported mode and index behavior before changing the query. Avoid copying a setting from a sample without reproducing your filtered workload. [3]
Our diagnosis separates a relevance filter from an authorization boundary. Test eligibility with controlled accounts and known documents, but do not remove tenant or permission constraints in production merely to improve apparent recall. A useful retrieval result is one that is both relevant and permitted for that caller.
In a test environment, compare the same question with a permitted narrow filter and a permitted broader scope. Record the expected eligible documents in each case. If the document disappears only under one filter, investigate its metadata and filtering stage before changing embeddings or increasing the context budget.
Is the answer grounded but incomplete?#
Groundedness and completeness are different checks. Microsoft describes groundedness in terms of the precision of a response’s support, and completeness in terms of covering expected information. A response can avoid unsupported claims while still omitting an important qualification or part of the requested answer. [1]
- Label the expected evidenceFor each representative question, identify the supporting material and what a complete answer must include. Preserve disputed cases for human review.
- Measure retrieval separatelyCheck whether the supporting document survived selection and entered the final context. Diagnose missing evidence before changing generation.
- Review the final claimsCompare the answer with the actual context, then separately check whether it covers the question. Keep unsupported additions and omissions as distinct failure categories.
- Repeat after changesUse the same labelled cases and inspect regressions. Avoid declaring success from a small group of newly chosen examples that all favor the change.
Keep a retrieval failure and an answer failure apart#
Suppose the current AX-17 guide reaches the final context, but the assistant still describes the retired workflow. The earlier selection defect is no longer the explanation. Review whether stale text remains alongside it, whether the response follows the supplied evidence and whether the prompt tells the model how to handle conflicting versions. Groundedness and completeness remain separate checks: a supported answer can still omit an essential step. [1]
Store a compact, access-controlled diagnostic record: question ID, index revision, effective permitted filters, retrieved identities, selected context identities and the evaluation verdict. Do not log full confidential chunks by default. Enough traceability to reproduce a failure is useful; an indiscriminate copy of the corpus is a different data-handling decision.
The release decision should name which failure improved and which cases were unaffected. If a filter fix restores one tenant’s document but weakens isolation, the system is worse. If a ranking change improves exact identifiers and harms paraphrases, make that trade-off visible before deploying it. “The answers seem better” cannot tell an engineer which stage is now reliable.
RAG retrieval questions#
- Should I switch models when RAG answers are wrong?
- Inspect retrieval first. If the necessary evidence is missing from the final context, a model change does not repair that upstream selection failure. Separate evidence delivery from the model’s use of evidence before choosing a change. [1]
- Is vector search always enough for technical questions?
- Not necessarily. Exact identifiers and specialized terms can need precise text matching. Azure AI Search supports hybrid retrieval, but compare strategies on your labelled questions rather than assuming one configuration is universally best. [2]
- Can filtering affect which documents are returned?
- Yes. Filter timing and candidate selection affect vector-query behavior. Inspect the supported mode, applied conditions and expected eligible documents. Keep authorization constraints intact while diagnosing relevance and metadata errors. [3]
- Can a grounded answer still be incomplete?
- Yes. Groundedness asks whether the response is supported by context; completeness asks whether expected information was covered. Evaluate both, so a careful but partial answer does not pass as a complete solution. [1]
