When does retrieval help medical question answering? A question-type sensitivity analysis and failure taxonomy for text-based (image-free) medical RAG, with an interpretable clinical filter (CAF-RAG)

Abstract

Retrieval-augmented generation (RAG) is widely assumed to improve medical question answering, yet the conditions under which retrieval helps, leaves performance unchanged, or actively harms it remain poorly characterized. This paper is a diagnostic study of that question rather than the proposal of a new retrieval framework. We evaluate a deliberately text-only (image-free) pipeline in which the model receives the question and retrieved text but never the medical image, using SLAKE and VQA-RAD as sources of clinically diverse question types, and we validate externally on PubMedQA ( 𝑛 = 1 0 0 0 ). We compare six retrieval conditions across two language models. The conditions are parametric-only No-RAG, dense RAG, hybrid BM25 + dense, two cross-encoder rerankers (general and biomedical), and CAF-RAG, a training-free clinical filter whose auditable keep/remove decisions make it a diagnostic instrument. The two models are GPT-4o-mini and the open medical model BioMistral-7B, which read byte-identical retrieved context. Our central finding is negative and consistent: text-based retrieval does not significantly outperform parametric knowledge on any group we tested, and on external data it is significantly worse (PubMedQA, −0.120, 𝑝 < 0 . 0 0 0 1 ). No passage-selection method recovers it: neither reranker significantly beats plain RAG, and CAF-RAG is statistically indistinguishable from a cross-encoder reranker (＋0.002, 𝑝 = 0 . 9 1 ) while recovering part of the accuracy that retrieval costs (＋0.034 over RAG on SLAKE-CLOSED, 𝑝 = 0 . 0 2 7 ). The two models agree to within 0.002 on both effects, indicating these are properties of retrieval rather than of one model. We identify a behavioral mechanism: retrieved context displaces the model from its parametric answer prior. It moves the model toward affirmation when the context is topically related to the question (both models swing sharply toward “Yes” on SLAKE), and toward hedging when it is not (both swing toward “Maybe” on PubMedQA). We also report an audit of our own evaluation: a substring-matching artifact in lexical scoring inflated retrieval conditions more than the terse parametric baseline, and we replace it with a scorer validated against 197 blinded human gradings (91.9% agreement, 𝜅 = 0 . 7 9 ). All per-question outputs, the rebuilt knowledge base, retrieval contexts, and the scorer with its documented failure modes are released.