A Unified Deep Learning and Retrieval-Augmented System for Chest X-Ray Decision Support

Abstract

Thoracic diseases remain a leading cause of global morbidity and mortality, where chest radiography plays a pivotal role in early detection and diagnosis. In this work, we present a multi-modal framework that integrates deep learning-based image classification with retrieval-augmented natural language generation to support clinicians and patients in interpreting chest radiographs. Using the NIH ChestXray14 dataset containing over 100,000 frontal-view radiographs annotated with 14 disease labels, we train a multi-label classifier based on the InceptionResNetV2 architecture. The model achieved a training accuracy of 89.8 % and validation accuracy of 89.5 %, with a macro-average ROC AUC of 0.81 across all disease categories. To make these results actionable, we extend the system with an interactive web-based interface that enables users to upload radiographs and receive diagnostic predictions. Beyond image analysis, a retrievalaugmented generation (RAG) pipeline leverages BioBERT embeddings and a FAISS-powered vector database of chunked Mayo Clinic knowledge articles enriched with metadata. This ensures precise retrieval of condition-specific medical information, which is then summarized by a Llama-3.2 (Unsloth) instruction-tuned LLM into concise, patient-friendly responses. This integration allows users to ask follow-up questions, obtain precautionary advice, and understand diagnostic limitations in a safe, informative, and user-friendly environment. The proposed framework, formalized through a unified algorithm, demonstrates the potential of combining convolutional neural networks, vector database-driven retrieval, and instruction-tuned large language models to advance automated radiology support systems. The system offers practical clinical value by providing traceable, evidence-grounded explanations, and an informal user review by medical trainees confirmed improved interpretability compared to standard classifiers.