Uncertainty-Aware Multimodal Fusion with Spatial Attention for Robust Pneumonia Detection in Chest Radiographs

DOI: https://doie.org/10.65985/JBSE.2026972469

Rahul Burkul, Sandhya Waghere

Keywords:

Pneumonia Detection, Multimodal Learning, Chest X-ray, Deep Learning, Attention Mechanism, Uncertainty Estimation, Medical Imaging.

Abstract:

The development of deep convolutional neural networks has led to accurate automatic detection of pneumonia from chest X-ray images. However, most existing solutions employ only one source of information, namely the imagery data, while ignoring other types of data which may be helpful during the diagnostic process. Although some attempts were made to address multimodal learning tasks, straightforward approaches such as concatenation and even gated fusion do not consistently yield any improvements. The current paper presents a new framework which enables efficient multimodal fusion of chest X-ray images and patient metadata by combining two approaches, namely region-level attention and uncertainty-based gating. It allows to weight both modalities adaptively depending on the relevance of specific regions for predictions and their uncertainty level. The evaluation carried out using the CheXpert dataset shows that traditional multimodal fusion approaches bring little benefits. In contrast, the use of the proposed solution allows to achieve significantly higher results with respect to AUC (0.80), compared to the gated fusion (0.76) and image-only models (0.77).

PDF