Unified Synergistic Deep Learning Framework for Multimodal 2-D and 3-D Radiographic Data Analysis: Model Development and Validation

Recent synergistic deep learning techniques are underexplored in analyzing heterogeneous 2-D and 3-D radiographic data. Despite progress, existing heterogeneous approaches for 3-D volumetric data often rely on image-based methods, requiring manual selection of relevant slices and expert guidance. A prevailing challenge remains in harmonizing the analysis of volumetric radiographic data with variable lengths. To address these challenges, we proposed a unified deep learning-driven computer-aided diagnostic framework that analyzes multimodal 2-D and 3-D radiographic data to enhance diagnostic accuracy and aid clinical decision-making in radiology. The proposed framework primarily leverages the synergistic fusion of multi-level features within a lightweight Vision Transformer by introducing Multilevel-Multilayer Perceptron heads, which exploit and aggregate multilevel spatial features from the given input scan. A recurrent module further exploits 3D structural features, ensuring accurate decisions for volumetric data by dynamically adjusting its computation graphs to varying input lengths of volumetric radiographic data. Subsequently, a contextual map extraction module is designed to generate a well-localized activation map for the input scan, suppressing background activation from patch-level processing in the transformer module. Finally, we applied the proposed model to build a classification-driven radiographic retrieval system to retrieve relevant radiographic scans from the database that closely resemble the input test sample. We empirically validate our method on six publicly accessible radiographic datasets, including both X-ray and CT scans, demonstrating superiority (p-value <0.01) over existing alternatives. Our proposed approach outperforms existing methods, achieving notable performance metrics: 96.67% accuracy, 96.88% F1-score, and 96.75% average precision, with a true positive rate of 96.75% and a true negative rate of 97.02%. This study marks a significant advancement in automating lung infection detection through multimodal imaging.