Deep Learning for Multimodal Breast Cancer Characterization With Emergence of Terahertz and Infrared Imaging

While significant progress has been made in multimodal image data fusion tasks, the application of deep learning for multisource processing of terahertz (THz) and infrared (IR) image datasets has not yet been explored. It is through deep learning-based multisource data fusion and computational modeling that outcomes of interest, such as smart clinical decision support systems, ultimately facilitate improved patient-level decision-making. In this article, we are focusing on multisource medical image data processing streamlined to the disease classification and characterization tasks. Specifically, we propose the framework and algorithms for multimodal deep learning-based models for classifying THz and IR pixel multimodal datasets for breast cancer applications. Initially, a brief overview of the principles of multimodal data fusion, the dynamics and interaction mechanisms in biological systems, as well as parameter extraction and modeling in both THz and IR thermography is reported. A deep learning-based multimodal framework for breast cancer classification and characterization in IR thermography and THz imaging is proposed. Further, for proof of concept, the decision-level fusion is performed on the IR thermography dataset to test the model performance. The model performance is evaluated using the metrics which are achieved as: validation accuracy of 96.6% and sensitivity of 96.3%. The confusion matrix and receiver operating characteristics are computed with the area under the curve (AUC) of 0.99. Last, a roadmap for future research is proposed based on recent trends.