Deep Learning for Multimodal Breast Cancer Characterization with Emergence of Terahertz and Infrared Imaging

While the significant progress has been made in multimodal image data fusion tasks, the application of deep learning for multisource processing of terahertz (THz) and infrared (IR) image datasets has not yet been explored. It is through deep learning based multisource data fusion and computational modelling that outcomes of interest such as smart clinical decision support systems ultimately facilitate improved patient level decision making. In this paper, we are focusing on multisource medical image data processing streamlined to the disease classification and characterization tasks. Specifically, we propose the framework and algorithms for multimodal deep learning-based models for classifying THz and IR pixel multimodal datasets for the breast cancer application. Initially, a brief overview of the principles of multimodal data fusion, the dynamics & interaction mechanisms in biological systems as well as parameter extraction and modelling in both THz and IR thermography is reported. A deep learning based multimodal framework for breast cancer classification and characterization in IR thermography and THz imaging is proposed. Further, for proof of concept the decision level fusion is performed on IR thermography dataset to test the model performance. The model performance is evaluated using the metrics which are achieved as: validation accuracy 96.6% and sensitivity 96.3%. The confusion matrix and receiver operating characteristics are computed with Area Under the Curve (AUC) of 0.99. Lastly, a roadmap for future research is proposed through recent trends.