Multi-Modal Fusion Learning For Cervical Dysplasia Diagnosis

Fusion of multi-modal information from a patient’s screening tests can help improve the diagnostic accuracy of cervical dysplasia. In this paper, we present a novel multi-modal deep learning fusion network, called MultiFuseNet, for cervical dysplasia diagnosis, utilizing multi-modal data from cervical screening results. To exploit the relations among different image modalities, we propose an Attention Mutual-Enhance (AME) module to fuse features of each modality at the feature extraction stage. Specifically, we first develop the Fused Faster R-CNN with AME modules for automatic cervix region detection and fused image feature learning, and then incorporate non-image information into the learning model to jointly learn non-linear correlations among all the modalities. To effectively train the Fused Faster R-CNN, we employ an alternating training scheme. Experimental results show the effectiveness of our method, which achieves an average accuracy of 87.4% (88.6% sensitivity and 86.1% specificity) on a large dataset, outperforming the methods using any single modality alone and the known multi-modal methods.