The possibility of the combination of OCT and fundus images for improving the diagnostic accuracy of deep learning for age-related macular degeneration: a preliminary experiment

AbstractRecently, researchers have built new deep learning (DL) models using a single image modality to diagnose age-related macular degeneration (AMD). Retinal fundus and optical coherence tomography (OCT) images in clinical settings are the most important modalities investigating AMD. Whether concomitant use of fundus and OCT data in DL technique is beneficial has not been so clearly identified. This experimental analysis used OCT and fundus image data of postmortems from the Project Macula. The DL based on OCT, fundus, and combination of OCT and fundus were invented to diagnose AMD. These models consisted of pre-trained VGG-19 and transfer learning using random forest. Following the data augmentation and training process, the DL using OCT alone showed diagnostic efficiency with area under the curve (AUC) of 0.906 (95% confidence interval, 0.891–0.921) and 82.6% (81.0–84.3%) accuracy rate. The DL using fundus alone exhibited AUC of 0.914 (0.900–0.928) and 83.5% (81.8–85.0%) accuracy rate. Combined usage of the fundus with OCT increased the diagnostic power with AUC of 0.969 (0.956–0.979) and 90.5% (89.2–91.8%) accuracy rate. The Delong test showed that the DL using both OCT and fundus data outperformed the DL using OCT alone (P value < 0.001) and fundus image alone (P value < 0.001). This multimodal random forest model showed even better performance than a restricted Boltzmann machine (P value = 0.002) and deep belief network algorithms (P value = 0.042). According to Duncan’s multiple range test, the multimodal methods significantly improved the performance obtained by the single-modal methods. In this preliminary study, a multimodal DL algorithm based on the combination of OCT and fundus image raised the diagnostic accuracy compared to this data alone. Future diagnostic DL needs to adopt the multimodal process to combine various types of imaging for a more precise AMD diagnosis. Graphical abstractThe basic architectural structure of the tested multimodal deep learning model based on pre-trained deep convolutional neural network and random forest using the combination of OCT and fundus image.