Multi-Level Multi-Modality Fusion Radiomics: Application to PET and CT Imaging for Prognostication of Head and Neck Cancer

To characterize intra-tumor heterogeneity comprehensively, we propose a multi-level fusion strategy to combine PET and CT information at the image-, matrix-and feature-levels towards improved prognosis. Specifically, we developed fusion radiomics in the context of 3 prognostic outcomes in a multi-center setting (4 centers) involving 296 head & neck cancer patients. Eight clinical parameters were first utilized to build a (1) clinical model. We also built models by extracting 127 radiomics features from (2) PET images alone; (3–8) PET and CT images fused via wavelet-based fusion (WF) using CT-weights of 0.2, 0.4, 0.6 and 0.8, gradient transfer fusion (GTF), and guided filtering-based fusion (GFF); (9) fused matrices (sumMat); (10–11) fused features constructed via feature averaging (avgFea) and feature concatenation (conFea); and finally, (12) CT images alone; above models were also expanded to include both clinical and radiomics features. Seven variations of training and testing partitions were investigated. Highest performance in 5, 6 and 5 partitions was achieved by image-level fusion strategies for RFS, MFS and OS prediction, respectively. Among all partitions, WF0.6 and WF0.8 showed significantly higher performance than CT model for RFS (C-index: 0.60 ± 0.04 vs. 0.56 ± 0.03, p-value: 0.015) and MFS (C-index: 0.71 ± 0.13 vs. 0.62 ± 0.08, p-value: 0.020) predictions, respectively. In partition CER 23 vs. 14, WF0.6 significantly outperformed Clinical model for RFS prediction (C-index: 0.67 vs. 0.53, p-value: 0.003); both avgFea and WF0.6 showed C-index of 0.64 and significantly higher than that of PET only (C-index: 0.51, p-value: 0.018 and 0.031, respectively) for OS prediction. Fusion radiomics modeling showed varying improvements compared to single modality models for different outcome predictions in different partitions, highlighting the importance of generalizing radiomics models. Image-level fusion holds potential to capture more useful characteristics.