Radiomics feature reliability assessed by intraclass correlation coefficient: a systematic review.

Radiomics research is rapidly growing in recent years, but more concerns on radiomics reliability are also raised. This review attempts to update and overview the current status of radiomics reliability research in the ever expanding medical literature from the perspective of a single reliability metric of intraclass correlation coefficient (ICC). To conduct this systematic review, Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed. After literature search and selection, a total of 481 radiomics studies using CT, PET, or MRI, covering a wide range of subject and disease types, were included for review. In these highly heterogeneous studies, feature reliability to image segmentation was much more investigated than reliability to other factors, such as image acquisition, reconstruction, post-processing, and feature quantification. The reported ICCs also suggested high radiomics feature reliability to image segmentation. Image acquisition was found to introduce much more feature variability than image segmentation, in particular for MRI, based on the reported ICC values. Image post-processing and feature quantification yielded different levels of radiomics reliability and might be used to mitigate image acquisition-induced variability. Some common flaws and pitfalls in ICC use were identified, and suggestions on better ICC use were given. Due to the extremely high study heterogeneities and possible risks of bias, the degree of radiomics feature reliability that has been achieved could not yet be safely synthesized or derived in this review. More future researches on radiomics reliability are warranted.

[1]  Paul Kinahan,et al.  Radiomics: Images Are More than Pictures, They Are Data , 2015, Radiology.

[2]  Ginu A. Thomas,et al.  A Coclinical Radiogenomic Validation Study: Conserved Magnetic Resonance Radiomic Appearance of Periostin-Expressing Glioblastoma in Patients and Xenograft Models , 2018, Clinical Cancer Research.

[3]  Hajar Moradmand,et al.  Impact of image preprocessing methods on reproducibility of radiomic features in multimodal magnetic resonance imaging in glioblastoma , 2019, Journal of applied clinical medical physics.

[4]  Hye Rim Cho,et al.  Reproducibility of histogram and texture parameters derived from intravoxel incoherent motion diffusion-weighted MRI of FN13762 rat breast Carcinomas. , 2014, Anticancer research.

[5]  Sanjay Kalra,et al.  Reliability of 3D texture analysis: A multicenter MRI study of the brain , 2020, Journal of magnetic resonance imaging : JMRI.

[6]  S. Park,et al.  Quality of science and reporting of radiomics in oncologic studies: room for improvement according to radiomics quality score and TRIPOD statement , 2019, European Radiology.

[7]  T. Yen,et al.  Deep learning for fully automated tumor segmentation and extraction of magnetic resonance radiomics features in cervical cancer , 2019, European Radiology.

[8]  Olivier Gevaert,et al.  A Rapid Segmentation-Insensitive “Digital Biopsy” Method for Radiomic Feature Extraction: Method and Pilot Study Using CT Images of Non–Small Cell Lung Cancer , 2016, Tomography.

[9]  M. Kuo,et al.  Quality of Radiomic Features in Glioblastoma Multiforme: Impact of Semi-Automated Tumor Segmentation Software , 2017, Korean journal of radiology.

[10]  R. Steenbakkers,et al.  The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping. , 2020, Radiology.

[11]  Dimitris Visvikis,et al.  Radiomics: Data Are Also Images , 2019, The Journal of Nuclear Medicine.

[12]  How to evaluate agreement between quantitative measurements. , 2019, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[13]  Harini Veeraraghavan,et al.  Reliability of tumor segmentation in glioblastoma: impact on the robustness of MRI-radiomic features. , 2019, Medical physics.

[14]  P. Nagpal,et al.  Differentiation of non-small cell lung cancer and histoplasmosis pulmonary nodules: insights from radiomics model performance compared with clinician observers. , 2019, Translational lung cancer research.

[15]  Lin-Feng Yan,et al.  Distinguishing brain inflammation from grade II glioma in population without contrast enhancement: a radiomics analysis based on conventional MRI. , 2020, European journal of radiology.

[16]  A. Madabhushi,et al.  Stable and discriminating radiomic predictor of recurrence in early stage non-small cell lung cancer: Multi-site study. , 2020, Lung cancer.

[17]  Xin Gao,et al.  Generative adversarial network‐based super‐resolution of diffusion‐weighted imaging: Application to tumour radiomics in breast cancer , 2020, NMR in biomedicine.

[18]  Y. He,et al.  Reproducibility of radiomics features derived from intravoxel incoherent motion diffusion-weighted MRI of cervical cancer , 2020, Acta radiologica.

[19]  Laurence E Court,et al.  The Incremental Value of Subjective and Quantitative Assessment of 18F-FDG PET for the Prediction of Pathologic Complete Response to Preoperative Chemoradiotherapy in Esophageal Cancer , 2016, The Journal of Nuclear Medicine.

[20]  P. V. van Ooijen,et al.  Experimental Multicenter and Multivendor Evaluation of the Performance of PET Radiomic Features Using 3-Dimensionally Printed Phantom Inserts , 2020, The Journal of Nuclear Medicine.

[21]  S. Röhrich,et al.  Variability of computed tomography radiomics features of fibrosing interstitial lung disease: A test-retest study. , 2020, Methods.

[22]  R. Gillies,et al.  Repeatability and Reproducibility of Radiomic Features: A Systematic Review , 2018, International journal of radiation oncology, biology, physics.

[23]  Xuelei Ma,et al.  Current status and quality of radiomics studies in lymphoma: a systematic review , 2020, European Radiology.

[24]  Wolfgang Weber,et al.  Reliability of PET/CT Shape and Heterogeneity Features in Functional and Morphologic Components of Non–Small Cell Lung Cancer Tumors: A Repeatability Analysis in a Prospective Multicenter Cohort , 2016, The Journal of Nuclear Medicine.

[25]  Hidetaka Arimura,et al.  Identification of optimal mother wavelets in survival prediction of lung cancer patients using wavelet decomposition‐based radiomic features , 2018, Medical physics.

[26]  Matthias Guckenberger,et al.  Comparison of robust to standardized CT radiomics models to predict overall survival for non-small cell lung cancer patients. , 2020, Medical physics.

[27]  Valentina D. A. Corino,et al.  Assessment of the effect of intensity standardization on the reliability of T1-weighted MRI radiomic features: experiment on a virtual phantom , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[28]  Irène Buvat,et al.  Radiomics in Nuclear Medicine Applied to Radiation Therapy: Methods, Pitfalls, and Challenges. , 2018, International journal of radiation oncology, biology, physics.

[29]  Yu Sub Sung,et al.  Quantitative Computed Tomography Features for Predicting Tumor Recurrence in Patients with Surgically Resected Adenocarcinoma of the Lung , 2017, PloS one.

[30]  Dengwang Li,et al.  Reproducibility of radiomic features with GrowCut and GraphCut semiautomatic tumor segmentation in hepatocellular carcinoma , 2017 .

[31]  Sang Joon Park,et al.  CT reconstruction algorithms affect histogram and texture analysis: evidence for liver parenchyma, focal solid liver lesions, and renal cysts , 2018, European Radiology.

[32]  Chuan Huang,et al.  Robustness of radiomic features in magnetic resonance imaging: review and a phantom study , 2019, Visual Computing for Industry, Biomedicine, and Art.

[33]  Ruchika Verma,et al.  Reproducibility analysis of multi-institutional paired expert annotations and radiomic features of the Ivy Glioblastoma Atlas Project (Ivy GAP) dataset. , 2020, Medical physics.

[34]  Ioannis Sechopoulos,et al.  Deep learning-based segmentation of breast masses in dedicated breast CT imaging: Radiomic feature stability between radiologists and artificial intelligence , 2020, Comput. Biol. Medicine.

[35]  Mitsuhiro Nakamura,et al.  Comparison of radiomic features in diagnostic CT images with and without contrast enhancement in the delayed phase for NSCLC patients. , 2020, Physica medica : PM : an international journal devoted to the applications of physics to medicine and biology : official journal of the Italian Association of Biomedical Physics.

[36]  G. Moore,et al.  Leveraging Clinical Imaging Archives for Radiomics: Reliability of Automated Methods for Brain Volume Measurement. , 2017, Radiology.

[37]  H. Aerts,et al.  Applications and limitations of radiomics , 2016, Physics in medicine and biology.

[38]  Jan Hendrik Moltz,et al.  Stability of radiomic features of liver lesions from manual delineation in CT scans , 2019, Medical Imaging.

[39]  Jiazhou Wang,et al.  Reproducibility with repeat CT in radiomics study for rectal cancer , 2016, Oncotarget.

[40]  Cornelis H. Slump,et al.  Repeatability of 18F‐FDG PET radiomic features: A phantom study to explore sensitivity to image reconstruction settings, noise, and delineation method , 2018, Medical physics.

[41]  G Y Zou,et al.  Sample size formulas for estimating intraclass correlation coefficients with precision and assurance , 2012, Statistics in medicine.

[42]  A. Rahmim,et al.  Repeatability of Radiomic Features in Magnetic Resonance Imaging of Glioblastoma: Test-Retest and Image Registration Analyses. , 2020, Medical physics.

[43]  K. Miles.  Radiomics for personalised medicine: the long road ahead , 2020, British Journal of Cancer.

[44]  Xiaofeng Tao,et al.  Machine learning–based CT texture analysis to predict HPV status in oropharyngeal squamous cell carcinoma: comparison of 2D and 3D segmentation , 2020, European Radiology.

[45]  A. Benson,et al.  Detection of Immunotherapeutic Response in a Transgenic Mouse Model of Pancreatic Ductal Adenocarcinoma Using Multiparametric MRI Radiomics: A Preliminary Investigation. , 2020, Academic radiology.

[46]  J. Marescaux,et al.  Radiomics in hepatocellular carcinoma: a quantitative review , 2019, Hepatology International.

[47]  O. Riesterer,et al.  Stability of radiomic features in CT perfusion maps , 2016, Physics in medicine and biology.

[48]  J. Unkelbach,et al.  Interchangeability of radiomic features between [18F]‐FDG PET/CT and [18F]‐FDG PET/MR , 2019, Medical physics.

[49]  D. Alis,et al.  The influence of cardiac motion on radiomics features: radiomics features of non-enhanced CMR cine images greatly vary through the cardiac cycle , 2020, European Radiology.

[50]  Jianhua Ma,et al.  Radiomic signature of 18F fluorodeoxyglucose PET/CT for prediction of gastric cancer survival and chemotherapeutic benefits , 2018, Theranostics.

[51]  G. Collins,et al.  Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement , 2015, Annals of Internal Medicine.

[52]  J. E. van Timmeren,et al.  Tracking tumor biology with radiomics: A systematic review utilizing a radiomics quality score. , 2018, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[53]  Timothy F Chen,et al.  Interrater agreement and interrater reliability: key concepts, approaches, and applications. , 2013, Research in social & administrative pharmacy : RSAP.

[54]  R. Jeraj,et al.  Response-to-repeatability of quantitative imaging features for longitudinal response assessment , 2019, Physics in medicine and biology.

[55]  Yanhui Ding,et al.  Stability of MRI Radiomics Features of Hippocampus: An Integrated Analysis of Test-Retest and Inter-Observer Variability , 2019, IEEE Access.

[56]  Effect of image reconstruction algorithms on volumetric and radiomic parameters of coronary plaques. , 2018, Journal of cardiovascular computed tomography.

[57]  Vicky Goh,et al.  The effects of segmentation algorithms on the measurement of 18F-FDG PET texture parameters in non-small cell lung cancer , 2017, EJNMMI Research.

[58]  R. Cuocolo,et al.  Prostate MRI radiomics: A systematic review and radiomic quality score assessment. , 2020, European journal of radiology.

[59]  Jiangdian Song,et al.  A review of original articles published in the emerging field of radiomics. , 2020, European journal of radiology.

[60]  C. Fuller,et al.  Stability analysis of CT radiomic features with respect to segmentation variation in oropharyngeal cancer , 2019, Clinical and translational radiation oncology.

[61]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[62]  M. Ladd,et al.  Radiomics in diffusion data: a test-retest, inter- and intra-reader DWI phantom study. , 2020, Clinical radiology.

[63]  Benjamin Haibe-Kains,et al.  Vulnerabilities of radiomic signature development: The need for safeguards. , 2019, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[64]  E. S. Durmaz,et al.  Reliability of Single-Slice-Based 2D CT Texture Analysis of Renal Masses: Influence of Intra- and Interobserver Manual Segmentation Variability on Radiomic Feature Reproducibility. , 2019, AJR. American journal of roentgenology.

[65]  C. Faivre-Finn,et al.  Radiomics as a personalized medicine tool in lung cancer: Separating the hope from the hype , 2020, Lung cancer.

[66]  Abhishek Midya,et al.  Radiomic feature reproducibility in contrast-enhanced CT of the pancreas is affected by variabilities in scan parameters and manual segmentation , 2019, European Radiology.

[67]  Maciej A Mazurowski,et al.  Breast cancer MRI radiomics: An overview of algorithmic features and impact of inter‐reader variability in annotating tumors , 2018, Medical physics.

[68]  J. Canales‐Vázquez,et al.  Radiomics of CT Features May Be Nonreproducible and Redundant: Influence of CT Acquisition Parameters. , 2018, Radiology.

[69]  Olivier Gevaert,et al.  Core samples for radiomics features that are insensitive to tumor segmentation: method and pilot study using CT images of hepatocellular carcinoma , 2015, Journal of medical imaging.

[70]  M. L. Belli,et al.  Quantifying the robustness of [18F]FDG-PET/CT radiomic features with respect to tumor delineation in head and neck and pancreatic cancer patients. , 2018, Physica medica : PM : an international journal devoted to the applications of physics to medicine and biology : official journal of the Italian Association of Biomedical Physics.

[71]  Ender Konukoglu,et al.  Post-radiochemotherapy PET radiomics in head and neck cancer - The influence of radiomics implementation on the reproducibility of local control tumor models. , 2017, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[72]  P. Lambin,et al.  CT-based Radiomics for Risk Stratification in Prostate Cancer. , 2019, International journal of radiation oncology, biology, physics.

[73]  J. Lee,et al.  Evaluation of the Impact of Iterative Reconstruction Algorithms on Computed Tomography Texture Features of the Liver Parenchyma Using the Filtration-Histogram Method , 2019, Korean journal of radiology.

[74]  Qianjin Feng,et al.  Robustness versus disease differentiation when varying parameter settings in radiomics features: application to nasopharyngeal PET/CT , 2018, European Radiology.

[75]  Rakesh K. Gupta,et al.  Normative Baseline for Radiomics in Brain MRI: Evaluating the Robustness, Regional Variations, and Reproducibility on FLAIR Images , 2020, Journal of magnetic resonance imaging : JMRI.

[76]  P. Lambin,et al.  Stability of FDG-PET Radiomics features: An integrated analysis of test-retest and inter-observer variability , 2013, Acta oncologica.

[77]  Terry K Koo,et al.  A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. , 2016, Journal Chiropractic Medicine.

[78]  G. Rizzo,et al.  T2w-MRI signal normalization affects radiomics features reproducibility. , 2020, Medical physics.

[79]  Alberto Torresin,et al.  Standardization of CT radiomics features for multi-center analysis: impact of software settings and parameters , 2020, Physics in medicine and biology.