Reproducibility and Generalizability in Radiomics Modeling: Possible Strategies in Radiologic and Statistical Perspectives

Radiomics, which involves the use of high-dimensional quantitative imaging features for predictive purposes, is a powerful tool for developing and testing medical hypotheses. Radiologic and statistical challenges in radiomics include those related to the reproducibility of imaging data, control of overfitting due to high dimensionality, and the generalizability of modeling. The aims of this review article are to clarify the distinctions between radiomics features and other omics and imaging data, to describe the challenges and potential strategies in reproducibility and feature selection, and to reveal the epidemiological background of modeling, thereby facilitating and promoting more reproducible and generalizable radiomics research.

[1]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[2]  L. Ferguson,et al.  External validity, generalizability, and knowledge utilization. , 2004, Journal of nursing scholarship : an official publication of Sigma Theta Tau International Honor Society of Nursing.

[3]  P. Lambin,et al.  Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach , 2014, Nature Communications.

[4]  H. Kim,et al.  Radiomics as a Quantitative Imaging Biomarker: Practical Considerations and the Current Standpoint in Neuro-oncologic Studies , 2018, Nuclear Medicine and Molecular Imaging.

[5]  Chaofeng Liang,et al.  A Fully-Automatic Multiparametric Radiomics Model: Towards Reproducible and Prognostic Imaging Signature for Prediction of Overall Survival in Glioblastoma Multiforme , 2017, Scientific Reports.

[6]  V. Montori,et al.  External validity, generalisability, applicability and directness: a brief primer , 2018, BMJ evidence-based medicine.

[7]  Anant Madabhushi,et al.  Radiomics Analysis on FLT-PET/MRI for Characterization of Early Treatment Response in Renal Cell Carcinoma: A Proof-of-Concept Study1 , 2016, Translational oncology.

[8]  Thomas Frauenfelder,et al.  Influence of inter-observer delineation variability on radiomics stability in different tumor sites , 2018, Acta oncologica.

[9]  Brian B. Avants,et al.  The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) , 2015, IEEE Transactions on Medical Imaging.

[10]  Howard Y. Chang,et al.  Decoding global gene expression programs in liver cancer by noninvasive imaging , 2007, Nature Biotechnology.

[11]  N. Paragios,et al.  Promises and challenges for the implementation of computational medical imaging (radiomics) in oncology , 2017, Annals of oncology : official journal of the European Society for Medical Oncology.

[12]  V. Stodden,et al.  Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals , 2013, PloS one.

[13]  M. Martel,et al.  High quality machine-robust image features: identification in nonsmall cell lung cancer computed tomography images. , 2013, Medical physics.

[14]  Andriy Fedorov,et al.  Computational Radiomics System to Decode the Radiographic Phenotype. , 2017, Cancer research.

[15]  Peter Balter,et al.  Delta-radiomics features for the prediction of patient outcomes in non–small cell lung cancer , 2017, Scientific Reports.

[16]  How Far Are We from Using Radiomics Assessment of Gliomas in Clinical Practice? , 2018, Radiology.

[17]  Azuraliza Abu Bakar,et al.  A review of feature selection techniques in sentiment analysis , 2019, Intell. Data Anal..

[18]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[19]  Zaiyi Liu,et al.  Effects of contrast-enhancement, reconstruction slice thickness and convolution kernel on the diagnostic performance of radiomics signature in solitary pulmonary nodule , 2016, Scientific Reports.

[20]  Anil K. Jain,et al.  Statistical Pattern Recognition: A Review , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Patrick Granton,et al.  Radiomics: extracting more information from medical images using advanced feature analysis. , 2012, European journal of cancer.

[22]  Pedro Larrañaga,et al.  A review of feature selection techniques in bioinformatics , 2007, Bioinform..

[23]  Christopher Joseph Pal,et al.  Brain tumor segmentation with Deep Neural Networks , 2015, Medical Image Anal..

[24]  Olivier Saut,et al.  T2‐based MRI Delta‐radiomics improve response prediction in soft‐tissue sarcomas treated by neoadjuvant chemotherapy. , 2019, Journal of magnetic resonance imaging : JMRI.

[25]  Jinzhong Yang,et al.  Measuring Computed Tomography Scanner Variability of Radiomics Features , 2015, Investigative radiology.

[26]  Sung Tae Kim,et al.  Diffusion radiomics as a diagnostic model for atypical manifestation of primary central nervous system lymphoma: development and multicenter external validation , 2018, Neuro-oncology.

[27]  Lawrence H. Schwartz,et al.  Assessing Agreement between Radiomic Features Computed for Multiple CT Imaging Settings , 2016, PloS one.

[28]  Andre Dekker,et al.  Radiomics: the process and the challenges. , 2012, Magnetic resonance imaging.

[29]  Maria Antonietta Mazzei,et al.  The role of delta radiomics in gastric cancer. , 2018, Quantitative imaging in medicine and surgery.

[30]  E. V. van Beek,et al.  Radiomics and its emerging role in lung cancer research, imaging biomarkers and clinical management: State of the art. , 2017, European journal of radiology.

[31]  Cheng Li,et al.  Adjusting batch effects in microarray expression data using empirical Bayes methods. , 2007, Biostatistics.

[32]  Robert J. Gillies,et al.  Test–Retest Reproducibility Analysis of Lung CT Image Features , 2014, Journal of Digital Imaging.

[33]  H. Aerts,et al.  Applications and limitations of radiomics , 2016, Physics in medicine and biology.

[34]  S. Choi,et al.  Incorporating diffusion- and perfusion-weighted MRI into a radiomics model improves diagnostic performance for pseudoprogression in glioblastoma patients , 2018, Neuro-oncology.

[35]  Walter Noordzij,et al.  Prediction of Response to Neoadjuvant Chemotherapy and Radiation Therapy with Baseline and Restaging 18F-FDG PET Imaging Biomarkers in Patients with Esophageal Cancer. , 2018, Radiology.

[36]  Paul Kinahan,et al.  Radiomics: Images Are More than Pictures, They Are Data , 2015, Radiology.

[37]  Ginu A. Thomas,et al.  A Coclinical Radiogenomic Validation Study: Conserved Magnetic Resonance Radiomic Appearance of Periostin-Expressing Glioblastoma in Patients and Xenograft Models , 2018, Clinical Cancer Research.

[38]  John D. Storey The positive false discovery rate: a Bayesian interpretation and the q-value , 2003 .

[39]  R. Jeraj,et al.  Variability of textural features in FDG PET images due to different acquisition modes and reconstruction parameters , 2010, Acta oncologica.

[40]  Yoav Benjamini,et al.  Identifying differentially expressed genes using false discovery rate controlling procedures , 2003, Bioinform..

[41]  Galit Shmueli,et al.  To Explain or To Predict? , 2010, 1101.0891.

[42]  Vicky Goh,et al.  The precision of textural analysis in 18F-FDG-PET scans of oesophageal cancer , 2015, European Radiology.

[43]  Brigid Wilson,et al.  Implementing Reproducible Research , 2014 .

[44]  H. Barnhart,et al.  The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions , 2015, Statistical methods in medical research.

[45]  K. Aldape,et al.  Identification of noninvasive imaging surrogates for brain tumor gene-expression modules , 2008, Proceedings of the National Academy of Sciences.

[46]  P. Lambin,et al.  Robust Radiomics Feature Quantification Using Semiautomatic Volumetric Segmentation , 2014, PloS one.

[47]  C.J.H. Mann,et al.  Clinical Prediction Models: A Practical Approach to Development, Validation and Updating , 2009 .

[48]  R. Gillies,et al.  EP-2132: Repeatability and reproducibility of radiomic features: results of a systematic review , 2018 .

[49]  Christine M. Micheel,et al.  COMMITTEE ON THE REVIEW OF OMICS-BASED TESTS FOR PREDICTING PATIENT OUTCOMES IN CLINICAL TRIALS , 2012 .

[50]  Ji-Hong Hong,et al.  Diffusion radiomics analysis of intratumoral heterogeneity in a murine prostate cancer model following radiotherapy: Pixelwise correlation with histology , 2017, Journal of magnetic resonance imaging : JMRI.

[51]  A. Frigessi,et al.  Principles and methods of integrative genomic analyses in cancer , 2014, Nature Reviews Cancer.

[52]  Stuart A. Taylor,et al.  Imaging biomarker roadmap for cancer studies , 2016, Nature Reviews Clinical Oncology.

[53]  R. Tibshirani,et al.  Significance analysis of microarrays applied to the ionizing radiation response , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[54]  P. Lambin,et al.  Radiomics: the bridge between medical imaging and personalized medicine , 2017, Nature Reviews Clinical Oncology.

[55]  Floris H. P. van Velden,et al.  Test-Retest Variability of Various Quantitative Measures to Characterize Tracer Uptake and/or Tracer Uptake Heterogeneity in Metastasized Liver for Patients with Colorectal Carcinoma , 2014, Molecular Imaging and Biology.

[56]  Mithat Gönen,et al.  Quantitative imaging biomarkers: A review of statistical methods for technical performance assessment , 2015, Statistical methods in medical research.

[57]  S. Plevritis,et al.  Glioblastoma multiforme: exploratory radiogenomic analysis by using quantitative image features. , 2014, Radiology.

[58]  J. Canales‐Vázquez,et al.  Radiomics of CT Features May Be Nonreproducible and Redundant: Influence of CT Acquisition Parameters. , 2018, Radiology.

[59]  M. Soussan,et al.  A Postreconstruction Harmonization Method for Multicenter Radiomic Studies in PET , 2018, The Journal of Nuclear Medicine.

[60]  G. Omenn,et al.  Evolution of Translational Omics: Lessons Learned and the Path Forward , 2013 .

[61]  Justin Guinney,et al.  Impact of Bioinformatic Procedures in the Development and Translation of High-Throughput Molecular Classifiers in Oncology , 2013, Clinical Cancer Research.

[62]  Binsheng Zhao,et al.  Evaluating variability in tumor measurements from same-day repeat CT scans of patients with non-small cell lung cancer. , 2009, Radiology.

[63]  S. Park,et al.  Methodologic Guide for Evaluating Clinical Performance and Effect of Artificial Intelligence Technology for Medical Diagnosis and Prediction. , 2018, Radiology.

[64]  M. Hatt,et al.  18F-FDG PET Uptake Characterization Through Texture Analysis: Investigating the Complementary Nature of Heterogeneity and Functional Tumor Volume in a Multi–Cancer Site Patient Cohort , 2015, The Journal of Nuclear Medicine.

[65]  M. Hatt,et al.  Reproducibility of Tumor Uptake Heterogeneity Characterization Through Textural Feature Analysis in 18F-FDG PET , 2012, The Journal of Nuclear Medicine.

[66]  Ronald Boellaard,et al.  Repeatability of Radiomic Features in Non-Small-Cell Lung Cancer [18F]FDG-PET/CT Studies: Impact of Reconstruction and Delineation , 2016, Molecular Imaging and Biology.

[67]  Jiazhou Wang,et al.  Test–Retest Data for Radiomics Feature Stability Analysis: Generalizable or Study-Specific? , 2016, Tomography.

[68]  Marco De Spirito,et al.  Delta radiomics for rectal cancer response prediction with hybrid 0.35 T magnetic resonance-guided radiotherapy (MRgRT): a hypothesis-generating study for an innovative personalized medicine approach , 2018, La radiologia medica.

[69]  P. Lambin,et al.  Stability of FDG-PET Radiomics features: An integrated analysis of test-retest and inter-observer variability , 2013, Acta oncologica.

[70]  S. Park Diagnostic Case-Control versus Diagnostic Cohort Studies for Clinical Validation of Artificial Intelligence Algorithm Performance. , 2019, Radiology.

[71]  Fanny Orlhac,et al.  Validation of A Method to Compensate Multicenter Effects Affecting CT Radiomics. , 2019, Radiology.

[72]  E. Gehan,et al.  The properties of high-dimensional data spaces: implications for exploring gene and protein expression data , 2008, Nature Reviews Cancer.

[73]  Theo Stijnen,et al.  Methods for calculating sensitivity and specificity of clustered data: a tutorial. , 2012, Radiology.

[74]  Richard Bellman,et al.  Adaptive Control Processes: A Guided Tour , 1961, The Mathematical Gazette.

[75]  Qianjin Feng,et al.  Robustness of Radiomic Features in [11C]Choline and [18F]FDG PET/CT Imaging of Nasopharyngeal Carcinoma: Impact of Segmentation and Discretization , 2016, Molecular Imaging and Biology.

[76]  David T. W. Jones,et al.  Radiomic subtyping improves disease stratification beyond key molecular, clinical, and standard imaging characteristics in patients with glioblastoma , 2018, Neuro-oncology.

[77]  Vicky Goh,et al.  Challenges and Promises of PET Radiomics , 2018, International journal of radiation oncology, biology, physics.

[78]  O. Riesterer,et al.  Stability of radiomic features in CT perfusion maps , 2016, Physics in medicine and biology.