Reproducible radiomics through automated machine learning validated on twelve clinical applications

Radiomics uses quantitative medical imaging features to predict clinical outcomes. Currently, in a new clinical application, finding the optimal radiomics method out of the wide range of available options has to be done manually through a heuristic trial-and-error process. In this study we propose a framework for automatically optimizing the construction of radiomics workflows per application. To this end, we formulate radiomics as a modular workflow and include a large collection of common algorithms for each component. To optimize the workflow per application, we employ automated machine learning using a random search and ensembling. We evaluate our method in twelve di ff erent clinical applications, resulting in the following area under the curves: 1) liposarcoma (0.83); 2) desmoid-type fibromatosis (0.82); 3) primary liver tumors 4) gastrointestinal stromal tumors 5) colorectal liver metastases 6) melanoma metastases 7) hepatocellular carcinoma 8) mesenteric fibrosis (0.80); 9) prostate cancer (0.72); 10) glioma (0.71); 11) Alzheimer’s disease (0.87); and 12) head and neck cancer (0.84). We show that our framework has a competitive performance compared human experts, outperforms a radiomics baseline, and performs similar or superior to Bayesian optimization and more advanced ensemble approaches. Concluding, our method fully automatically optimizes the construction of radiomics workflows, thereby streamlining the search for radiomics biomarkers in new applications. To facilitate reproducibility and future research, we publicly release six datasets, the software implementation of our framework, and the code to reproduce this study.

[1]  F. Hutter,et al.  SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization , 2021, J. Mach. Learn. Res..

[2]  Bjoern H Menze,et al.  The Medical Segmentation Decathlon , 2021, Nature Communications.

[3]  Wiro J. Niessen,et al.  Differential Diagnosis and Molecular Stratification of Gastrointestinal Stromal Tumors on CT Images Using a Radiomics Approach , 2020, Journal of Digital Imaging.

[4]  W. Niessen,et al.  Classification of Clinically Significant Prostate Cancer on Multi-Parametric MRI: A Validation Study Comparing Deep Learning and Radiomics , 2021, Cancers.

[5]  Yan Liu,et al.  MRI-based peritumoral radiomics analysis for preoperative prediction of lymph node metastasis in early-stage cervical cancer: A multi-center study. , 2021, Magnetic resonance imaging.

[6]  S. Klein,et al.  Distinguishing pure histopathological growth patterns of colorectal liver metastases on CT using deep learning and radiomics: a pilot study , 2021, Clinical & Experimental Metastasis.

[7]  Marina De Vos,et al.  The WORC* database: MRI and CT scans, segmentations, and clinical labels for 930 patients from six radiomics studies , 2021, medRxiv.

[8]  S. Klein,et al.  Automated differentiation of malignant and benign primary solid liver lesions on MRI: an externally validated radiomics model , 2021, medRxiv.

[9]  P. Lambin,et al.  A review in radiomics: Making personalized medicine a reality via routine imaging , 2021, Medicinal research reviews.

[10]  Guoqiang Zhong,et al.  Differentiable Light-Weight Architecture Search , 2021, 2021 IEEE International Conference on Multimedia and Expo (ICME).

[11]  S. Sleijfer,et al.  The BRAF P.V600E Mutation Status of Melanoma Lung Metastases Cannot Be Discriminated on Computed Tomography by LIDC Criteria nor Radiomics Using Machine Learning , 2021, Journal of personalized medicine.

[12]  S. Klein,et al.  A Multi-Center, Multi-Vendor Study to Evaluate the Generalizability of a Radiomics Model for Classifying Prostate cancer: High Grade vs. Low Grade , 2021, Diagnostics.

[13]  P. Deyn,et al.  Cross-cohort generalizability of deep and conventional machine learning for MRI-based diagnosis and prediction of Alzheimer’s disease , 2020, NeuroImage: Clinical.

[14]  Hugo Jair Escalante,et al.  Guest Editorial: Automated Machine Learning , 2021, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Jens Petersen,et al.  nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation , 2020, Nature Methods.

[16]  S. Sleijfer,et al.  Differential diagnosis and mutation stratification of desmoid-type fibromatosis on MRI using radiomics. , 2020, European journal of radiology.

[17]  M. Dietzel,et al.  A decade of radiomics research: are images really data or just patterns in the noise? , 2020, European Radiology.

[18]  Alaa Tharwat,et al.  Classification assessment methods , 2020, Applied Computing and Informatics.

[19]  Long Jiang Zhang,et al.  Development and Validation of a Radiomics Nomogram Based on 18F-Fluorodeoxyglucose Positron Emission Tomography/Computed Tomography and Clinicopathological Factors to Predict the Survival Outcomes of Patients With Non-Small Cell Lung Cancer , 2020, Frontiers in Oncology.

[20]  S. Klein,et al.  Data belonging to Predicting the 1p/19q co-deletion status of presumed low grade glioma with an externally validated machine learning algorithm , 2020 .

[21]  Issam El Naqa,et al.  Machine and deep learning methods for radiomics. , 2020, Medical physics.

[22]  Jiangdian Song,et al.  A review of original articles published in the emerging field of radiomics. , 2020, European journal of radiology.

[23]  R. Steenbakkers,et al.  The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping. , 2020, Radiology.

[24]  Howard Bowman,et al.  I Tried a Bunch of Things: The Dangers of Unexpected Overfitting in Classification , 2016, bioRxiv.

[25]  Stefan Klein,et al.  Radiomics: Data mining using quantitative medical image features , 2020 .

[26]  Dapeng Hao,et al.  Radiomics nomogram for differentiating between benign and malignant soft‐tissue masses of the extremities , 2020, Journal of magnetic resonance imaging : JMRI.

[27]  Kathryn J Fowler,et al.  Assessing Radiology Research on Artificial Intelligence: A Brief Guide for Authors, Reviewers, and Readers-From the Radiology Editorial Board. , 2019, Radiology.

[28]  S. Sleijfer,et al.  Radiomics approach to distinguish between well differentiated liposarcomas and lipomas on MRI , 2019, The British journal of surgery.

[29]  Fanny Orlhac,et al.  The Dark Side of Radiomics: On the Paramount Importance of Publishing Negative Results , 2019, The Journal of Nuclear Medicine.

[30]  Feihong Yu,et al.  Ultrasound-based radiomics nomogram: A potential biomarker to predict axillary lymph node metastasis in early-stage invasive breast cancer. , 2019, European journal of radiology.

[31]  Qiyong Gong,et al.  Automated Machine Learning Based on Radiomics Features Predicts H3 K27M Mutation in Midline Gliomas of the Brain. , 2019, Neuro-oncology.

[32]  S. Klein,et al.  Predicting the 1p/19q Codeletion Status of Presumed Low-Grade Glioma with an Externally Validated Machine Learning Algorithm , 2019, Clinical Cancer Research.

[33]  Yi Liao,et al.  Identification of suspicious invasive placentation based on clinical MRI data using textural features and automated machine learning , 2019, European Radiology.

[34]  Martina Sollini,et al.  Towards clinical application of image mining: a systematic review on artificial intelligence and radiomics , 2019, European Journal of Nuclear Medicine and Molecular Imaging.

[35]  Stefano Trebeschi,et al.  Radiogenomics: bridging imaging and genomics , 2019, Abdominal Radiology.

[36]  Wiro J. Niessen,et al.  Classification Of Prostate Cancer: High Grade Versus Low Grade Using A Radiomics Approach , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[37]  Vishwa S. Parekh,et al.  Deep learning and radiomics in precision medicine , 2019, Expert review of precision medicine and drug development.

[38]  Ronald Boellaard,et al.  RaCaT: An open source and easy to use radiomics calculator tool , 2019, PloS one.

[39]  Konstantinos N. Plataniotis,et al.  From Handcrafted to Deep-Learning-Based Cancer Radiomics: Challenges and opportunities , 2018, IEEE Signal Processing Magazine.

[40]  Frank Hutter,et al.  Neural Architecture Search: A Survey , 2018, J. Mach. Learn. Res..

[41]  Hamed R. Bonab,et al.  Less Is More: A Comprehensive Framework for the Number of Components of Ensemble Classifiers , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[42]  Sergio Escalera,et al.  Analysis of the AutoML Challenge Series 2015-2018 , 2019, Automated Machine Learning.

[43]  Aaron Klein,et al.  Auto-sklearn: Efficient and Robust Automated Machine Learning , 2019, Automated Machine Learning.

[44]  Lars Kotthoff,et al.  Automated Machine Learning: Methods, Systems, Challenges , 2019, The Springer Series on Challenges in Machine Learning.

[45]  Massimo Bellomi,et al.  Radiomics: the facts and the challenges of image analysis , 2018, European Radiology Experimental.

[46]  Laurence E. Court,et al.  The utility of quantitative CT radiomics features for improved prediction of radiation pneumonitis , 2018, Medical physics.

[47]  Irène Buvat,et al.  LIFEx: A Freeware for Radiomic Feature Calculation in Multimodality Imaging to Accelerate Advances in the Characterization of Tumor Heterogeneity. , 2018, Cancer research.

[48]  Harini Veeraraghavan,et al.  Technical Note: Extension of CERR for computational radiomics: A comprehensive MATLAB platform for reproducible radiomics research. , 2018, Medical physics.

[49]  R. Gillies,et al.  Repeatability and Reproducibility of Radiomic Features: A Systematic Review , 2018, International journal of radiation oncology, biology, physics.

[50]  M. Field,et al.  The role of deep learning and radiomic feature extraction in cancer-specific predictive modelling: a review , 2018, Translational Cancer Research.

[51]  J. E. van Timmeren,et al.  Tracking tumor biology with radiomics: A systematic review utilizing a radiomics quality score. , 2018, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[52]  M. Hatt,et al.  IBSI: an international community radiomics standardization initiative , 2018 .

[53]  Stefan Klein,et al.  Classification of malignant and benign liver tumors using a radiomics approach , 2018, Medical Imaging.

[54]  Randal S. Olson,et al.  Benchmarking Relief-Based Feature Selection Methods , 2017, J. Biomed. Informatics.

[55]  Andriy Fedorov,et al.  Computational Radiomics System to Decode the Radiographic Phenotype. , 2017, Cancer research.

[56]  P. Lambin,et al.  Radiomics: the bridge between medical imaging and personalized medicine , 2017, Nature Reviews Clinical Oncology.

[57]  Christos Davatzikos,et al.  Brain Cancer Imaging Phenomics Toolkit (brain-CaPTk): An Interactive Platform for Quantitative Analysis of Glioblastoma , 2017, BrainLes@MICCAI.

[58]  Hyunjin Park,et al.  Classification of low-grade and high-grade glioma using multi-modal image radiomics features , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[59]  D. Dong,et al.  Radiomics Features of Multiparametric MRI as Novel Prognostic Factors in Advanced Nasopharyngeal Carcinoma , 2017, Clinical Cancer Research.

[60]  Philippe Lambin,et al.  Quantitative radiomics studies for tissue characterization: a review of technology and methodological procedures , 2017, The British journal of radiology.

[61]  Stuart A. Taylor,et al.  Imaging biomarker roadmap for cancer studies , 2016, Nature Reviews Clinical Oncology.

[62]  Fernando Nogueira,et al.  Imbalanced-learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning , 2016, J. Mach. Learn. Res..

[63]  Lars Kotthoff,et al.  Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA , 2017, J. Mach. Learn. Res..

[64]  Randal S. Olson,et al.  TPOT: A Tree-based Pipeline Optimization Tool for Automating Machine Learning , 2016, AutoML@ICML.

[65]  H. Aerts The Potential of Radiomic-Based Phenotyping in Precision Medicine: A Review. , 2016, JAMA oncology.

[66]  Yanqi Huang,et al.  Development and Validation of a Radiomics Nomogram for Preoperative Prediction of Lymph Node Metastasis in Colorectal Cancer. , 2016, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[67]  Wiro J. Niessen,et al.  Fastr: A Workflow Engine for Advanced Data Flows in Medical Image Analysis , 2016, Front. ICT.

[68]  H. Aerts,et al.  Applications and limitations of radiomics , 2016, Physics in medicine and biology.

[69]  Vishwa Parekh,et al.  Radiomics: a new application from established techniques , 2016, Expert review of precision medicine and drug development.

[70]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[71]  Nick C Fox,et al.  Magnetic resonance imaging in Alzheimer's Disease Neuroimaging Initiative 2 , 2015, Alzheimer's & Dementia.

[72]  Jinzhong Yang,et al.  IBEX: an open infrastructure software platform to facilitate collaborative work in radiomics. , 2015, Medical physics.

[73]  G. Houston,et al.  Diagnostic classification of arterial spin labeling and structural MRI in presenile early stage dementia , 2014, Human brain mapping.

[74]  P. Lambin,et al.  Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach , 2014, Nature Communications.

[75]  Hung-Ming Wang,et al.  Development and Evaluation of an Open-Source Software Package “CGITA” for Quantifying Tumor Heterogeneity with Molecular Images , 2014, BioMed research international.

[76]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[77]  Patrick Granton,et al.  Radiomics: extracting more information from medical images using advanced feature analysis. , 2012, European journal of cancer.

[78]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[79]  Cha Zhang,et al.  Ensemble Machine Learning , 2012 .

[80]  Maciej Klemm,et al.  Institute of Electrical and Electronics Engineers (IEEE) , 2011 .

[81]  Geoffrey S Ginsburg,et al.  Personalized medicine: progress and promise. , 2011, Annual review of genomics and human genetics.

[82]  L. Hood,et al.  Predictive, personalized, preventive, participatory (P4) cancer medicine , 2011, Nature Reviews Clinical Oncology.

[83]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[84]  Michal Strzelecki,et al.  MaZda - A software package for image texture analysis , 2009, Comput. Methods Programs Biomed..

[85]  Haibo He,et al.  ADASYN: Adaptive synthetic sampling approach for imbalanced learning , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[86]  Daniel S. Marcus,et al.  The extensible neuroimaging archive toolkit , 2007, Neuroinformatics.

[87]  M.A. El-Sharkawi,et al.  Pareto Multi Objective Optimization , 2005, Proceedings of the 13th International Conference on, Intelligent Systems Application to Power Systems.

[88]  Hui Han,et al.  Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning , 2005, ICIC.

[89]  Foster J. Provost,et al.  ROC confidence bands: an empirical evaluation , 2005, ICML.

[90]  Xin Yao,et al.  DIVACE: Diverse and Accurate Ensemble Learning Algorithm , 2004, IDEAL.

[91]  Rich Caruana,et al.  Ensemble selection from libraries of models , 2004, ICML.

[92]  Jasbir S. Arora,et al.  Survey of multi-objective optimization methods for engineering , 2004 .

[93]  Yoshua Bengio,et al.  Inference for the Generalization Error , 1999, Machine Learning.

[94]  Andy B. Yoo,et al.  Approved for Public Release; Further Dissemination Unlimited X-ray Pulse Compression Using Strained Crystals X-ray Pulse Compression Using Strained Crystals , 2002 .

[95]  Peter Kovesi,et al.  Phase Congruency Detects Corners and Edges , 2003, DICTA.

[96]  G. Ginsburg,et al.  The path to personalized medicine. , 2002, Current opinion in chemical biology.

[97]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[98]  Alejandro F. Frangi,et al.  Muliscale Vessel Enhancement Filtering , 1998, MICCAI.

[99]  Job Scheduling Strategies for Parallel Processing , 1996, Lecture Notes in Computer Science.

[100]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1997, EuroCOLT.

[101]  Robert Tibshirani,et al.  Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy , 1986 .

[102]  Robert C. Wolpert,et al.  A Review of the , 1985 .

[103]  R. Dennis Cook,et al.  Cross-Validation of Regression Models , 1984 .