A Novel Benchmarking Approach to Assess the Agreement among Radiomic Tools.

Background The translation of radiomic models into clinical practice is hindered by the limited reproducibility of features across software and studies. Standardization is needed to accelerate this process and to bring radiomics closer to clinical deployment. Purpose To assess the standardization level of seven radiomic software programs and investigate software agreement as a function of built-in image preprocessing (eg, interpolation and discretization), feature aggregation methods, and the morphological characteristics (ie, volume and shape) of the region of interest (ROI). Materials and Methods The study was organized into two phases: In phase I, the two Image Biomarker Standardization Initiative (IBSI) phantoms were used to evaluate the IBSI compliance of seven software programs. In phase II, the reproducibility of all IBSI-standardized radiomic features across tools was assessed with two custom Italian multicenter Shared Understanding of Radiomic Extractors (ImSURE) digital phantoms that allowed, in conjunction with a systematic feature extraction, observations on whether and how feature matches between program pairs varied depending on the preprocessing steps, aggregation methods, and ROI characteristics. Results In phase I, the software programs showed different levels of completeness (ie, the number of computable IBSI benchmark values). However, the IBSI-compliance assessment revealed that they were all standardized in terms of feature implementation. When considering additional preprocessing steps, for each individual program, match percentages fell by up to 30%. In phase II, the ImSURE phantoms showed that software agreement was dependent on discretization and aggregation as well as on ROI shape and volume factors. Conclusion The agreement of radiomic software varied in relation to factors that had already been standardized (eg, interpolation and discretization methods) and factors that need standardization. Both dependences must be resolved to ensure the reproducibility of radiomic features and to pave the way toward the clinical adoption of radiomic models. © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by Steiger in this issue.

[1]  F. Bazzoli,et al.  The Heterogeneity of Skewness in T2W-Based Radiomics Predicts the Response to Neoadjuvant Chemoradiotherapy in Locally Advanced Rectal Cancer , 2021, Diagnostics.

[2]  A. Rahmim,et al.  Standardization in Quantitative Imaging: A Multicenter Comparison of Radiomic Features from Different Software Packages on Digital Reference Objects and Patient Data Sets , 2020, Tomography.

[3]  C. Faivre-Finn,et al.  Reliability and prognostic value of radiomic features are highly dependent on choice of feature extraction platform , 2020, European Radiology.

[4]  I. El Naqa,et al.  Electron Density and Biologically Effective Dose (BED) Radiomics-Based Machine Learning Models to Predict Late Radiation-Induced Subcutaneous Fibrosis , 2020, Frontiers in Oncology.

[5]  R. Steenbakkers,et al.  The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping. , 2020, Radiology.

[6]  Georg Langs,et al.  Introduction to Radiomics , 2020, The Journal of Nuclear Medicine.

[7]  L. Strigari,et al.  Novel cancer therapies for advanced cutaneous melanoma: The added value of radiomics in the decision making process–A systematic review , 2020, Cancer medicine.

[8]  Marta Paiusco,et al.  Technical Note: An IBEX adaption towards image biomarker standardization. , 2019, Medical physics.

[9]  S. A. Gan,et al.  Comparison of radiomics tools for image analyses and clinical prediction in nasopharyngeal carcinoma , 2019, The British journal of radiology.

[10]  Ronald Boellaard,et al.  RaCaT: An open source and easy to use radiomics calculator tool , 2019, PloS one.

[11]  Steffen Löck,et al.  Assessing robustness of radiomic features by image perturbation , 2018, Scientific Reports.

[12]  R. Gillies,et al.  Repeatability and Reproducibility of Radiomic Features: A Systematic Review , 2018, International journal of radiation oncology, biology, physics.

[13]  Steffen Löck,et al.  Why validation of prognostic models matters? , 2018, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[14]  P. Lambin,et al.  Radiomics: the bridge between medical imaging and personalized medicine , 2017, Nature Reviews Clinical Oncology.

[15]  Yanqi Huang,et al.  Radiomics Signature: A Potential Biomarker for the Prediction of Disease-Free Survival in Early-Stage (I or II) Non-Small Cell Lung Cancer. , 2016, Radiology.

[16]  Paul Kinahan,et al.  Radiomics: Images Are More than Pictures, They Are Data , 2015, Radiology.