Influence of inter-observer delineation variability on radiomics stability in different tumor sites

Abstract Background: Radiomics is a promising methodology for quantitative analysis and description of radiological images using advanced mathematics and statistics. Tumor delineation, which is still often done manually, is an essential step in radiomics, however, inter-observer variability is a well-known uncertainty in radiation oncology. This study investigated the impact of inter-observer variability (IOV) in manual tumor delineation on the reliability of radiomic features (RF). Methods: Three different tumor types (head and neck squamous cell carcinoma (HNSCC), malignant pleural mesothelioma (MPM) and non-small cell lung cancer (NSCLC)) were included. For each site, eleven individual tumors were contoured on CT scans by three experienced radiation oncologists. Dice coefficients (DC) were calculated for quantification of delineation variability. RF were calculated with an in-house developed software implementation, which comprises 1404 features: shape (n = 18), histogram (n = 17), texture (n = 137) and wavelet (n = 1232). The IOV of RF was studied using the intraclass correlation coefficient (ICC). An ICC >0.8 indicates a good reproducibility. For the stable RF, an average linkage hierarchical clustering was performed to identify classes of uncorrelated features. Results: Median DC was high for NSCLC (0.86, range 0.57–0.90) and HNSCC (0.72, 0.21–0.89), whereas it was low for MPM (0.26, 0–0.9) indicating substantial IOV. Stability rate of RF correlated with DC and depended on tumor site, showing a high stability in NSCLC (90% of total parameters), acceptable stability in HNSCC (59% of total parameters) and low stability in MPM (36% of total parameters). Shape features showed the weakest stability across all tumor types. Hierarchical clustering revealed 14 groups of correlated and stable features for NSCLC and 6 groups for both HNSCC and MPM. Conclusion: Inter-observer delineation variability has a relevant influence on radiomics analysis and is strongly influenced by tumor type. This leads to a reduced number of suitable imaging features.

[1]  Arjan Bel,et al.  Definition of gross tumor volume in lung cancer: inter-observer variability. , 2002, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[2]  L. R. Dice.  Measures of the Amount of Ecologic Association Between Species , 1945 .

[3]  Ronald Boellaard,et al.  Repeatability of Radiomic Features in Non-Small-Cell Lung Cancer [18F]FDG-PET/CT Studies: Impact of Reconstruction and Delineation , 2016, Molecular Imaging and Biology.

[4]  M. V. van Herk,et al.  Irradiation of paranasal sinus tumors, a delineation and dose comparison study. , 2002, International journal of radiation oncology, biology, physics.

[5]  S. Walter,et al.  Sample size and optimal designs for reliability studies. , 1998, Statistics in medicine.

[6]  Matthias Guckenberger,et al.  Comparison of PET and CT radiomics for prediction of local tumor control in head and neck squamous cell carcinoma , 2017, Acta oncologica.

[7]  Michael Bremer,et al.  The delineation of target volumes for radiotherapy of lung cancer patients. , 2009, Radiotherapy and Oncology.

[8]  P. Lambin,et al.  Stability of FDG-PET Radiomics features: An integrated analysis of test-retest and inter-observer variability , 2013, Acta oncologica.

[9]  M. Hatt,et al.  Robustness of intratumour 18F-FDG PET uptake heterogeneity quantification for therapy response prediction in oesophageal carcinoma , 2013, European Journal of Nuclear Medicine and Molecular Imaging.

[10]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[11]  Philippe Lambin,et al.  Quantitative radiomics studies for tissue characterization: a review of technology and methodological procedures , 2017, The British journal of radiology.

[12]  H. Aerts,et al.  Applications and limitations of radiomics , 2016, Physics in medicine and biology.

[13]  E. R. van den Heuvel,et al.  3D Variation in delineation of head and neck organs at risk , 2012, Radiation oncology.

[14]  P. Lambin,et al.  Robust Radiomics Feature Quantification Using Semiautomatic Volumetric Segmentation , 2014, PloS one.

[15]  Bernard Dubray,et al.  Conformal radiotherapy for lung cancer: different delineation of the gross tumor volume (GTV) by radiologists and radiation oncologists. , 2002, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[16]  M. V. van Herk,et al.  Target delineation variability and corresponding margins of peripheral early stage NSCLC treated with stereotactic body radiotherapy. , 2015, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[17]  Ender Konukoglu,et al.  Post-radiochemotherapy PET radiomics in head and neck cancer - The influence of radiomics implementation on the reproducibility of local control tumor models. , 2017, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[18]  S Senan,et al.  Evaluation of a target contouring protocol for 3D conformal radiotherapy in non-small cell lung cancer. , 1999, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[19]  W. Tomé,et al.  Variations in target delineation for head and neck IMRT: An international multi-institutional study , 2004 .

[20]  Patrick Granton,et al.  Radiomics: extracting more information from medical images using advanced feature analysis. , 2012, European journal of cancer.

[21]  R. Cowan,et al.  Assessing the effect of a contouring protocol on postprostatectomy radiotherapy clinical target volumes and interphysician variation. , 2009, International journal of radiation oncology, biology, physics.

[22]  Terry K Koo,et al.  A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. , 2016, Journal of chiropractic medicine.

[23]  Niko Papanikolaou,et al.  Prospective randomized double-blind pilot study of site-specific consensus atlas implementation for rectal cancer target volume delineation in the cooperative group setting. , 2011, International journal of radiation oncology, biology, physics.

[24]  J. Rich.  Cancer stem cells: understanding tumor hierarchy and heterogeneity , 2016, Medicine.

[25]  Dimitris Visvikis,et al.  Characterization of PET/CT images using texture analysis: the past, the present… any future? , 2016, European Journal of Nuclear Medicine and Molecular Imaging.

[26]  G. Botti,et al.  Limits and potential of targeted sequencing analysis of liquid biopsy in patients with lung and colon carcinoma , 2016, Oncotarget.

[27]  A. Riegel,et al.  Variability of gross tumor volume delineation in head-and-neck cancer using CT and PET/CT fusion. , 2005, International journal of radiation oncology, biology, physics.