Inter-Method Performance Study of Tumor Volumetry Assessment on Computed Tomography Test-Retest Data.

RATIONALE AND OBJECTIVES Tumor volume change has potential as a biomarker for diagnosis, therapy planning, and treatment response. Precision was evaluated and compared among semiautomated lung tumor volume measurement algorithms from clinical thoracic computed tomography data sets. The results inform approaches and testing requirements for establishing conformance with the Quantitative Imaging Biomarker Alliance (QIBA) Computed Tomography Volumetry Profile. MATERIALS AND METHODS Industry and academic groups participated in a challenge study. Intra-algorithm repeatability and inter-algorithm reproducibility were estimated. Relative magnitudes of various sources of variability were estimated using a linear mixed effects model. Segmentation boundaries were compared to provide a basis on which to optimize algorithm performance for developers. RESULTS Intra-algorithm repeatability ranged from 13% (best performing) to 100% (least performing), with most algorithms demonstrating improved repeatability as the tumor size increased. Inter-algorithm reproducibility was determined in three partitions and was found to be 58% for the four best performing groups, 70% for the set of groups meeting repeatability requirements, and 84% when all groups but the least performer were included. The best performing partition performed markedly better on tumors with equivalent diameters greater than 40 mm. Larger tumors benefitted by human editing but smaller tumors did not. One-fifth to one-half of the total variability came from sources independent of the algorithms. Segmentation boundaries differed substantially, not ony in overall volume but also in detail. CONCLUSIONS Nine of the 12 participating algorithms pass precision requirements similar to what is indicated in the QIBA Profile, with the caveat that the present study was not designed to explicitly evaluate algorithm profile conformance. Change in tumor volume can be measured with confidence to within ±14% using any of these nine algorithms on tumor sizes greater than 10 mm. No partition of the algorithms was able to meet the QIBA requirements for interchangeability down to 10 mm, although the partition comprising best performing algorithms did meet this requirement for a tumor size of greater than approximately 40 mm.

[1]  D. DeMets,et al.  Biomarkers and surrogate endpoints: Preferred definitions and conceptual framework , 2001, Clinical pharmacology and therapeutics.

[2]  A. Aisen,et al.  Effect of varying CT section width on volumetric measurement of lung tumors and application of compensatory equations. , 2003, Radiology.

[3]  A. D. Van den Abbeele,et al.  Considerations for the Use of Imaging Tools for Phase II Treatment Trials in Oncology , 2009, Clinical Cancer Research.

[4]  Andrew J. Buckler,et al.  A procedural template for the qualification of imaging as a biomarker, using volumetric CT as an example , 2009, 2009 IEEE Applied Imagery Pattern Recognition Workshop (AIPR 2009).

[5]  L. R. Dice.  Measures of the Amount of Ecologic Association Between Species , 1945 .

[6]  Lawrence H. Schwartz,et al.  Imaging Surrogates of Tumor Response to Therapy: Anatomic and Functional Biomarkers* , 2009, Journal of Nuclear Medicine.

[7]  Michael L Maitland,et al.  Volumes to Learn: Advancing Therapeutics with Innovative Computed Tomography Image Data Analysis , 2010, Clinical Cancer Research.

[8]  J. Crowley,et al.  The IASLC Lung Cancer Staging Project: Proposals for the Revision of the TNM Stage Groupings in the Forthcoming (Seventh) Edition of the TNM Classification of Malignant Tumours , 2007, Journal of thoracic oncology : official publication of the International Association for the Study of Lung Cancer.

[9]  T. Sørensen,et al.  A method of establishing group of equal amplitude in plant sociobiology based on similarity of species content and its application to analyses of the vegetation on Danish commons , 1948 .

[10]  Tim Leiner,et al.  Systematic error in lung nodule volumetry: effect of iterative reconstruction versus filtered back projection at different CT parameters. , 2012, AJR. American journal of roentgenology.

[11]  Michael F. McNitt-Gray,et al.  Determining the Variability of Lesion Size Measurements from CT Patient Data Sets Acquired under “No Change” Conditions12 , 2015, Translational oncology.

[12]  Haesun Choi,et al.  Response evaluation of gastrointestinal stromal tumors. , 2008, The oncologist.

[13]  Hiroto Hatabu,et al.  Imaging of lung cancer in the era of molecular medicine. , 2011, Academic radiology.

[14]  P. Jaccard.  THE DISTRIBUTION OF THE FLORA IN THE ALPINE ZONE.1 , 1912 .

[15]  D. Sullivan,et al.  A collaborative enterprise for multi-stakeholder participation in the advancement of quantitative imaging. , 2011, Radiology.

[16]  Kazuo Awai,et al.  Computer-aided volumetry of pulmonary nodules exhibiting ground-glass opacity at MDCT. , 2010, AJR. American journal of roentgenology.

[17]  D. Altman,et al.  Measuring agreement in method comparison studies , 1999, Statistical methods in medical research.

[18]  J M Bland,et al.  Statistical methods for assessing agreement between two methods of clinical measurement , 1986 .

[19]  Kyle J Myers,et al.  Minimum detectable change in lung nodule volume in a phantom CT study. , 2013, Academic radiology.

[20]  Kyle J Myers,et al.  Noncalcified lung nodules: volumetric assessment with thoracic CT. , 2009, Radiology.

[21]  Andrew J Buckler,et al.  The use of volumetric CT as an imaging biomarker in lung cancer. , 2010, Academic Radiology.

[22]  Ma Daqing,et al.  Volumetric measurement pulmonary ground-glass opacity nodules with multi-detector CT: effect of various tube current on measurement accuracy--a chest CT phantom study. , 2009, Academic radiology.

[23]  L. Lin,et al.  A concordance correlation coefficient to evaluate reproducibility. , 1989, Biometrics.

[24]  B. Nan,et al.  Pulmonary nodule volumetric measurement variability as a function of CT slice thickness and nodule morphology. , 2007, AJR. American journal of roentgenology.

[25]  J. Gurland,et al.  Case for using only maximum diameter in measuring tumors. , 1966, Cancer chemotherapy reports.

[26]  Binsheng Zhao,et al.  Comparison of 1D, 2D, and 3D nodule sizing methods by radiologists for spherical and complex nodules on thoracic CT phantom images. , 2014, Academic radiology.

[27]  Marcos Salganicoff,et al.  Accuracy of automated volumetry of pulmonary nodules across different multislice CT scanners , 2007, European Radiology.

[28]  Kathryn Trinkaus,et al.  Prognostic Significance of Tumor Size in Patients with Stage III Non–Small-Cell Lung Cancer: A Surveillance, Epidemiology, and End Results (SEER) Survey from 1998 to 2003 , 2012, Journal of thoracic oncology : official publication of the International Association for the Study of Lung Cancer.

[29]  C. Jaffe.  Measures of response: RECIST, WHO, and new alternatives. , 2006, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[30]  Binsheng Zhao,et al.  Evaluating variability in tumor measurements from same-day repeat CT scans of patients with non-small cell lung cancer. , 2009, Radiology.

[31]  H D Royal,et al.  Technology assessment: scientific challenges. , 1994, AJR. American journal of roentgenology.

[32]  Lawrence H. Schwartz,et al.  A simulation study to evaluate the impact of the number of lesions measured on response assessment. , 2009, European journal of cancer.

[33]  Geoffrey McLennan,et al.  Computed tomography assessment of response to therapy: tumor volume change measurement, truth data, and error. , 2009, Translational oncology.

[34]  M. Robins,et al.  Volumetric quantification of lung nodules in CT with iterative reconstruction (ASiR and MBIR). , 2013, Medical physics.

[35]  N. Petrick,et al.  Volume estimation of low-contrast lesions with CT: a comparison of performances from a phantom study, simulations and theoretical analysis , 2015, Physics in medicine and biology.

[36]  J Anthony Seibert,et al.  Description and implementation of a quality control program in an imaging-based clinical trial. , 2006, Academic radiology.

[37]  H. Chan,et al.  Accuracy of the CT numbers of simulated lung nodules imaged with multi-detector CT scanners. , 2006, Medical physics.

[38]  H. Barnhart,et al.  The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions , 2015, Statistical methods in medical research.

[39]  Kyle J. Myers,et al.  A resource for the assessment of lung nodule size estimation methods: database of thoracic CT scans of an anthropomorphic phantom◊ , 2010, Optics express.

[40]  Nicholas Petrick,et al.  Volumetric CT in lung cancer: an example for the qualification of imaging as a biomarker. , 2010, Academic radiology.

[41]  J. Hanley,et al.  The effect of measuring error on the results of therapeutic trials in advanced cancer , 1976, Cancer.

[42]  William M. Wells,et al.  Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation , 2004, IEEE Transactions on Medical Imaging.

[43]  Marios A. Gavrielides,et al.  Evaluation of 1D, 2D and 3D nodule size estimation by radiologists for spherical and non-spherical nodules through CT thoracic phantom imaging , 2011, Medical Imaging.

[44]  Ehsan Samei,et al.  Quantitative CT: technique dependence of volume estimation on pulmonary nodules , 2012, Physics in medicine and biology.

[45]  J. Kawamoto,et al.  An update and our experience with metastatic liver disease. , 2007, Hepato-gastroenterology.

[46]  Kyle J. Myers,et al.  Information-Theoretic Approach for Analyzing Bias and Variance in Lung Nodule Size Estimation With CT: A Phantom Study , 2010, IEEE Transactions on Medical Imaging.

[47]  Torsten Rohlfing,et al.  Performance-based classifier combination in atlas-based image segmentation using expectation-maximization parameter estimation , 2004, IEEE Transactions on Medical Imaging.

[48]  Janet Woodcock,et al.  The FDA critical path initiative and its influence on new drug development. , 2008, Annual review of medicine.

[49]  Rolf W. Günther,et al.  Semi-Automated Quantification of Hepatic Lesions in a Phantom , 2009, Investigative radiology.

[50]  L. Schwartz,et al.  Change in lung tumor volume as a biomarker of treatment response: a critical review of the evidence. , 2010, Annals of oncology : official journal of the European Society for Medical Oncology.

[51]  P M A van Ooijen,et al.  Inter- and intrascanner variability of pulmonary nodule volumetry on low-dose 64-row CT: an anthropomorphic phantom study. , 2013, The British journal of radiology.

[52]  A P Reeves,et al.  Data sets for the qualification of volumetric CT as a quantitative imaging biomarker in lung cancer. , 2010, Optics express.

[53]  Kyle J Myers,et al.  Benefit of overlapping reconstruction for improving the quantitative assessment of CT lung nodule volume. , 2013, Academic radiology.

[54]  Kyle J. Myers,et al.  A template-based approach for the analysis of lung nodules in a volumetric CT phantom study , 2009, Medical Imaging.

[55]  Paul J Nietert,et al.  Imprecision in automated volume measurements of pulmonary nodules and its effect on the level of uncertainty in volume doubling time estimation. , 2009, Chest.

[56]  Antoni B. Chan,et al.  On measuring the change in size of pulmonary nodules , 2006, IEEE Transactions on Medical Imaging.

[57]  Kyle J. Myers,et al.  Volume error analysis for lung nodules attached to pulmonary vessels in an anthropomorphic thoracic phantom , 2008, SPIE Medical Imaging.

[58]  Mithat Gönen,et al.  Quantitative imaging biomarkers: A review of statistical methods for technical performance assessment , 2015, Statistical methods in medical research.

[59]  James G. Ravenel,et al.  Pulmonary nodule volume: effects of reconstruction parameters on automated measurements--a phantom study. , 2008, Radiology.

[60]  M Heller,et al.  Precision of computer-aided volumetry of artificial small solid pulmonary nodules in ex vivo porcine lungs. , 2007, The British journal of radiology.

[61]  Huiman X Barnhart,et al.  Applications of the repeatability of quantitative imaging biomarkers: a review of statistical analysis of repeat data sets. , 2009, Translational oncology.