Autosegmentation for thoracic radiation treatment planning: A grand challenge at AAPM 2017

PURPOSE This report presents the methods and results of the Thoracic Auto-Segmentation Challenge organized at the 2017 Annual Meeting of American Association of Physicists in Medicine. The purpose of the challenge was to provide a benchmark dataset and platform for evaluating performance of autosegmentation methods of organs at risk (OARs) in thoracic CT images. METHODS Sixty thoracic CT scans provided by three different institutions were separated into 36 training, 12 offline testing, and 12 online testing scans. Eleven participants completed the offline challenge, and seven completed the online challenge. The OARs were left and right lungs, heart, esophagus, and spinal cord. Clinical contours used for treatment planning were quality checked and edited to adhere to the RTOG 1106 contouring guidelines. Algorithms were evaluated using the Dice coefficient, Hausdorff distance, and mean surface distance. A consolidated score was computed by normalizing the metrics against interrater variability and averaging over all patients and structures. RESULTS The interrater study revealed highest variability in Dice for the esophagus and spinal cord, and in surface distances for lungs and heart. Five out of seven algorithms that participated in the online challenge employed deep-learning methods. Although the top three participants using deep learning produced the best segmentation for all structures, there was no significant difference in the performance among them. The fourth place participant used a multi-atlas-based approach. The highest Dice scores were produced for lungs, with averages ranging from 0.95 to 0.98, while the lowest Dice scores were produced for esophagus, with a range of 0.55-0.72. CONCLUSION The results of the challenge showed that the lungs and heart can be segmented fairly accurately by various algorithms, while deep-learning methods performed better on the esophagus. Our dataset together with the manual contours for all training cases continues to be available publicly as an ongoing benchmarking resource.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  David R. Maffitt,et al.  De-identification of Medical Images with Retention of Scientific Research Value. , 2015, Radiographics : a review publication of the Radiological Society of North America, Inc.

[3]  G. Sharp,et al.  Vision 20/20: perspectives on automated image segmentation for radiotherapy. , 2014, Medical physics.

[4]  Vladimir Pekar,et al.  Head and Neck Auto-segmentation Challenge : Segmentation of the Parotid Glands , 2010 .

[5]  Dean F. Sittig,et al.  Prospective randomized double-blind study of atlas-based organ-at-risk autosegmentation-assisted radiation planning in head and neck cancer. , 2014, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[6]  Rick S. Blum,et al.  A statistical modeling approach for evaluating auto-segmentation methods for image-guided radiotherapy , 2012, Comput. Medical Imaging Graph..

[7]  Jean-François Daisne,et al.  Atlas-based automatic segmentation of head and neck organs at risk and nodal target volumes: a clinical validation , 2013, Radiation oncology.

[8]  Blake Hannaford,et al.  Evaluation of segmentation methods on head and neck CT: Auto‐segmentation challenge 2015 , 2017, Medical physics.

[9]  Jinzhong Yang,et al.  Atlas ranking and selection for automatic segmentation of the esophagus from CT scans , 2017, Physics in medicine and biology.

[10]  Mechthild Krause,et al.  Internal and external validation of an ESTRO delineation guideline - dependent automated segmentation tool for loco-regional radiation therapy of early breast cancer. , 2016, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[11]  Cedric X. Yu,et al.  Guidance document on delivery, treatment planning, and clinical implementation of IMRT: report of the IMRT Subcommittee of the AAPM Radiation Therapy Committee. , 2003, Medical physics.

[12]  E Weiss,et al.  SU-C-WAB-03: Assessing the Correlation Between Quantitative Measures of Contour Variability and Physician's Qualitative Measure for Clinical Usefulness of Auto-Segmentation in Prostate Cancer Radiotherapy. , 2013, Medical physics.

[13]  Stephen M. Moore,et al.  The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository , 2013, Journal of Digital Imaging.

[14]  J. Fowler,et al.  Image guidance for precise conformal radiotherapy. , 2003, International journal of radiation oncology, biology, physics.

[15]  Carlos Ortiz-de-Solorzano,et al.  Combination Strategies in Multi-Atlas Image Segmentation: Application to Brain MR Data , 2009, IEEE Transactions on Medical Imaging.

[16]  A. Lomax,et al.  Systematic evaluation of three different commercial software solutions for automatic segmentation for adaptive therapy in head-and-neck, prostate and pleural cancer , 2012, Radiation oncology.

[17]  Michael McKay,et al.  Evaluation of atlas-based auto-segmentation software in prostate cancer patients , 2014, Journal of medical radiation sciences.

[18]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[19]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Helena R. Torres,et al.  A novel multi‐atlas strategy with dense deformation field reconstruction for abdominal and thoracic multi‐organ segmentation from computed tomography , 2018, Medical Image Anal..

[21]  V. Pekar,et al.  Head and Neck Auto-segmentation Challenge , 2009, The MIDAS Journal.

[22]  K A Langmack,et al.  The utility of atlas-assisted segmentation in the male pelvis is dependent on the interobserver agreement of the structures segmented. , 2014, The British journal of radiology.

[23]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[24]  Benjamin Movsas,et al.  Consideration of dose limits for organs at risk of thoracic radiotherapy: atlas for lung, proximal bronchial tree, esophagus, spinal cord, ribs, and brachial plexus. , 2011, International journal of radiation oncology, biology, physics.

[25]  Joshua D. Lawson,et al.  Evaluation of automatic atlas-based lymph node segmentation for head-and-neck cancer. , 2010, International journal of radiation oncology, biology, physics.

[26]  J. Williams.  Challenge! , 1978, British journal of sports medicine.

[27]  Paul Aljabar,et al.  Clinical evaluation of atlas and deep learning based automatic contouring for lung cancer. , 2017, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[28]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Paul A. Yushkevich,et al.  Multi-Atlas Segmentation with Joint Label Fusion , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  M. J. Gooding,et al.  Multicenter Clinical Assessment of DIR Atlas-Based Autocontouring , 2013 .