Evaluation of Registration Methods on Thoracic CT: The EMPIRE10 Challenge

EMPIRE10 (Evaluation of Methods for Pulmonary Image REgistration 2010) is a public platform for fair and meaningful comparison of registration algorithms which are applied to a database of intra patient thoracic CT image pairs. Evaluation of nonrigid registration techniques is a nontrivial task. This is compounded by the fact that researchers typically test only on their own data, which varies widely. For this reason, reliable assessment and comparison of different registration algorithms has been virtually impossible in the past. In this work we present the results of the launch phase of EMPIRE10, which comprised the comprehensive evaluation and comparison of 20 individual algorithms from leading academic and industrial research groups. All algorithms are applied to the same set of 30 thoracic CT pairs. Algorithm settings and parameters are chosen by researchers expert in the con figuration of their own method and the evaluation is independent, using the same criteria for all participants. All results are published on the EMPIRE10 website (http://empire10.isi.uu.nl). The challenge remains ongoing and open to new participants. Full results from 24 algorithms have been published at the time of writing. This paper details the organization of the challenge, the data and evaluation methods and the outcome of the initial launch with 20 algorithms. The gain in knowledge and future work are discussed.

Brian B. Avants | D. Louis Collins | Nikos Paragios | Cristian Lorenz | Sven Kabus | Josien P. W. Pluim | Marc Modat | Sébastien Ourselin | Tom Vercauteren | James C. Gee | Gang Song | Marleen de Bruijne | Simon Rit | Mark Jenkinson | Grégoire Malandain | Martin Urschler | Gary E. Christensen | Joseph M. Reinhardt | Nassir Navab | Bram van Ginneken | Jon Sporring | Rui Li | Dirk Loeckx | Olivier Commowick | Alexander Schmidt-Richberg | Tal Arbel | Dante De Nigris | Marius Staring | Berend C. Stoel | Xiao Han | Nicholas J. Tustison | Max A. Viergever | Mattias P. Heinrich | Stefan Klein | Julia A. Schnabel | Dirk Smeets | Jef Vandemeulebroucke | René Werner | Jan Ehrhardt | Vincent Garcia | Marta Peroni | Vladlena Gorbunova | Jamie McClelland | Kai Ding | Kunlin Cao | Manuel Werlberger | Xiang Deng | Ben Glocker | Nicholas Ayache | Keelin Murphy | Kaifang Du | David Sarrut | Sascha E. A. Muenzing | Gregory C. Sharp | Marleen de Bruijne | G. Christensen | Manuel Werlberger | N. Paragios | J. Gee | G. Sharp | N. Navab | K. Murphy | B. Ginneken | J. Reinhardt | S. Kabus | K. Ding | X. Deng | K. Cao | K. Du | Vincent Garcia | T. Vercauteren | N. Ayache | O. Commowick | G. Malandain | B. Glocker | V. Gorbunova | J. Sporring | Xiao Han | M. Heinrich | J. Schnabel | M. Jenkinson | C. Lorenz | M. Modat | J. McClelland | S. Ourselin | M. Viergever | D. Collins | T. Arbel | M. Peroni | R. Li | A. Schmidt-Richberg | J. Ehrhardt | R. Werner | D. Smeets | D. Loeckx | G. Song | N. Tustison | B. Avants | M. Staring | S. Klein | B. Stoel | M. Urschler | J. Vandemeulebroucke | S. Rit | D. Sarrut | J. Pluim | R. Li | Rui Li | R. Li | Rui Li

[1]  Lisa M. Brown,et al.  A survey of image registration techniques , 1992, CSUR.

[2]  Benoit M. Dawant,et al.  Comparison and evaluation of retrospective intermodality image registration techniques , 1996, Medical Imaging.

[3]  Gerald Q. Maguire,et al.  Comparison and evaluation of retrospective intermodality brain image registration techniques. , 1997, Journal of computer assisted tomography.

[4]  Stephan Heckers,et al.  A Method for Assessing the Accuracy of Intersubject Registration of the Human Brain Using Anatomic Landmarks , 1999, NeuroImage.

[5]  Simon R. Arridge,et al.  A survey of hierarchical non-linear medical image registration , 1999, Pattern Recognit..

[6]  Hervé Delingette,et al.  Automatic Detection and Segmentation of Evolving Processes in 3D Medical Images: Application to Multiple Sclerosis , 1999, IPMI.

[7]  D. Louis Collins,et al.  Retrospective evaluation of intersubject brain registration , 2003, IEEE Transactions on Medical Imaging.

[8]  David J. Hawkes,et al.  Validation of nonrigid image registration using finite-element methods: application to breast MR images , 2003, IEEE Transactions on Medical Imaging.

[9]  Max A. Viergever,et al.  Mutual-information-based registration of medical images: a survey , 2003, IEEE Transactions on Medical Imaging.

[10]  Michael Unser,et al.  Fast parametric elastic image registration , 2003, IEEE Trans. Image Process..

[11]  Jan Flusser,et al.  Image registration methods: a survey , 2003, Image Vis. Comput..

[12]  Raymond Lister,et al.  Grand challenges , 2005, SGCS.

[13]  Joe Y. Chang,et al.  Validation of an accelerated ‘demons’ algorithm for deformable image registration in radiation therapy , 2005, Physics in medicine and biology.

[14]  C. Ling,et al.  Evaluation of an automated deformable image matching method for quantifying lung motion in respiration-correlated CT images. , 2006, Medical physics.

[15]  D. Xu,et al.  Nodule management protocol of the NELSON randomised lung cancer screening trial. , 2006, Lung cancer.

[16]  Imran A. Pirwani,et al.  Introduction to the Non-rigid Image Registration Evaluation Project (NIREP) , 2006, WBIR.

[17]  L Dong,et al.  Quantification of accuracy of the automated nonlinear image matching and anatomical labeling (ANIMAL) nonlinear registration algorithm for 4D CT images of lung. , 2007, Medical physics.

[18]  Horst Bischof,et al.  A Duality Based Algorithm for TV- L 1-Optical-Flow Image Registration , 2007, MICCAI.

[19]  Patrick Clarysse,et al.  A Comparison Framework for Breathing Motion Estimation Methods From 4-D Imaging , 2007, IEEE Transactions on Medical Imaging.

[20]  H. Bischof,et al.  A Framework for Comparison and Evaluation of Nonlinear Intra-Subject Image Registration Algorithms , 2007, The Insight Journal.

[21]  Josien P. W. Pluim,et al.  Semi-automatic Reference Standard Construction for Quantitative Evaluation of Lung CT Registration , 2008, MICCAI.

[22]  Steve B. Jiang,et al.  4D-CT lung motion estimation with deformable registration: quantification of motion nonlinearity and hysteresis. , 2008, Medical physics.

[23]  Quan Chen,et al.  Objective assessment of deformable image registration in radiotherapy: a multi-institution study. , 2007, Medical physics.

[24]  Bram van Ginneken,et al.  Supervised Enhancement Filters: Application to Fissure Detection in Chest CT Scans , 2008, IEEE Transactions on Medical Imaging.

[25]  Nassir Navab,et al.  Dense image registration through MRFs and efficient linear programming , 2008, Medical Image Anal..

[26]  Gregory C Sharp,et al.  Evaluation of deformable registration of patient lung 4DCT with subanatomical region segmentations. , 2008, Medical physics.

[27]  Cristian Lorenz,et al.  Validation and comparison of registration methods for free-breathing 4D lung CT , 2008, SPIE Medical Imaging.

[28]  Marleen de Bruijne,et al.  Weight Preserving Image Registration for Monitoring Disease Progression in Lung CT , 2008, MICCAI.

[29]  Tom Vercauteren,et al.  Diffeomorphic demons: Efficient non-parametric image registration , 2009, NeuroImage.

[30]  R. Castillo,et al.  A framework for evaluation of deformable image registration spatial accuracy using large landmark point sets , 2009, Physics in medicine and biology.

[31]  Arno Klein,et al.  Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration , 2009, NeuroImage.

[32]  B. van Ginneken,et al.  Automatic lung segmentation from thoracic computed tomography scans using a hybrid approach with error detection. , 2009, Medical physics.

[33]  Michael A Yassa,et al.  A quantitative evaluation of cross-participant registration techniques for MRI studies of the medial temporal lobe , 2009, NeuroImage.

[34]  Cristian Lorenz,et al.  Evaluation of 4D-CT Lung Registration , 2009, MICCAI.

[35]  Sascha E. A. Muenzing,et al.  Knowledge Driven Regularization of the Deformation Field for PDE Based Non-Rigid Registration Algorithms , 2010 .

[36]  Paul Suetens,et al.  3D lung registration using splineMIRIT and Robust Tree Registration (RTR) , 2010 .

[37]  Grégoire Malandain,et al.  Diffeomorphic demons and the EMPIRE10 challenge , 2010 .

[38]  K. Brock.  Results of a multi-institution deformable registration accuracy study (MIDRAS). , 2010, International journal of radiation oncology, biology, physics.

[39]  Sébastien Ourselin,et al.  Fast free-form deformation using graphics processing units , 2010, Comput. Methods Programs Biomed..

[40]  Horst Bischof,et al.  Robust Optical Flow Based Deformable Registration of Thoracic CT Images , 2010 .

[41]  Gary E. Christensen,et al.  Unifying Vascular Information in Intensity-Based Nonrigid Lung CT Registration , 2010, WBIR.

[42]  Vincent Garcia,et al.  A Robust and Efficient Block Matching Framework for Non Linear Registration of Thoracic CT Images , 2010 .

[43]  Stefan Klein,et al.  Pulmonary Image Registration with elastix using a Standard Intensity-Based Algorithm , 2010 .

[44]  A. Schmidt-Richberg,et al.  Diffeomorphic Diffusion Registration of Lung CT Images , 2010 .

[45]  Steve B Jiang,et al.  Implementation and evaluation of various demons deformable image registration algorithms on a GPU. , 2010, Physics in medicine and biology.

[46]  N Kandasamy,et al.  On developing B-spline registration algorithms for multi-core processors , 2010, Physics in medicine and biology.

[47]  N. Komodakis,et al.  Non-rigid Registration using Discrete MRFs: Application to Thoracic CT Images , 2010 .

[48]  Marc Modat,et al.  Lung registration using the NiftyReg package , 2010 .

[49]  G. Christensen,et al.  Regularized Nonrigid Registration of Lung CT I m ages by Preserving Tissue Volu m e and Vesselness Measure , 2010 .

[50]  Marleen de Bruijne,et al.  Mass Preserving Image Registration: Results of Evaluation of Methods for Pulmonary Image Registration 2010 Challenge , 2010 .

[51]  M. Brady,et al.  Discontinuity preserving regularisation for variational optical-flow registration using the modified L p norm , 2010 .

[52]  Xiao Han.  Feature-constrained Nonlinear Registration of Lung CT Images , 2010 .

[53]  Max A. Viergever,et al.  elastix: A Toolbox for Intensity-Based Medical Image Registration , 2010, IEEE Transactions on Medical Imaging.

[54]  Heinz Handels,et al.  Statistical Modeling of 4D Respiratory Lung Motion Using Diffeomorphic Image Registration , 2011, IEEE Transactions on Medical Imaging.

[55]  Max A. Viergever,et al.  Semi-automatic construction of reference standards for evaluation of image registration , 2011, Medical Image Anal..

[56]  Arno Klein,et al.  A reproducible evaluation of ANTs similarity metric performance in brain image registration , 2011, NeuroImage.

[57]  J. Gee,et al.  Pulmonary kinematics from image data: a review. , 2011, Academic radiology.

[58]  V.R.S Mani,et al.  Survey of Medical Image Registration , 2013 .

[59]  Balraj Naren,et al.  Medical Image Registration , 2022 .