Deep Learning-Based Regression and Classification for Automatic Landmark Localization in Medical Images

In this study, we propose a fast and accurate method to automatically localize anatomical landmarks in medical images. We employ a global-to-local localization approach using fully convolutional neural networks (FCNNs). First, a global FCNN localizes multiple landmarks through the analysis of image patches, performing regression and classification simultaneously. In regression, displacement vectors pointing from the center of image patches towards landmark locations are determined. In classification, presence of landmarks of interest in the patch is established. Global landmark locations are obtained by averaging the predicted displacement vectors, where the contribution of each displacement vector is weighted by the posterior classification probability of the patch that it is pointing from. Subsequently, for each landmark localized with global localization, local analysis is performed. Specialized FCNNs refine the global landmark locations by analyzing local sub-images in a similar manner, i.e. by performing regression and classification simultaneously and combining the results. Evaluation was performed through localization of 8 anatomical landmarks in CCTA scans, 2 landmarks in olfactory MR scans, and 19 landmarks in cephalometric X-rays. We demonstrate that the method performs similarly to a second observer and is able to localize landmarks in a diverse set of medical images, differing in image modality, image dimensionality, and anatomical coverage.

[1]  Ben Glocker,et al.  Stratified Decision Forests for Accurate Anatomical Landmark Localization in Cardiac Images , 2017, IEEE Transactions on Medical Imaging.

[2]  Shaohua Kevin Zhou,et al.  Discriminative anatomy detection: Classification vs regression , 2014, Pattern Recognit. Lett..

[3]  Horst Bischof,et al.  Global localization of 3D anatomical structures by pre-filtered Hough Forests and discrete optimization , 2013, Medical Image Anal..

[4]  Adnan Kastrati,et al.  Standardized imaging for aortic annular sizing: implications for transcatheter valve selection. , 2013, JACC. Cardiovascular imaging.

[5]  Jochen Peters,et al.  Patient Specific Models for Planning and Guidance of Minimally Invasive Aortic Valve Implantation , 2010, MICCAI.

[6]  Bulat Ibragimov,et al.  Computerized Cephalometry by Game Theory with Shape-and Appearance-Based Landmark Refinement , 2016 .

[7]  Timothy F. Cootes,et al.  A benchmark for comparison of dental radiography analysis algorithms , 2016, Medical Image Anal..

[8]  Timothy F. Cootes,et al.  Fully Automatic Cephalometric Evaluation using Random Forest Regression-Voting , 2015, ISBI 2015.

[9]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[11]  Gernot Brockmann,et al.  Automatic Aorta Segmentation and Valve Landmark Detection in C-Arm CT for Transcatheter Aortic Valve Implantation , 2012, IEEE Transactions on Medical Imaging.

[12]  Max A. Viergever,et al.  Semi-automatic construction of reference standards for evaluation of image registration , 2011, Medical Image Anal..

[13]  Jun Du,et al.  Fast and Robust Detection of Anatomical Landmarks Using Cascaded 3D Convolutional Networks Guided by Linear Square Regression , 2018, CCBR.

[14]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[15]  Bulat Ibragimov,et al.  Fully automated quantitative cephalometry using convolutional neural networks , 2017, Journal of medical imaging.

[16]  Mohammad A. Dabbah,et al.  Detection and location of 127 anatomical landmarks in diverse CT datasets , 2014, Medical Imaging.

[17]  Dorin Comaniciu,et al.  3D Deep Learning for Efficient and Robust Landmark Detection in Volumetric Data , 2015, MICCAI.

[18]  Christian Payer,et al.  Integrating spatial configuration into heatmap regression based CNNs for landmark localization , 2019, Medical Image Anal..

[19]  H. Marquering,et al.  Automatic aortic root landmark detection in CTA images for preprocedural planning of transcatheter aortic valve implantation , 2015, The International Journal of Cardiovascular Imaging.

[20]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[21]  Nassir Navab,et al.  Dynamic Model-Driven Quantitative and Visual Evaluation of the Aortic Valve from 4D CT , 2008, MICCAI.

[22]  Il Dong Yun,et al.  Partial Policy-Based Reinforcement Learning for Anatomical Landmark Localization in 3D Medical Images , 2018, IEEE Transactions on Medical Imaging.

[23]  Max A. Viergever,et al.  Multi-Atlas-Based Segmentation With Local Decision Fusion—Application to Cardiac and Aortic Segmentation in CT Scans , 2009, IEEE Transactions on Medical Imaging.

[24]  Dimitris N. Metaxas,et al.  Automated anatomical landmark detection ondistal femur surface using convolutional neural network , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[25]  Yaozong Gao,et al.  Collaborative regression-based anatomical landmark detection. , 2015, Physics in medicine and biology.

[26]  Rui Liao,et al.  Automatic pose initialization for accurate 2D/3D registration applied to abdominal aortic aneurysm endovascular repair , 2012, Medical Imaging: Image-Guided Procedures.

[27]  Xiaoguang Lu,et al.  Discriminative Context Modeling Using Auxiliary Markers for LV Landmark Detection from a Single MR Image , 2012, STACOM.

[28]  Ana Maria Mendonça,et al.  A Pixel-Wise Distance Regression Approach for Joint Retinal Optical Disc and Fovea Detection , 2018, MICCAI.

[29]  Dongqing Zhang,et al.  HeadLocNet: Deep convolutional neural networks for accurate classification and multi-landmark localization of head CTs , 2020, Medical Image Anal..

[30]  Carlos Vázquez,et al.  Automatic spine and pelvis detection in frontal X-rays using deep neural networks for patch displacement learning , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[31]  Walid Abdullah Al,et al.  Automatic aortic valve landmark localization in coronary CT angiography using colonial walk , 2018, PloS one.

[32]  Gernot Brockmann,et al.  Automatic Aorta Segmentation and Valve Landmark Detection in C-Arm CT: Application to Aortic Valve Implantation , 2010, MICCAI.

[33]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Martin Urschler,et al.  From Local to Global Random Regression Forests: Exploring Anatomical Landmark Localization , 2016, MICCAI.

[35]  Alison Q. O'Neil,et al.  Attaining Human-Level Performance with Atlas Location Autocontext for Anatomical Landmark Detection in 3D CT Data , 2018, ECCV Workshops.

[36]  Dorin Comaniciu,et al.  Multi-Scale Deep Reinforcement Learning for Real-Time 3D-Landmark Detection in CT Scans , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Martin Urschler,et al.  Integrating geometric configuration and appearance information into a unified framework for anatomical landmark localization , 2018, Medical Image Anal..

[38]  Loïc Le Folgoc,et al.  Evaluating reinforcement learning agents for anatomical landmark detection , 2019, Medical Image Anal..

[39]  Bishesh Khanal,et al.  Fast Multiple Landmark Localisation Using a Patch-based Iterative Network , 2018, MICCAI.

[40]  Ulas Bagci,et al.  Deep Geodesic Learning for Segmentation and Anatomical Landmarking , 2018, IEEE Transactions on Medical Imaging.

[41]  Dwarikanath Mahapatra,et al.  Landmark Detection in Cardiac MRI Using Learned Local Image Statistics , 2012, STACOM.

[42]  Max A. Viergever,et al.  Coronary artery centerline extraction in cardiac CT angiography using a CNN‐based orientation classifier , 2018, Medical Image Anal..

[43]  Daguang Xu,et al.  Supervised Action Classifier: Approaching Landmark Detection as Image Partitioning , 2017, MICCAI.

[44]  Jun Zhang,et al.  Detecting Anatomical Landmarks From Limited Medical Imaging Data Using Two-Stage Task-Oriented Deep Neural Networks , 2017, IEEE Transactions on Image Processing.

[45]  Yaozong Gao,et al.  Robust Anatomical Landmark Detection for MR Brain Image Registration , 2014, MICCAI.

[46]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[47]  Sehat Ullah,et al.  Medical image registration in image guided surgery: Issues, challenges and research opportunities , 2017 .

[48]  Karl Rohr.  Landmark-Based Image Analysis: Using Geometric And Intensity Models , 2010 .