Evaluating reinforcement learning agents for anatomical landmark detection

Graphical abstract Figure. No Caption available. ABSTRACT Automatic detection of anatomical landmarks is an important step for a wide range of applications in medical image analysis. Manual annotation of landmarks is a tedious task and prone to observer errors. In this paper, we evaluate novel deep reinforcement learning (RL) strategies to train agents that can precisely and robustly localize target landmarks in medical scans. An artificial RL agent learns to identify the optimal path to the landmark by interacting with an environment, in our case 3D images. Furthermore, we investigate the use of fixed‐ and multi‐scale search strategies with novel hierarchical action steps in a coarse‐to‐fine manner. Several deep Q‐network (DQN) architectures are evaluated for detecting multiple landmarks using three different medical imaging datasets: fetal head ultrasound (US), adult brain and cardiac magnetic resonance imaging (MRI). The performance of our agents surpasses state‐of‐the‐art supervised and RL methods. Our experiments also show that multi‐scale search strategies perform significantly better than fixed‐scale agents in images with large field of view and noisy background such as in cardiac MRI. Moreover, the novel hierarchical steps can significantly speed up the searching process by a factor of 4–5 times.

[1]  Dorin Comaniciu,et al.  Robust Multi-scale Anatomical Landmark Detection in Incomplete 3D-CT Data , 2017, MICCAI.

[2]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[3]  Fred Nicolls,et al.  Active shape models with SIFT descriptors and MARS , 2015, 2014 International Conference on Computer Vision Theory and Applications (VISAPP).

[4]  Isabelle Bloch,et al.  Multi-organ localization with cascaded global-to-local regression and shape prior , 2015, Medical Image Anal..

[5]  Aamer Aziz,et al.  Rapid and automatic localization of the anterior and posterior commissure point landmarks in MR volumetric neuroimages. , 2006, Academic radiology.

[6]  Dorin Comaniciu,et al.  An Artificial Agent for Anatomical Landmark Detection in Medical Images , 2016, MICCAI.

[7]  Libero Verard,et al.  Fully automatic identification of AC and PC landmarks on brain MRI using scene analysis , 1997, IEEE Transactions on Medical Imaging.

[8]  Daniel Rueckert,et al.  A Probabilistic Patch-Based Label Fusion Model for Multi-Atlas Segmentation With Registration Refinement: Application to Cardiac MR Images , 2013, IEEE Transactions on Medical Imaging.

[9]  Tom Schaul,et al.  Dueling Network Architectures for Deep Reinforcement Learning , 2015, ICML.

[10]  Felix Lau,et al.  Computationally Efficient Cardiac Views Projection Using 3D Convolutional Neural Networks , 2017, DLMIA/ML-CDS@MICCAI.

[11]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[12]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[13]  Antonio Criminisi,et al.  Regression forests for efficient anatomy detection and localization in computed tomography scans , 2013, Medical Image Anal..

[14]  Dorin Comaniciu,et al.  Automatic View Planning for Cardiac MRI Acquisition , 2011, MICCAI.

[15]  Bishesh Khanal,et al.  Fast Multiple Landmark Localisation Using a Patch-based Iterative Network , 2018, MICCAI.

[16]  Hado van Hasselt,et al.  Double Q-learning , 2010, NIPS.

[17]  Iwao Kanno,et al.  Automatic detection of the mid-sagittal plane in 3-D brain images , 1997, IEEE Transactions on Medical Imaging.

[18]  Babak A. Ardekani,et al.  Model-based automatic detection of the anterior and posterior commissures on MRI scans , 2009, NeuroImage.

[19]  Loïc Le Folgoc,et al.  Automatic View Planning with Multi-scale Deep Reinforcement Learning Agents , 2018, MICCAI.

[20]  Martin Urschler,et al.  From Local to Global Random Regression Forests: Exploring Anatomical Landmark Localization , 2016, MICCAI.

[21]  Svetlana Lazebnik,et al.  Active Object Localization with Deep Reinforcement Learning , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[22]  Mikkel B. Stegmann,et al.  Mid-sagittal plane and mid-sagittal surface optimization in brain MRI using a local symmetry measure , 2005, SPIE Medical Imaging.

[23]  Horst Bischof,et al.  Regressing Heatmaps for Multiple Landmark Localization Using CNNs , 2016, MICCAI.

[24]  Daguang Xu,et al.  Supervised Action Classifier: Approaching Landmark Detection as Image Partitioning , 2017, MICCAI.

[25]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[26]  D. Petrovska-Delacretaz,et al.  Automatic landmark location with a Combined Active Shape Model , 2009, 2009 IEEE 3rd International Conference on Biometrics: Theory, Applications, and Systems.

[27]  Simon Andermatt,et al.  Multi-dimensional Gated Recurrent Units for Automated Anatomical Landmark Localization , 2017, ArXiv.

[28]  C. Jack,et al.  Alzheimer's Disease Neuroimaging Initiative , 2008 .

[29]  J. Alison Noble,et al.  Image Analysis Using Machine Learning: Anatomical Landmarks Detection in Fetal Ultrasound Images , 2012, 2012 IEEE 36th Annual Computer Software and Applications Conference.

[30]  Daniel Rueckert,et al.  Laplacian Eigenmaps Manifold Learning for Landmark Localization in Brain MR Images , 2011, MICCAI.

[31]  Daniel Rueckert,et al.  Population-based studies of myocardial hypertrophy: high resolution cardiovascular magnetic resonance atlases improve statistical power , 2014, Journal of Cardiovascular Magnetic Resonance.

[32]  Ben Glocker,et al.  Stratified Decision Forests for Accurate Anatomical Landmark Localization in Cardiac Images , 2017, IEEE Transactions on Medical Imaging.

[33]  Alejandro F Frangi,et al.  Automatic construction of 3-D statistical deformation models of the brain using nonrigid registration , 2003, IEEE Transactions on Medical Imaging.

[34]  Hamid R. Tizhoosh,et al.  A Reinforcement Learning Framework for Medical Image Segmentation , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[35]  Michael Brady,et al.  Personalized Graphical Models for Anatomical Landmark Localization in Whole-Body Medical Images , 2014, International Journal of Computer Vision.

[36]  Dorin Comaniciu,et al.  Multi-Scale Deep Reinforcement Learning for Real-Time 3D-Landmark Detection in CT Scans , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Daniel Rueckert,et al.  Landmark localisation in brain MR images using feature point descriptors based on 3D local self-similarities , 2012, 2012 9th IEEE International Symposium on Biomedical Imaging (ISBI).

[38]  Gustavo Carneiro,et al.  Deep Reinforcement Learning for Active Breast Lesion Detection from DCE-MRI , 2017, MICCAI.

[39]  Günther Platsch,et al.  Improved Anatomical Landmark Localization in Medical Images Using Dense Matching of Graphical Models , 2010, BMVC.

[40]  Martin Urschler,et al.  Integrating geometric configuration and appearance information into a unified framework for anatomical landmark localization , 2018, Medical Image Anal..

[41]  Dorin Comaniciu,et al.  An Artificial Agent for Robust Image Registration , 2016, AAAI.

[42]  Hervé Delingette,et al.  Robust Non-rigid Registration Through Agent-Based Action Learning , 2017, MICCAI.

[43]  Dorin Comaniciu,et al.  3D Deep Learning for Efficient and Robust Landmark Detection in Volumetric Data , 2015, MICCAI.

[44]  Jun Zhang,et al.  Detecting Anatomical Landmarks From Limited Medical Imaging Data Using Two-Stage Task-Oriented Deep Neural Networks , 2017, IEEE Transactions on Image Processing.

[45]  Yaozong Gao,et al.  Robust Anatomical Landmark Detection for MR Brain Image Registration , 2014, MICCAI.

[46]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[47]  Benoit M. Dawant,et al.  Automatic Localization of the Anterior Commissure, Posterior Commissure, and Midsagittal Plane in MRI Scans using Regression Forests , 2015, IEEE Journal of Biomedical and Health Informatics.

[48]  Margrit Betke,et al.  Landmark detection in the chest and registration of lung surfaces with an application to nodule registration , 2003, Medical Image Anal..

[49]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[50]  Hamid R. Tizhoosh,et al.  Using reinforcement learning for image thresholding , 2003, CCECE 2003 - Canadian Conference on Electrical and Computer Engineering. Toward a Caring and Humane Technology (Cat. No.03CH37436).

[51]  Martin A. Riedmiller,et al.  Reinforcement learning without an explicit terminal state , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[52]  Long-Ji Lin,et al.  Reinforcement learning for robots using neural networks , 1992 .