Towards intelligent robust detection of anatomical structures in incomplete volumetric data

HighlightsMulti‐scale DRL with robust statistical shape modeling for anatomy detection.Multi‐scale processing enables real‐time speed and high detection accuracy.Robust and principled recognition of anatomy that is missing from the field‐of‐view.Extensive experiments on up to 50 anatomical landmarks and over 5000 3D‐CT scans. Graphical abstract Figure. No caption available. ABSTRACT Robust and fast detection of anatomical structures represents an important component of medical image analysis technologies. Current solutions for anatomy detection are based on machine learning, and are generally driven by suboptimal and exhaustive search strategies. In particular, these techniques do not effectively address cases of incomplete data, i.e., scans acquired with a partial field‐of‐view. We address these challenges by following a new paradigm, which reformulates the detection task to teaching an intelligent artificial agent how to actively search for an anatomical structure. Using the principles of deep reinforcement learning with multi‐scale image analysis, artificial agents are taught optimal navigation paths in the scale‐space representation of an image, while accounting for structures that are missing from the field‐of‐view. The spatial coherence of the observed anatomical landmarks is ensured using elements from statistical shape modeling and robust estimation theory. Experiments show that our solution outperforms marginal space deep learning, a powerful deep learning method, at detecting different anatomical structures without any failure. The dataset contains 5043 3D‐CT volumes from over 2000 patients, totaling over 2,500,000 image slices. In particular, our solution achieves 0% false‐positive and 0% false‐negative rates at detecting whether the landmarks are captured in the field‐of‐view of the scan (excluding all border cases), with an average detection accuracy of 2.78 mm. In terms of runtime, we reduce the detection‐time of the marginal space deep learning method by 20–30 times to under 40 ms, an unmatched performance for high resolution incomplete 3D‐CT data.

[1]  Yoshitaka Masutani,et al.  Automatic detection of over 100 anatomical landmarks in medical CT images: A framework with independent detectors and combinatorial optimization , 2017, Medical Image Anal..

[2]  Fang Lu,et al.  Automatic 3D liver location and segmentation via convolutional neural network and graph cut , 2016, International Journal of Computer Assisted Radiology and Surgery.

[3]  Dorin Comaniciu,et al.  Marginal Space Deep Learning: Efficient Architecture for Volumetric Image Parsing , 2016, IEEE Transactions on Medical Imaging.

[4]  Xiang Zhang,et al.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks , 2013, ICLR.

[5]  Isabelle Bloch,et al.  Multi-organ localization with cascaded global-to-local regression and shape prior , 2015, Medical Image Anal..

[6]  Paul A. Yushkevich,et al.  Fully automatic segmentation of the mitral leaflets in 3D transesophageal echocardiographic images using multi-atlas joint label fusion and deformable medial modeling , 2014, Medical Image Anal..

[7]  Martin Urschler,et al.  From Local to Global Random Regression Forests: Exploring Anatomical Landmark Localization , 2016, MICCAI.

[8]  Julia F. Barrett,et al.  Artifacts in CT: recognition and avoidance. , 2004, Radiographics : a review publication of the Radiological Society of North America, Inc.

[9]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[10]  Daguang Xu,et al.  Automatic Liver Segmentation Using an Adversarial Image-to-Image Network , 2017, MICCAI.

[11]  Dit-Yan Yeung,et al.  Learning a Deep Compact Image Representation for Visual Tracking , 2013, NIPS.

[12]  Paul A. Yushkevich,et al.  Medially constrained deformable modeling for segmentation of branching medial structures: Application to aortic valve segmentation and morphometry , 2015, Medical Image Anal..

[13]  Susan Ofner,et al.  Comparison of reliability in anatomical landmark identification using two-dimensional digital cephalometrics and three-dimensional cone beam computed tomography in vivo. , 2009, Dento maxillo facial radiology.

[14]  Yiqiang Zhan,et al.  Active Scheduling of Organ Detection and Segmentation in Whole-Body Medical Images , 2008, MICCAI.

[15]  Yang Wang,et al.  Prediction Based Collaborative Trackers (PCT): A Robust and Accurate Approach Toward 3D Medical Object Tracking , 2011, IEEE Transactions on Medical Imaging.

[16]  R. Bellman.  Dynamic programming. , 1957, Science.

[17]  Dorin Comaniciu,et al.  A boosting regression approach to medical anatomy detection , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Seyed-Ahmad Ahmadi,et al.  Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields , 2016, MICCAI.

[19]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[20]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Tony Lindeberg,et al.  Scale-Space Theory in Computer Vision , 1993, Lecture Notes in Computer Science.

[22]  Horst Bischof,et al.  Regressing Heatmaps for Multiple Landmark Localization Using CNNs , 2016, MICCAI.

[23]  J. Victor,et al.  How precise can bony landmarks be determined on a CT scan of the knee? , 2009, The Knee.

[24]  Huan Huang,et al.  National trends in advanced outpatient diagnostic imaging utilization: an analysis of the medical expenditure panel survey, 2000-2009 , 2013, BMC Medical Imaging.

[25]  Antonio Criminisi,et al.  Regression Forests for Efficient Anatomy Detection and Localization in CT Studies , 2010, MCV.

[26]  Dorin Comaniciu,et al.  Search strategies for multiple landmark detection by submodular maximization , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[27]  D. Rueckert,et al.  Stratified Decision Forests for Accurate Anatomical Landmark Localization in Cardiac Images. , 2017, IEEE transactions on medical imaging.

[28]  Nilay D Shah,et al.  Trends in Computed Tomography Utilization Rates: A Longitudinal Practice-Based Study , 2014, Journal of patient safety.

[29]  Dorin Comaniciu,et al.  Marginal Space Deep Learning: Efficient Architecture for Detection in Volumetric Image Data , 2015, MICCAI.

[30]  Dorin Comaniciu,et al.  3D Deep Learning for Efficient and Robust Landmark Detection in Volumetric Data , 2015, MICCAI.

[31]  Ben Glocker,et al.  Robust Registration of Longitudinal Spine CT , 2014, MICCAI.

[32]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[33]  Horst Bischof,et al.  Global localization of 3D anatomical structures by pre-filtered Hough Forests and discrete optimization , 2013, Medical Image Anal..

[34]  Laurent D. Cohen,et al.  Automatic Detection and Segmentation of Kidneys in 3D CT Images Using Random Forests , 2012, MICCAI.

[35]  Andrew Zisserman,et al.  MLESAC: A New Robust Estimator with Application to Estimating Image Geometry , 2000, Comput. Vis. Image Underst..

[36]  Antonio Criminisi,et al.  Fast Multiple Organ Detection and Localization in Whole-Body MR Dixon Sequences , 2011, MICCAI.

[37]  Chengwen Chu,et al.  Fully Automatic Localization and Segmentation of 3D Vertebral Bodies from CT/MR Images via a Learning-Based Method , 2015, PloS one.

[38]  Michael A. Speidel,et al.  Robust 5DOF Transesophageal Echo Probe Tracking at Fluoroscopic Frame Rates , 2015, MICCAI.

[39]  Dorin Comaniciu,et al.  Multi-Scale Deep Reinforcement Learning for Real-Time 3D-Landmark Detection in CT Scans , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Martin Urschler,et al.  Integrating geometric configuration and appearance information into a unified framework for anatomical landmark localization , 2018, Medical Image Anal..

[41]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[42]  Hugo A. Katus,et al.  Automatic image-to-model framework for patient-specific electromechanical modeling of the heart , 2014, 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI).

[43]  Dorin Comaniciu,et al.  Four-Chamber Heart Modeling and Automatic Segmentation for 3-D Cardiac CT Volumes Using Marginal Space Learning and Steerable Features , 2008, IEEE Transactions on Medical Imaging.

[44]  Dorin Comaniciu,et al.  Robust Multi-scale Anatomical Landmark Detection in Incomplete 3D-CT Data , 2017, MICCAI.

[45]  Dorin Comaniciu,et al.  An Artificial Agent for Anatomical Landmark Detection in Medical Images , 2016, MICCAI.

[46]  Rainald Löhner,et al.  Efficient simulation of blood flow past complex endovascular devices using an adaptive embedding technique , 2005, IEEE Transactions on Medical Imaging.