Deep Learning and Convolutional Neural Networks for Medical Image Computing

These are exciting times for medical image processing. Innovations in deep learning and the increasing availability of large annotated medical image datasets are leading to dramatic advances in automated understanding of medical images. From this perspective, I give a personal view of how computer-aided diagnosis of medical images has evolved and how the latest advances are leading to dramatic improvements today. I discuss the impact of deep learning on automated disease detection and organ and lesion segmentation, with particular attention to applications in diagnostic radiology. I provide some examples of how time-intensive and expensive manual annotation of huge medical image datasets by experts can be sidestepped by using weakly supervised learning from routine clinically generated medical reports. Finally, I identify the remaining knowledge gaps that must be overcome to achieve clinician-level performance of automated medical image processing systems. Computer-aided diagnosis (CAD) in medical imaging has flourished over the past several decades. New advances in computer software and hardware and improved quality of images from scanners have enabled this progress. The main motivations for CAD have been to reduce error and to enable more efficient measurement and interpretation of images. From this perspective, I will describe how deep learning has led to radical changes in howCAD research is conducted and in howwell it performs. For brevity, I will include automated disease detection and image processing under the rubric of CAD. Financial Disclosure The author receives patent royalties from iCAD Medical. Disclaimer No NIH endorsement of any product or company mentioned in this manuscript should be inferred. The opinions expressed herein are the author’s and do not necessarily represent those of NIH. R.M. Summers (B) Imaging Biomarkers and Computer-Aided Diagnosis Laboratory, Radiology and Imaging Sciences, National Institutes of Health Clinical Center, Bldg. 10, Room 1C224D MSC 1182, Bethesda, MD 20892-1182, USA e-mail: rms@nih.gov URL: http://www.cc.nih.gov/about/SeniorStaff/ronald_summers.html © Springer International Publishing Switzerland 2017 L. Lu et al. (eds.), Deep Learning and Convolutional Neural Networks for Medical Image Computing, Advances in Computer Vision and Pattern Recognition, DOI 10.1007/978-3-319-42999-1_1 3

[1]  Bolei Zhou,et al.  Learning Deep Features for Scene Recognition using Places Database , 2014, NIPS.

[2]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  N. Karssemeijer,et al.  An automatic method to discriminate malignant masses from normal tissue in digital mammograms1 , 2000, Physics in medicine and biology.

[4]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Jitendra Malik,et al.  Learning Rich Features from RGB-D Images for Object Detection and Segmentation , 2014, ECCV.

[6]  Gustavo Carneiro,et al.  Combining deep learning and level set for the automated segmentation of the left ventricle of the heart from cardiac cine magnetic resonance , 2017, Medical Image Anal..

[7]  Luca Maria Gambardella,et al.  Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks , 2013, MICCAI.

[8]  Andrew Zisserman,et al.  The devil is in the details: an evaluation of recent feature encoding methods , 2011, BMVC.

[9]  Berkman Sahiner,et al.  Computer-aided detection of breast masses on full field digital mammograms. , 2005, Medical physics.

[10]  Georg Langs,et al.  Unsupervised Pre-training Across Image Domains Improves Lung Tissue Classification , 2014, MCV.

[11]  K. Straif,et al.  Breast-cancer screening--viewpoint of the IARC Working Group. , 2015, The New England journal of medicine.

[12]  N Karssemeijer,et al.  Use of border information in the classification of mammographic masses , 2006, Physics in medicine and biology.

[13]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[14]  Nico Karssemeijer,et al.  Breast Tissue Segmentation and Mammographic Risk Scoring Using Deep Learning , 2014, Digital Mammography / IWDM.

[15]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[16]  Ronald M. Summers,et al.  2D View Aggregation for Lymph Node Detection Using a Shallow Hierarchy of Linear Classifiers , 2014, MICCAI.

[17]  Antoine Geissbühler,et al.  Building a reference multimedia database for interstitial lung diseases , 2012, Comput. Medical Imaging Graph..

[18]  Nima Tajbakhsh,et al.  Computer-Aided Pulmonary Embolism Detection Using a Novel Vessel-Aligned Multi-planar Image Representation and Convolutional Neural Networks , 2015, MICCAI.

[19]  Ronald M. Summers,et al.  Holistic classification of CT attenuation patterns for interstitial lung diseases via deep convolutional neural networks , 2018, Comput. methods Biomech. Biomed. Eng. Imaging Vis..

[20]  Weimin Huang,et al.  Brain tumor grading based on Neural Networks and Convolutional Neural Networks , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[21]  Ghassan Hamarneh,et al.  Mammography Segmentation with Maximum Likelihood Active Contours , 2022 .

[22]  A. Jemal,et al.  Cancer Statistics, 2008 , 2008, CA: a cancer journal for clinicians.

[23]  Yongyi Yang,et al.  Computer-Aided Detection and Diagnosis of Breast Cancer With Mammography: Recent Advances , 2009, IEEE Transactions on Information Technology in Biomedicine.

[24]  Camille Couprie,et al.  Learning Hierarchical Features for Scene Labeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Shuiwang Ji,et al.  Deep convolutional neural networks for multi-modality isointense infant brain image segmentation , 2015, NeuroImage.

[26]  M. Roizen.  Forecasting the Future of Cardiovascular Disease in the United States: A Policy Statement From the American Heart Association , 2012 .

[27]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[28]  Ivan Laptev,et al.  Is object localization for free? - Weakly-supervised learning with convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Jaime S. Cardoso,et al.  Closed Shortest Path in the Original Coordinates with an Application to Breast Cancer , 2015, Int. J. Pattern Recognit. Artif. Intell..

[30]  Hamid Jafarkhani,et al.  A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI , 2015, Medical Image Anal..

[31]  Max A. Viergever,et al.  Automatic Coronary Calcium Scoring in Cardiac CT Angiography Using Convolutional Neural Networks , 2015, MICCAI.

[32]  Paul A. Yushkevich,et al.  Multi-Atlas Segmentation with Joint Label Fusion , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Jitendra Malik,et al.  Region-Based Convolutional Networks for Accurate Object Detection and Segmentation , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Lubomir M. Hadjiiski,et al.  Characterization of mammographic masses based on level set segmentation with new image features and patient information. , 2007, Medical physics.

[35]  Le Lu,et al.  Computer Aided Diagnosis Using Multilevel Image Features on Large-Scale Evaluation , 2013, MCV.

[36]  Xiaojing Ye,et al.  Coarse-to-fine classification via parametric and nonparametric models for computer-aided diagnosis , 2011, CIKM '11.

[37]  Ivan Laptev,et al.  Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  Jinbo Bi,et al.  Effective 3D object detection and regression using probabilistic segmentation features in CT images , 2011, CVPR 2011.

[39]  Georg Langs,et al.  Mapping visual features to semantic profiles for retrieval in medical imaging , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Luc Van Gool,et al.  The Pascal Visual Object Classes Challenge: A Retrospective , 2014, International Journal of Computer Vision.

[41]  Ronald M. Summers,et al.  DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation , 2015, MICCAI.

[42]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[43]  Bram van Ginneken,et al.  Automatic classification of pulmonary peri-fissural nodules in computed tomography using an ensemble of 2D views and a convolutional neural network out-of-the-box , 2015, Medical Image Anal..

[44]  Ronald M. Summers,et al.  Interleaved Text/Image Deep Mining on a Large-Scale Radiology Image Database , 2017, Deep Learning and Convolutional Neural Networks for Medical Image Computing.

[45]  Charless C. Fowlkes,et al.  Do We Need More Training Data or Better Models for Object Detection? , 2012, BMVC.

[46]  Xinbo Gao,et al.  A deep feature based framework for breast masses classification , 2016, Neurocomputing.

[47]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[48]  David Dagan Feng,et al.  Feature-Based Image Patch Approximation for Lung Tissue Classification , 2013, IEEE Transactions on Medical Imaging.

[49]  Ronald M. Summers,et al.  Leveraging Mid-Level Semantic Boundary Cues for Automated Lymph Node Detection , 2015, MICCAI.

[50]  Wei Shen,et al.  Multi-scale Convolutional Neural Networks for Lung Nodule Classification , 2015, IPMI.

[51]  Gustavo Carneiro,et al.  Combining Multiple Dynamic Models and Deep Learning Architectures for Tracking the Left Ventricle Endocardium in Ultrasound Data , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Mia K Markey,et al.  A model-based framework for the detection of spiculated masses on mammography. , 2008, Medical physics.

[53]  Jitendra Malik,et al.  Indoor Scene Understanding with RGB-D Images: Bottom-up Segmentation, Object Detection and Semantic Segmentation , 2015, International Journal of Computer Vision.

[54]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[55]  Rainer Stotzka,et al.  An Example-Based System to Support the Segmentation of Stellate Lesions , 2005, Bildverarbeitung für die Medizin.

[56]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[57]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[58]  Gustavo Carneiro,et al.  The Segmentation of the Left Ventricle of the Heart From Ultrasound Data Using Deep Learning Architectures and Derivative-Based Search Methods , 2012, IEEE Transactions on Image Processing.

[59]  Heng Huang,et al.  Large Margin Local Estimate With Applications to Medical Image Classification , 2015, IEEE Transactions on Medical Imaging.

[60]  M. Masotti,et al.  A novel featureless approach to mass detection in digital mammograms based on support vector machines. , 2004, Physics in medicine and biology.

[61]  Dorin Comaniciu,et al.  Lymph Node Detection and Segmentation in Chest Ct Data Using Discriminative Learning and a Spatial Prior , 2022 .

[62]  Ronald M. Summers,et al.  Segmentation label propagation using deep convolutional neural networks and dense conditional random field , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[63]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[64]  S Tangaro,et al.  A completely automated CAD system for mass detection in a large mammographic database. , 2006, Medical physics.

[65]  Antonio Torralba,et al.  Small codes and large image databases for recognition , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[66]  Andrew Zisserman,et al.  Return of the Devil in the Details: Delving Deep into Convolutional Nets , 2014, BMVC.

[67]  Ronald M. Summers,et al.  Improving Computer-Aided Detection Using Convolutional Neural Networks and Random View Aggregation , 2015, IEEE Transactions on Medical Imaging.

[68]  Ben Glocker,et al.  Mediastinal atlas creation from 3-D chest computed tomography images: Application to automated detection and station mapping of lymph nodes , 2012, Medical Image Anal..

[69]  Anthony Maida,et al.  Natural Image Bases to Represent Neuroimaging Data , 2013, ICML.

[70]  Stefan Carlsson,et al.  CNN Features Off-the-Shelf: An Astounding Baseline for Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[71]  Gustavo Carneiro,et al.  Unregistered Multiview Mammogram Analysis with Pre-trained Deep Learning Models , 2015, MICCAI.

[72]  Xun Xu,et al.  Automatic Detection and Segmentation of Lymph Nodes From CT Data , 2012, IEEE Transactions on Medical Imaging.

[73]  Sepp Hochreiter,et al.  The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions , 1998, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[74]  Behrouz Minaei,et al.  Assessment of a novel mass detection algorithm in mammograms. , 2013, Journal of cancer research and therapeutics.

[75]  Xiaogang Wang,et al.  Medical image classification with convolutional neural network , 2014, 2014 13th International Conference on Control Automation Robotics & Vision (ICARCV).

[76]  Hao Chen,et al.  Automatic Fetal Ultrasound Standard Plane Detection Using Knowledge Transferred Recurrent Neural Networks , 2015, MICCAI.

[77]  Xiantong Zhen,et al.  Multi-scale deep networks and regression forests for direct bi-ventricular volume estimation , 2016, Medical Image Anal..

[78]  Arnau Oliver,et al.  A review of automatic mass detection and segmentation in mammographic images , 2010, Medical Image Anal..

[79]  R. Robb,et al.  The Lung Tissue Research Consortium: An extensive open database containing histological, clinical, and radiological data to study chronic lung disease , 2006, The Insight Journal.

[80]  Gregory Shakhnarovich,et al.  Feedforward semantic segmentation with zoom-out features , 2014, CVPR.

[81]  Brian B. Avants,et al.  The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) , 2015, IEEE Transactions on Medical Imaging.

[82]  Dinggang Shen,et al.  Deep Learning Based Imaging Data Completion for Improved Brain Disease Diagnosis , 2014, MICCAI.

[83]  Xiang Zhang,et al.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks , 2013, ICLR.

[84]  Hayit Greenspan,et al.  Chest pathology detection using deep learning with non-medical training , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[85]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).