The present and future of deep learning in radiology.

The advent of Deep Learning (DL) is poised to dramatically change the delivery of healthcare in the near future. Not only has DL profoundly affected the healthcare industry it has also influenced global businesses. Within a span of very few years, advances such as self-driving cars, robots performing jobs that are hazardous to human, and chat bots talking with human operators have proved that DL has already made large impact on our lives. The open source nature of DL and decreasing prices of computer hardware will further propel such changes. In healthcare, the potential is immense due to the need to automate the processes and evolve error free paradigms. The sheer quantum of DL publications in healthcare has surpassed other domains growing at a very fast pace, particular in radiology. It is therefore imperative for the radiologists to learn about DL and how it differs from other approaches of Artificial Intelligence (AI). The next generation of radiology will see a significant role of DL and will likely serve as the base for augmented radiology (AR). Better clinical judgement by AR will help in improving the quality of life and help in life saving decisions, while lowering healthcare costs. A comprehensive review of DL as well as its implications upon the healthcare is presented in this review. We had analysed 150 articles of DL in healthcare domain from PubMed, Google Scholar, and IEEE EXPLORE focused in medical imagery only. We have further examined the ethic, moral and legal issues surrounding the use of DL in medical imaging.

[1]  Luca Maria Gambardella,et al.  Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks , 2013, MICCAI.

[2]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[3]  Jasjit S. Suri,et al.  Greedy Technique and Its Validation for Fusion of Two Segmentation Paradigms Leads to an Accurate Intima–Media Thickness Measure in Plaque Carotid Arterial Ultrasound , 2010 .

[4]  Raúl San José Estépar,et al.  Disease Staging and Prognosis in Smokers Using Deep Learning in Chest Computed Tomography , 2018, American journal of respiratory and critical care medicine.

[5]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[6]  Nilanjan Dey,et al.  Automated stratification of liver disease in ultrasound: An online accurate feature classification paradigm , 2016, Comput. Methods Programs Biomed..

[7]  Manolis Kellis,et al.  Deep learning for regulatory genomics , 2015, Nature Biotechnology.

[8]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[9]  Sotirios A. Tsaftaris,et al.  Medical Image Computing and Computer Assisted Intervention , 2017 .

[10]  Bradley J. Erickson,et al.  Automated Segmentation of Hyperintense Regions in FLAIR MRI Using Deep Learning , 2016, Tomography.

[11]  Yaozong Gao,et al.  LINKS: Learning-based multi-source IntegratioN frameworK for Segmentation of infant brain images , 2014, NeuroImage.

[12]  Azriel Rosenfeld,et al.  Computer Vision , 1988, Adv. Comput..

[13]  Jasjit S. Suri,et al.  Extreme Learning Machine Framework for Risk Stratification of Fatty Liver Disease Using Ultrasound Tissue Characterization , 2017, Journal of Medical Systems.

[14]  U Rajendra Acharya,et al.  Ovarian Tumor Characterization using 3D Ultrasound , 2012, Technology in cancer research & treatment.

[15]  Jasjit S. Suri,et al.  Characterization of a Completely User-Independent Algorithm for Carotid Artery Segmentation in 2-D Ultrasound Images , 2007, IEEE Transactions on Instrumentation and Measurement.

[16]  Dinggang Shen,et al.  Deep Learning Based Imaging Data Completion for Improved Brain Disease Diagnosis , 2014, MICCAI.

[17]  Olivier Clatz,et al.  Spatial Decision Forests for MS Lesion Segmentation in Multi-Channel MR Images , 2010, MICCAI.

[18]  R. Malach,et al.  Intersubject Synchronization of Cortical Activity During Natural Vision , 2004, Science.

[19]  Joseph Y. Lo,et al.  Anomaly detection for medical images based on a one-class classification , 2018, Medical Imaging.

[20]  Vladimir Vapnik,et al.  An overview of statistical learning theory , 1999, IEEE Trans. Neural Networks.

[21]  Frank C. Leeming,et al.  Deep Learning and Emotion in Serious Games , 2009 .

[22]  D. Mozaffarian,et al.  Heart disease and stroke statistics--2010 update: a report from the American Heart Association. , 2010, Circulation.

[23]  Yaozong Gao,et al.  LINKS: Learning-based multi-source IntegratioN frameworK for Segmentation of infant brain images , 2015, NeuroImage.

[24]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  R. Bammer,et al.  Real‐time diffusion‐perfusion mismatch analysis in acute stroke , 2010, Journal of magnetic resonance imaging : JMRI.

[26]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[27]  Hamid Jafarkhani,et al.  A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI , 2015, Medical Image Anal..

[28]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Gustavo Carneiro,et al.  Combining deep learning and level set for the automated segmentation of the left ventricle of the heart from cardiac cine magnetic resonance , 2017, Medical Image Anal..

[30]  K. Mouridsen,et al.  Prediction of Tissue Outcome and Assessment of Treatment Effect in Acute Ischemic Stroke Using Deep Learning , 2018, Stroke.

[31]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[32]  D. Hubel,et al.  Shape and arrangement of columns in cat's striate cortex , 1963, The Journal of physiology.

[33]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Jasjit S. Suri,et al.  Symtosis: A liver ultrasound tissue characterization and risk stratification in optimized deep learning paradigm , 2018, Comput. Methods Programs Biomed..

[35]  Cheng-Yuan Liou,et al.  Autoencoder for words , 2014, Neurocomputing.

[36]  Byunghan Lee,et al.  Deep learning in bioinformatics , 2016, Briefings Bioinform..

[37]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[38]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[39]  B. L. Betechuoh,et al.  Autoencoder networks for HIV classification , 2006 .

[40]  Aly A. Valliani,et al.  Deep Learning and Neurology: A Systematic Review , 2019, Neurology and Therapy.

[41]  N. Rajpoot,et al.  Locality Sensitive Deep Learning for Detection and Classification of Nuclei in Routine Colon Cancer Histology Images , 2016, IEEE Transactions on Medical Imaging.

[42]  Siva R K Narla,et al.  The Evolution of Connected Vehicle Technology: From Smart Drivers to Smart Cars to....Self-Driving Cars. , 2013 .

[43]  Ha Hong,et al.  Performance-optimized hierarchical models predict neural responses in higher visual cortex , 2014, Proceedings of the National Academy of Sciences.

[44]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  L. Saba,et al.  Immune Checkpoint Inhibitor-Induced Pancreatic Injury: Imaging Findings and Literature Review , 2020, Targeted Oncology.

[46]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[47]  Jialin Peng,et al.  Automatic 3D liver segmentation based on deep learning and globally optimized surface evolution , 2016, Physics in medicine and biology.

[48]  Hao Chen,et al.  Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks , 2016, IEEE Transactions on Medical Imaging.

[49]  Haesung Yoon,et al.  Performance of deep learning-based algorithm for detection of ileocolic intussusception on abdominal radiographs of young children , 2019, Scientific Reports.

[50]  Ajay Gupta,et al.  Plaque Tissue Morphology-Based Stroke Risk Stratification Using Carotid Ultrasound: A Polling-Based PCA Learning Paradigm , 2017, Journal of Medical Systems.

[51]  W. MacNee,et al.  Evaluation of COPD Longitudinally to Identify Predictive Surrogate End-points (ECLIPSE) , 2008, European Respiratory Journal.

[52]  Dayong Wang,et al.  Deep Learning for Identifying Metastatic Breast Cancer , 2016, ArXiv.

[53]  M. Kremer,et al.  The O-Ring Theory of Economic Development , 1993 .

[54]  Marcel A J van Gerven,et al.  Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream , 2015, The Journal of Neuroscience.

[55]  K Miyasaka,et al.  [Picture archiving and communication system]. , 1989, Kokyu to junkan. Respiration & circulation.

[56]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[57]  Wenqing Sun,et al.  Computer aided lung cancer diagnosis with deep learning algorithms , 2016, SPIE Medical Imaging.

[58]  Zhongming Liu,et al.  Influences of High-Level Features, Gaze, and Scene Transitions on the Reliability of BOLD Responses to Natural Movie Stimuli , 2016, PloS one.

[59]  Aldo A. Faisal,et al.  Artificial Intelligence, Data Sensors and Interconnectivity: Future Opportunities for Heart Failure , 2019, Cardiac failure review.

[60]  Trevor Darrell,et al.  Deep Layer Aggregation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[61]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[62]  Jasjit S. Suri,et al.  2083667 Online System For Liver Disease Classification In Ultrasound , 2015 .

[63]  Charlene Liew.  The future of radiology augmented with Artificial Intelligence: A strategy for success. , 2018, European journal of radiology.

[64]  Wen-Huang Cheng,et al.  Computer-aided classification of lung nodules on computed tomography images via deep learning technique , 2015, OncoTargets and therapy.

[65]  Greg Humphreys,et al.  How GPUs Work , 2007, Computer.

[66]  Fucang Jia,et al.  Automatic Segmentation of Liver Tumor in CT Images with Deep Convolutional Neural Networks , 2015 .

[67]  Roger C. Tam,et al.  Manifold Learning of Brain MRIs by Deep Learning , 2013, MICCAI.

[68]  U. Rajendra Acharya,et al.  Ovarian Tumor Characterization and Classification Using Ultrasound—A New Online Paradigm , 2013, Journal of Digital Imaging.

[69]  Fabio A. González,et al.  Automatic detection of invasive ductal carcinoma in whole slide images with convolutional neural networks , 2014, Medical Imaging.

[70]  Nico Karssemeijer,et al.  Unsupervised Deep Learning Applied to Breast Density Segmentation and Mammographic Risk Scoring , 2016, IEEE Transactions on Medical Imaging.

[71]  Paul F. Whelan,et al.  Automatic segmentation of the left ventricle cavity and myocardium in MRI data , 2006, Comput. Biol. Medicine.

[72]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[73]  Razvan Pascanu,et al.  Theano: A CPU and GPU Math Compiler in Python , 2010, SciPy.

[74]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[75]  Seunghoon Hong,et al.  Learning Deconvolution Network for Semantic Segmentation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[76]  Bram van Ginneken,et al.  Fast Convolutional Neural Network Training Using Selective Data Sampling: Application to Hemorrhage Detection in Color Fundus Images , 2016, IEEE Transactions on Medical Imaging.

[77]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[78]  Jean-Yves Audibert.  Optimization for Machine Learning , 1995 .

[79]  Xiantong Zhen,et al.  Multi-scale deep networks and regression forests for direct bi-ventricular volume estimation , 2016, Medical Image Anal..

[80]  Jasjit S Suri,et al.  Economic impact of telemedicine: a survey. , 2005, Studies in health technology and informatics.

[81]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[82]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[83]  J. Suri.  Two-dimensional fast magnetic resonance brain segmentation , 2001, IEEE Engineering in Medicine and Biology Magazine.

[84]  Liam Paninski,et al.  Statistical models for neural encoding, decoding, and optimal stimulus design. , 2007, Progress in brain research.

[85]  J. Gallant,et al.  Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies , 2011, Current Biology.

[86]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[87]  Yoshua Bengio,et al.  Unsupervised and Transfer Learning Challenge: a Deep Learning Approach , 2011, ICML Unsupervised and Transfer Learning.

[88]  Ronald M. Summers,et al.  A New 2.5D Representation for Lymph Node Detection Using Random Sets of Deep Convolutional Neural Network Observations , 2014, MICCAI.

[89]  Marios Anthimopoulos,et al.  Lung Pattern Classification for Interstitial Lung Diseases Using a Deep Convolutional Neural Network , 2016, IEEE Transactions on Medical Imaging.

[90]  Christian Wachinger,et al.  DeepNAT: Deep convolutional neural network for segmenting neuroanatomy , 2017, NeuroImage.

[91]  Rasool Fakoor,et al.  Using deep learning to enhance cancer diagnosis and classication , 2013 .

[92]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[93]  U. Rajendra Acharya,et al.  Symptomatic vs. Asymptomatic Plaque Classification in Carotid Ultrasound , 2012, Journal of Medical Systems.

[94]  Tim Leiner,et al.  Dilated Convolutional Neural Networks for Cardiovascular MR Segmentation in Congenital Heart Disease , 2016, RAMBO+HVSMR@MICCAI.

[95]  Daniel L. K. Yamins,et al.  Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition , 2014, PLoS Comput. Biol..

[96]  Guang-Zhong Yang,et al.  Deep Learning for Health Informatics , 2017, IEEE Journal of Biomedical and Health Informatics.

[97]  Anne E Carpenter,et al.  Opportunities and obstacles for deep learning in biology and medicine , 2017, bioRxiv.

[98]  Jonathan C. Cohen,et al.  Prevalence of hepatic steatosis in an urban population in the United States: Impact of ethnicity , 2004, Hepatology.

[99]  Kumar,et al.  Neural Networks a Classroom Approach , 2004 .

[100]  Yang Song,et al.  Computer‐aided diagnosis of prostate cancer using a deep convolutional neural network from multiparametric MRI , 2018, Journal of magnetic resonance imaging : JMRI.

[101]  T. Malone,et al.  Essays on collective intelligence , 2016 .

[102]  Jasjit S Suri,et al.  State-of-the-art review on deep learning in medical imaging. , 2019, Frontiers in bioscience.

[103]  Yoshua Bengio,et al.  Deep Learning of Representations for Unsupervised and Transfer Learning , 2011, ICML Unsupervised and Transfer Learning.

[104]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[105]  Xiaogang Wang,et al.  Deep Learning Face Representation from Predicting 10,000 Classes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[106]  Youngjin Yoo,et al.  Modeling the Variability in Brain Morphology and Lesion Distribution in Multiple Sclerosis by Deep Learning , 2014, MICCAI.

[107]  Jasjit S. Suri,et al.  Evaluation of Carotid Wall Thickness by using Computed Tomography and Semiautomated Ultrasonographic Software , 2011 .

[108]  Richard O’Neil,et al.  Convolution operators and $L(p,q)$ spaces , 1963 .

[109]  R K Narla Siva.  コネクテッドビークル技術の進化:スマートドライバーからスマートカーへ...自律運転自動車へ | 文献情報 | J-GLOBAL 科学技術総合リンクセンター , 2013 .

[110]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[111]  Anil K. Jain,et al.  Unsupervised texture segmentation using Gabor filters , 1990, 1990 IEEE International Conference on Systems, Man, and Cybernetics Conference Proceedings.

[112]  Angel Cruz-Roa,et al.  Mitosis detection in breast cancer pathology images by combining handcrafted and convolutional neural network features , 2014, Journal of medical imaging.

[113]  Jasjit S Suri,et al.  Data mining framework for fatty liver disease classification in ultrasound: A hybrid feature extraction paradigm. , 2012, Medical physics.

[114]  Martin Cadík,et al.  FFT and Convolution Performance in Image Filtering on GPU , 2006, Tenth International Conference on Information Visualisation (IV'06).

[115]  Christian Igel,et al.  Deep Feature Learning for Knee Cartilage Segmentation Using a Triplanar Convolutional Neural Network , 2013, MICCAI.

[116]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[117]  Jasjit S. Suri,et al.  Computer Vision, Pattern Recognition and Image Processing in Left Ventricle Segmentation: The Last 50 Years , 2000, Pattern Analysis & Applications.

[118]  Geoffrey E. Hinton.  Deep belief networks , 2009, Scholarpedia.

[119]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[120]  Jasjit S. Suri,et al.  Deep learning strategy for accurate carotid intima-media thickness measurement: An ultrasound study on Japanese diabetic cohort , 2018, Comput. Biol. Medicine.