Deep learning application engine (DLAE): Development and integration of deep learning algorithms in medical imaging

Abstract Herein we introduce a deep learning (DL) application engine (DLAE) system concept, present potential uses of it, and describe pathways for its integration in clinical workflows. An open-source software application was developed to provide a code-free approach to DL for medical imaging applications. DLAE supports several DL techniques used in medical imaging, including convolutional neural networks, fully convolutional networks, generative adversarial networks, and bounding box detectors. Several example applications using clinical images were developed and tested to demonstrate the capabilities of DLAE. Additionally, a model deployment example was demonstrated in which DLAE was used to integrate two trained models into a commercial clinical software package.

[1]  Verónica Vilaplana,et al.  Brain MRI super-resolution using 3D generative adversarial networks , 2018, ArXiv.

[2]  Quanzheng Li,et al.  Attenuation correction for brain PET imaging using deep neural network based on Dixon and ZTE MR images , 2017, Physics in medicine and biology.

[3]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[4]  Rajat J Kudchadker,et al.  Multi-Tasking Neural Networks for Anatomy Segmentation in Prostate Brachytherapy MRI , 2019, Brachytherapy.

[5]  Dwarikanath Mahapatra,et al.  Image super-resolution using progressive generative adversarial networks for medical image analysis , 2019, Comput. Medical Imaging Graph..

[6]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[8]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[9]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Guang Yang,et al.  DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction , 2018, IEEE Transactions on Medical Imaging.

[11]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[12]  A. McMillan,et al.  Deep learning Mr imaging–based attenuation correction for PeT/Mr imaging 1 , 2017 .

[13]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[14]  François Chollet,et al.  Keras: The Python Deep Learning library , 2018 .

[15]  Pabitra Mitra,et al.  SUMNet: Fully Convolutional Model For Fast Segmentation of Anatomical Structures in Ultrasound Volumes , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[16]  Won-Ki Jeong,et al.  Compressed Sensing MRI Reconstruction Using a Generative Adversarial Network With a Cyclic Loss , 2017, IEEE Transactions on Medical Imaging.

[17]  Daniel Lévy,et al.  Breast Mass Classification from Mammograms using Deep Convolutional Neural Networks , 2016, ArXiv.

[18]  Ehsan Samei,et al.  Patient‐specific quantification of image quality: An automated technique for measuring the distribution of organ Hounsfield units in clinical chest CT images , 2017, Medical physics.

[19]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[20]  J. Boxerman,et al.  Physiologic MRI for assessment of response to therapy and prognosis in glioblastoma. , 2016, Neuro-oncology.

[21]  Jie Fu,et al.  Deep learning approaches using 2D and 3D convolutional neural networks for generating male pelvic synthetic CT from MRI. , 2018, Medical physics.

[22]  Tomas Kron,et al.  Deep Learning Renal Segmentation for Fully Automated Radiation Dose Estimation in Unsealed Source Therapy , 2018, Front. Oncol..

[23]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[24]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[25]  Jitendra Malik,et al.  Region-Based Convolutional Networks for Accurate Object Detection and Segmentation , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Guiying Wang,et al.  Identification of Metastatic Lymph Nodes in MR Imaging with Faster Region-Based Convolutional Neural Networks. , 2018, Cancer research.

[27]  Xiaoying Wang,et al.  Fully automatic segmentation on prostate MR images based on cascaded fully convolution network , 2018, Journal of magnetic resonance imaging : JMRI.

[28]  Jeremiah W Sanders,et al.  Development and clinical implementation of SeedNet: A sliding‐window convolutional neural network for radioactive seed identification in MRI‐assisted radiosurgery (MARS) , 2019, Magnetic resonance in medicine.

[29]  Steffen Renisch,et al.  Pseudo-CT image generation from mDixon MRI images using fully convolutional neural networks , 2019, Medical Imaging.

[30]  Christoph Meinel,et al.  Conditional Adversarial Network for Semantic Segmentation of Brain Tumor , 2017, ArXiv.

[31]  Birgi Tamersoy,et al.  Generating Synthetic X-Ray Images of a Person from the Surface Geometry , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[32]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[33]  Christoph Meinel,et al.  Conditional Generative Refinement Adversarial Networks for Unbalanced Medical Image Semantic Segmentation , 2018, ArXiv.

[34]  Bin Yang,et al.  MedGAN: Medical Image Translation using GANs , 2018, Comput. Medical Imaging Graph..

[35]  Morteza Mardani,et al.  Deep Generative Adversarial Neural Networks for Compressive Sensing MRI , 2019, IEEE Transactions on Medical Imaging.

[36]  Ehsan Samei,et al.  Automated, patient-specific estimation of regional imparted energy and dose from tube current modulated computed tomography exams across 13 protocols , 2017, Journal of medical imaging.

[37]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[38]  Jose Dolz,et al.  Few-shot 3D Multi-modal Medical Image Segmentation using Generative Adversarial Learning , 2018, ArXiv.

[39]  Achim Seeger,et al.  Comparison of three different MR perfusion techniques and MR spectroscopy for multiparametric assessment in distinguishing recurrent high-grade gliomas from stable disease. , 2013, Academic radiology.

[40]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Konstantinos Kamnitsas,et al.  Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation , 2016, Medical Image Anal..

[42]  Danni Ai,et al.  Multichannel Fully Convolutional Network for Coronary Artery Segmentation in X-Ray Angiograms , 2018, IEEE Access.

[43]  T. Bathala,et al.  Parallel imaging compressed sensing for accelerated imaging and improved signal-to-noise ratio in MRI-based postimplant dosimetry of prostate brachytherapy. , 2018, Brachytherapy.

[44]  Jingfei Ma,et al.  SeedNet for Automated Detection and Localization of Radioactive Seeds in Post-Implant MRI: A Comparison with and without the use of An Endorectal Coil for Imaging , 2018, Brachytherapy.

[45]  Shunxing Bao,et al.  Fully convolutional neural networks improve abdominal organ segmentation , 2018, Medical Imaging.

[46]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  R. Nishikawa,et al.  Automated mammographic breast density estimation using a fully convolutional network , 2018, Medical physics.

[48]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[49]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[51]  Andrew P. Leynes,et al.  Zero-Echo-Time and Dixon Deep Pseudo-CT (ZeDD CT): Direct Generation of Pseudo-CT Images for Pelvic PET/MRI Attenuation Correction Using Deep Convolutional Neural Networks with Multiparametric MRI , 2017, The Journal of Nuclear Medicine.

[52]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Ben Glocker,et al.  Automated cardiovascular magnetic resonance image analysis with fully convolutional networks , 2017, Journal of Cardiovascular Magnetic Resonance.

[54]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[55]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Ehsan Samei,et al.  Patient-specific quantification of image quality: An automated method for measuring spatial resolution in clinical CT images. , 2016, Medical physics.

[57]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[58]  Ehsan Samei,et al.  Image noise and dose performance across a clinical population: Patient size adaptation as a metric of CT performance , 2017, Medical physics.

[59]  Merel M. Jung,et al.  Inferring PET from MRI with pix2pix , 2018 .

[60]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[61]  Nassir Navab,et al.  Automatic Segmentation of Kidneys using Deep Learning for Total Kidney Volume Quantification in Autosomal Dominant Polycystic Kidney Disease , 2017, Scientific Reports.

[62]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[64]  Richard Kijowski,et al.  A deep learning approach for 18F-FDG PET attenuation correction , 2018, EJNMMI Physics.

[65]  Dean C. Barratt,et al.  Automatic Multi-Organ Segmentation on Abdominal CT With Dense V-Networks , 2018, IEEE Transactions on Medical Imaging.

[66]  Vincent Dumoulin,et al.  Deconvolution and Checkerboard Artifacts , 2016 .

[67]  Deniz Erdogmus,et al.  Tversky Loss Function for Image Segmentation Using 3D Fully Convolutional Deep Networks , 2017, MLMI@MICCAI.

[68]  Daniel L. Rubin,et al.  Deep learning enables automatic detection and segmentation of brain metastases on multisequence MRI , 2019, Journal of magnetic resonance imaging : JMRI.

[69]  Robby T. Tan,et al.  Convolutional Neural Networks for Automated Fracture Detection and Localization on Wrist Radiographs. , 2019, Radiology. Artificial intelligence.

[70]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[71]  Xiao Han,et al.  MR‐based synthetic CT generation using a deep convolutional neural network method , 2017, Medical physics.

[72]  Ehsan Samei,et al.  Estimating detectability index in vivo: development and validation of an automated methodology , 2017, Journal of medical imaging.

[73]  Xiaogang Wang,et al.  Medical image classification with convolutional neural network , 2014, 2014 13th International Conference on Control Automation Robotics & Vision (ICARCV).

[74]  Nima Tajbakhsh,et al.  Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? , 2016, IEEE Transactions on Medical Imaging.

[75]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).