Prospective Deployment of Deep Learning in MRI: A Framework for Important Considerations, Challenges, and Recommendations for Best Practices

Artificial intelligence algorithms based on principles of deep learning (DL) have made a large impact on the acquisition, reconstruction, and interpretation of MRI data. Despite the large number of retrospective studies using DL, there are fewer applications of DL in the clinic on a routine basis. To address this large translational gap, we review the recent publications to determine three major use cases that DL can have in MRI, namely, that of model‐free image synthesis, model‐based image reconstruction, and image or pixel‐level classification. For each of these three areas, we provide a framework for important considerations that consist of appropriate model training paradigms, evaluation of model robustness, downstream clinical utility, opportunities for future advances, as well recommendations for best current practices. We draw inspiration for this framework from advances in computer vision in natural imaging as well as additional healthcare fields. We further emphasize the need for reproducibility of research studies through the sharing of datasets and software.

[1]  Li Feng,et al.  MANTIS: Model‐Augmented Neural neTwork with Incoherent k‐space Sampling for efficient MR parameter mapping , 2019, Magnetic resonance in medicine.

[2]  Li Feng,et al.  SANTIS: Sampling-Augmented Neural neTwork with Incoherent Structure for MR image reconstruction , 2018, ArXiv.

[3]  Steen Moeller,et al.  Scan‐specific robust artificial‐neural‐networks for k‐space interpolation (RAKI) reconstruction: Database‐free deep learning for fast imaging , 2018, Magnetic resonance in medicine.

[4]  Jaakko Lehtinen,et al.  Noise2Noise: Learning Image Restoration without Clean Data , 2018, ICML.

[5]  Vahid Ghodrati,et al.  MR image reconstruction using deep learning: evaluation of network structure and loss functions. , 2019, Quantitative imaging in medicine and surgery.

[6]  Ronald M. Summers,et al.  Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New Technique , 2016 .

[7]  Peng Lai,et al.  Accelerating cardiac cine MRI using a deep learning‐based ESPIRiT reconstruction , 2019, Magnetic resonance in medicine.

[8]  Jin Hyung Lee,et al.  Diagnostic Accuracy of Quantitative Multi-Contrast 5-Minute Knee MRI Using Prospective Artificial Intelligence Image Quality Enhancement. , 2020, AJR. American journal of roentgenology.

[9]  Arvid Lundervold,et al.  An overview of deep learning in medical imaging focusing on MRI , 2018, Zeitschrift fur medizinische Physik.

[10]  C. Pal,et al.  Deep Learning: A Primer for Radiologists. , 2017, Radiographics : a review publication of the Radiological Society of North America, Inc.

[11]  Mubarak Shah,et al.  Norm-Preservation: Why Residual Networks Can Become Extremely Deep? , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Lorenzo Strigini,et al.  Effects of incorrect computer-aided detection (CAD) output on human decision-making in mammography. , 2004, Academic radiology.

[13]  Ian Pan,et al.  Deep Learning in Neuroradiology: A Systematic Review of Current Algorithms and Approaches for the New Wave of Imaging Technology. , 2020, Radiology. Artificial intelligence.

[14]  Florian Knoll,et al.  Artificial Intelligence in Musculoskeletal Imaging: Current Status and Future Directions. , 2019, AJR. American journal of roentgenology.

[15]  Timothy D. Solberg,et al.  Attention-Aware Discrimination for MR-to-CT Image Translation Using Cycle-Consistent Generative Adversarial Networks. , 2020, Radiology. Artificial intelligence.

[16]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[17]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Darvin Yi,et al.  Handling Missing MRI Input Data in Deep Learning Segmentation of Brain Metastases: A Multi-Center Study , 2019, ArXiv.

[19]  Jian Sun,et al.  ADMM-Net: A Deep Learning Approach for Compressive Sensing MRI , 2017, ArXiv.

[20]  Konstantinos Kamnitsas,et al.  Anatomically Constrained Neural Networks (ACNNs): Application to Cardiac Image Enhancement and Segmentation , 2017, IEEE Transactions on Medical Imaging.

[21]  Eric Van Reeth,et al.  Super-resolution in magnetic resonance imaging: A review , 2012 .

[22]  Jonathan I. Tamir,et al.  Unsupervised Deep Basis Pursuit: Learning inverse problems without ground-truth data , 2019, 1910.13110.

[23]  Carsten Rother,et al.  Can Virtual Contrast Enhancement in Brain MRI Replace Gadolinium?: A Feasibility Study. , 2019, Investigative radiology.

[24]  Andrew P. Leynes,et al.  Zero-Echo-Time and Dixon Deep Pseudo-CT (ZeDD CT): Direct Generation of Pseudo-CT Images for Pelvic PET/MRI Attenuation Correction Using Deep Convolutional Neural Networks with Multiparametric MRI , 2017, The Journal of Nuclear Medicine.

[25]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[26]  Michael G. Rabbat,et al.  Advancing machine learning for MR image reconstruction with an open competition: Overview of the 2019 fastMRI challenge , 2020, Magnetic resonance in medicine.

[27]  J. Pauly,et al.  Uncertainty Quantification in Deep MRI Reconstruction , 2019, IEEE Transactions on Medical Imaging.

[28]  Kawin Setsompop,et al.  DeepDTI: High-fidelity six-direction diffusion tensor imaging using deep learning , 2020, NeuroImage.

[29]  Florian Knoll,et al.  Artificial Intelligence for MR Image Reconstruction: An Overview for Clinicians , 2020, Journal of magnetic resonance imaging : JMRI.

[30]  Eric K. Gibbons,et al.  Utility of deep learning super‐resolution in the context of osteoarthritis MRI biomarkers , 2020, Journal of magnetic resonance imaging : JMRI.

[31]  John M. Pauly,et al.  Complex-Valued Convolutional Neural Networks for MRI Reconstruction , 2020, ArXiv.

[32]  J. Mongan,et al.  Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers. , 2020, Radiology. Artificial intelligence.

[33]  Jeff Wood,et al.  Super‐resolution musculoskeletal MRI using deep learning , 2018, Magnetic resonance in medicine.

[34]  Mehryar Mohri,et al.  L2 Regularization for Learning Kernels , 2009, UAI.

[35]  Valentina Pedoia,et al.  Rapid Knee MRI Acquisition and Analysis Techniques for Imaging Osteoarthritis , 2019, Journal of magnetic resonance imaging : JMRI.

[36]  Sebastian Kozerke,et al.  Deep variational network for rapid 4D flow MRI reconstruction , 2020, Nat. Mach. Intell..

[37]  Daniel Cremers,et al.  q-Space Deep Learning: Twelve-Fold Shorter and Model-Free Diffusion MRI Scans , 2016, IEEE Transactions on Medical Imaging.

[38]  Daniel Rueckert,et al.  A Deep Cascade of Convolutional Neural Networks for MR Image Reconstruction , 2017, IPMI.

[39]  E. Henriksen,et al.  The efficacy of using computer-aided detection (CAD) for detection of breast cancer in mammography screening: a systematic review , 2019, Acta radiologica.

[40]  Kevin Gimpel,et al.  A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks , 2016, ICLR.

[41]  Morteza Mardani,et al.  Deep Generative Adversarial Neural Networks for Compressive Sensing MRI , 2019, IEEE Transactions on Medical Imaging.

[42]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[43]  Mehmet Akcakaya,et al.  The International Workshop on Osteoarthritis Imaging Knee MRI Segmentation Challenge: A Multi-Institute Evaluation and Analysis Framework on a Standardized Dataset , 2020, Radiology. Artificial intelligence.

[44]  Daniel Rueckert,et al.  A Deep Cascade of Convolutional Neural Networks for Dynamic MR Image Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[45]  Ryanne A. Brown,et al.  Impact of a deep learning assistant on the histopathologic classification of liver cancer. , 2020, NPJ digital medicine.

[46]  Kawin Setsompop,et al.  Improving in vivo human cerebral cortical surface reconstruction using data-driven super-resolution. , 2020, Cerebral cortex.

[47]  Brian A. Hargreaves,et al.  Deep Learning Super-Resolution Enables Rapid Simultaneous Morphological and Quantitative Magnetic Resonance Imaging , 2018, MLMIR@MICCAI.

[48]  Jonathan I. Tamir,et al.  Variable-Density Single-Shot Fast Spin-Echo MRI with Deep Learning Reconstruction by Using Variational Networks. , 2018, Radiology.

[49]  Morteza Mardani,et al.  Compressed Sensing: From Research to Clinical Practice With Deep Neural Networks: Shortening Scan Times for Magnetic Resonance Imaging , 2020, IEEE Signal Processing Magazine.

[50]  H. Greenspan,et al.  Convolutional Neural Networks for Radiologic Images: A Radiologist's Guide. , 2019, Radiology.

[51]  Suyash Mohan,et al.  Artificial Intelligence System Approaching Neuroradiologist-level Differential Diagnosis Accuracy at Brain MRI. , 2020, Radiology.

[52]  J. Pauly,et al.  Deep learning enables reduced gadolinium dose for contrast‐enhanced brain MRI , 2018, Journal of magnetic resonance imaging : JMRI.

[53]  A. J. Mariano,et al.  Human–machine partnership with artificial intelligence for chest radiograph diagnosis , 2019, npj Digital Medicine.

[54]  Deepa Sheth,et al.  Artificial intelligence in the interpretation of breast cancer on MRI , 2019, Journal of magnetic resonance imaging : JMRI.

[55]  Brian A. Hargreaves,et al.  Technical Considerations for Semantic Segmentation in MRI using Convolutional Neural Networks , 2019, ArXiv.

[56]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[57]  Ryan L Brunsing,et al.  Deep residual network for off‐resonance artifact correction with application to pediatric body MRA with 3D cones , 2019, Magnetic resonance in medicine.

[58]  Yang Yang,et al.  Deep Learning Scaling is Predictable, Empirically , 2017, ArXiv.

[59]  J. Pauly,et al.  Wasserstein GANs for MR Imaging: From Paired to Unpaired Training , 2019, IEEE Transactions on Medical Imaging.

[60]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[61]  Sandeep Subramanian,et al.  Deep Complex Networks , 2017, ICLR.

[62]  Yan Xu,et al.  Deep learning of feature representation with multiple instance learning for medical image analysis , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[63]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[64]  Tolga Çukur,et al.  A Transfer‐Learning Approach for Accelerated MRI Using Deep Neural Networks , 2017, Magnetic resonance in medicine.

[65]  Vivek Muthurangu,et al.  Real‐time cardiovascular MR with spatio‐temporal artifact suppression using deep learning–proof of concept in congenital heart disease , 2018, Magnetic resonance in medicine.

[66]  Unsupervised Image Reconstruction using Deep Generative Adversarial Networks , 2020 .

[67]  Yongming Dai,et al.  Clinical Potential of UTE‐MRI for Assessing COVID‐19: Patient‐ and Lesion‐Based Comparative Analysis , 2020, Journal of magnetic resonance imaging : JMRI.

[68]  Christian F. Baumgartner,et al.  Clinical evaluation of fully automated thigh muscle and adipose tissue segmentation using a U-Net deep learning architecture in context of osteoarthritic knee pain , 2019, Magnetic Resonance Materials in Physics, Biology and Medicine.

[69]  K. Kinkel,et al.  Computer-aided detection (CAD) in mammography: does it help the junior or the senior radiologist? , 2005, European journal of radiology.

[70]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[71]  Ali Farhadi,et al.  Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping , 2020, ArXiv.

[72]  Ender Konukoglu,et al.  PHiSeg: Capturing Uncertainty in Medical Image Segmentation , 2019, MICCAI.

[73]  David Atkinson,et al.  ISMRM Raw data format: A proposed standard for MRI raw datasets , 2017, Magnetic resonance in medicine.

[74]  A. Ng,et al.  Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet , 2018, PLoS medicine.

[75]  Nigam H Shah,et al.  Ethics of Using and Sharing Clinical Imaging Data for Artificial Intelligence: A Proposed Framework. , 2020, Radiology.

[76]  Sina Honari,et al.  Distribution Matching Losses Can Hallucinate Features in Medical Image Translation , 2018, MICCAI.

[77]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[78]  Richard Kijowski,et al.  Fully Automated Diagnosis of Anterior Cruciate Ligament Tears on Knee MR Images by Using Deep Learning. , 2019, Radiology. Artificial intelligence.

[79]  Sandy H. Huang,et al.  Adversarial Attacks on Neural Network Policies , 2017, ICLR.

[80]  Guang Yang,et al.  Adversarial and Perceptual Refinement for Compressed Sensing MRI Reconstruction , 2018, MICCAI.

[81]  Xiaogang Wang,et al.  Saliency detection by multi-context deep learning , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[82]  Chi-Hieu Pham,et al.  Brain MRI super-resolution using deep 3D convolutional networks , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[83]  Kibok Lee,et al.  A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks , 2018, NeurIPS.

[84]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[85]  Andrew McCallum,et al.  Energy and Policy Considerations for Deep Learning in NLP , 2019, ACL.

[86]  Mathews Jacob,et al.  MoDL: Model-Based Deep Learning Architecture for Inverse Problems , 2017, IEEE Transactions on Medical Imaging.

[87]  Qianqian Zhang,et al.  MRI Gibbs‐ringing artifact reduction by means of machine learning using convolutional neural networks , 2019, Magnetic resonance in medicine.

[88]  Daniel K Sodickson,et al.  Assessment of the generalization of learned image reconstruction and the potential for transfer learning , 2019, Magnetic resonance in medicine.

[89]  Jong Chul Ye,et al.  Deep residual learning for compressed sensing MRI , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[90]  Bin Yang,et al.  Retrospective correction of motion‐affected MR images using deep learning frameworks , 2019, Magnetic resonance in medicine.

[91]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[92]  Yoshua Bengio,et al.  Unitary Evolution Recurrent Neural Networks , 2015, ICML.

[93]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[94]  Feng Shi,et al.  Brain MRI super resolution using 3D deep densely connected neural networks , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[95]  Thomas Pock,et al.  Learning a variational network for reconstruction of accelerated MRI data , 2017, Magnetic resonance in medicine.

[96]  Felix Eckstein,et al.  Five‐minute knee MRI for simultaneous morphometry and T2 relaxometry of cartilage and meniscus and for semiquantitative radiological assessment using double‐echo in steady‐state at 3T , 2018, Journal of magnetic resonance imaging : JMRI.

[97]  Ankur Taly,et al.  Axiomatic Attribution for Deep Networks , 2017, ICML.

[98]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[99]  Mateusz Buda,et al.  Deep learning in radiology: An overview of the concepts and a survey of the state of the art with focus on MRI , 2018, Journal of magnetic resonance imaging : JMRI.

[100]  Ganesh Adluru,et al.  Simultaneous NODDI and GFA parameter map generation from subsampled q‐space imaging using deep learning , 2018, Magnetic resonance in medicine.

[101]  Francesco Renna,et al.  On instabilities of deep learning in image reconstruction and the potential costs of AI , 2019, Proceedings of the National Academy of Sciences.

[102]  Brian A. Hargreaves,et al.  Time-saving opportunities in knee osteoarthritis: T2 mapping and structural imaging of the knee using a single 5-min MRI scan , 2019, European Radiology.

[103]  Bruce R. Rosen,et al.  Image reconstruction by domain-transform manifold learning , 2017, Nature.

[104]  Deborah Silver,et al.  Feature Visualization , 1994, Scientific Visualization.

[105]  Brian A Hargreaves,et al.  Combined 5‐minute double‐echo in steady‐state with separated echoes and 2‐minute proton‐density‐weighted 2D FSE sequence for comprehensive whole‐joint knee MRI assessment , 2019, Journal of magnetic resonance imaging : JMRI.

[106]  Tammy Riklin-Raviv,et al.  Ensemble of expert deep neural networks for spatio‐temporal denoising of contrast‐enhanced MRI sequences , 2017, Medical Image Anal..