A survey on deep learning in medical image reconstruction

Abstract Medical image reconstruction aims to acquire high-quality medical images for clinical usage at minimal cost and risk to the patients. Deep learning and its applications in medical imaging, especially in image reconstruction have received considerable attention in the literature in recent years. This study reviews records obtained electronically through the leading scientific databases (Magnetic Resonance Imaging journal, Google Scholar, Scopus, Science Direct, Elsevier, and from other journal publications) searched using three sets of keywords: (1) Deep learning, image reconstruction, medical imaging; (2) Medical imaging, Deep learning, Image reconstruction; (3) Open science, Open imaging data, Open software. The articles reviewed revealed that deep learning-based reconstruction methods improve the quality of reconstructed images qualitatively and quantitatively. However, deep learning techniques are generally computationally expensive, require large amounts of training datasets, lack decent theory to explain why the algorithms work, and have issues of generalization and robustness. The challenge of lack of enough training datasets is currently being addressed by using transfer learning techniques.

[1]  Michael Unser,et al.  CNN-Based Projected Gradient Descent for Consistent CT Image Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[2]  Mathews Jacob,et al.  Model based image reconstruction using deep learned priors (MODL) , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[3]  Hai Su,et al.  Deep Learning in Microscopy Image Analysis: A Survey , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Jong Chul Ye,et al.  Efficient B-Mode Ultrasound Image Reconstruction From Sub-Sampled RF Data Using Deep Learning , 2017, IEEE Transactions on Medical Imaging.

[5]  Francesco Renna,et al.  On instabilities of deep learning in image reconstruction and the potential costs of AI , 2019, Proceedings of the National Academy of Sciences.

[6]  Jin Liu,et al.  Deep iterative reconstruction estimation (DIRE): approximate iterative reconstruction estimation for low dose CT imaging , 2019, Physics in medicine and biology.

[7]  Jong Chul Ye,et al.  Deep learning with domain adaptation for accelerated projection‐reconstruction MR , 2018, Magnetic resonance in medicine.

[8]  Hidekata Hontani,et al.  Dynamic PET Image Reconstruction Using Nonnegative Matrix Factorization Incorporated With Deep Image Prior , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[9]  Dong Liang,et al.  Deep Magnetic Resonance Image Reconstruction: Inverse Problems Meet Neural Networks , 2020, IEEE Signal Processing Magazine.

[10]  Ghassan Hamarneh,et al.  Deep Learning Based Image Reconstruction for Diffuse Optical Tomography , 2018, MLMIR@MICCAI.

[11]  Hu Chen,et al.  Low-dose CT via convolutional neural network. , 2017, Biomedical optics express.

[12]  Namkug Kim,et al.  Deep Learning in Medical Imaging , 2019, Neurospine.

[13]  Brett Byram,et al.  Physics and data driven models for ultrasound image reconstruction , 2017, 2017 51st Asilomar Conference on Signals, Systems, and Computers.

[14]  Amir Akramin Shafie,et al.  MRI Reconstruction Using Discrete Fourier Transform: A tutorial , 2008 .

[15]  Daniel Rueckert,et al.  Convolutional Recurrent Neural Networks for Dynamic MR Image Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[16]  Jian Zhou,et al.  Deep Learning-based CT Image Reconstruction: Initial Evaluation Targeting Hypovascular Hepatic Metastases. , 2019, Radiology. Artificial intelligence.

[17]  Loic A. Royer,et al.  Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction , 2018, Nature Methods.

[18]  Hongming Shan,et al.  Deep Encoder-Decoder Adversarial Reconstruction (DEAR) Network for 3D CT from Few-View Data , 2019, Bioengineering.

[19]  Kerstin Hammernik,et al.  Machine learning for image reconstruction , 2020 .

[20]  Mei-Ling Shyu,et al.  A Survey on Deep Learning , 2018, ACM Comput. Surv..

[21]  Ken D. Sauer,et al.  2.5D Deep Learning For CT Image Reconstruction Using A Multi-GPU Implementation , 2018, 2018 52nd Asilomar Conference on Signals, Systems, and Computers.

[22]  Hyun Wook Park,et al.  ETER-net: End to End MR Image Reconstruction Using Recurrent Neural Network , 2018, MLMIR@MICCAI.

[23]  Hyunjin Park,et al.  Prospects of deep learning for medical imaging , 2018, Precision and Future Medicine.

[24]  June-Goo Lee,et al.  Deep Learning in Medical Imaging: General Overview , 2017, Korean journal of radiology.

[25]  John Sartori,et al.  Approximate Communication , 2018, ACM Comput. Surv..

[26]  Dragica Radosav,et al.  Deep Learning and Medical Diagnosis: A Review of Literature , 2018, Multimodal Technol. Interact..

[27]  X. Zhai,et al.  LoTToR: An Algorithm for Missing-Wedge Correction of the Low-Tilt Tomographic 3D Reconstruction of a Single-Molecule Structure , 2020, Scientific Reports.

[28]  Bin Dong,et al.  A Review on Deep Learning in Medical Image Reconstruction , 2019, Journal of the Operations Research Society of China.

[29]  Wei Chen,et al.  A Transfer Learning Approach for Microstructure Reconstruction and Structure-property Predictions , 2018, Scientific Reports.

[30]  Guang Yang,et al.  DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction , 2018, IEEE Transactions on Medical Imaging.

[31]  Jian Zhou,et al.  Deep learning reconstruction improves image quality of abdominal ultra-high-resolution CT , 2019, European Radiology.

[32]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[33]  Nikos Paragios,et al.  U-ReSNet: Ultimate Coupling of Registration and Segmentation with Deep Nets , 2019, MICCAI.

[34]  Jing Liu,et al.  Self-attention convolutional neural network for improved MR image reconstruction , 2019, Inf. Sci..

[35]  Jong Hoon Kim,et al.  Penalized PET Reconstruction Using Deep Learning Prior and Local Linear Fitting , 2018, IEEE Transactions on Medical Imaging.

[36]  Changde Du,et al.  Sharing deep generative representation for perceived image reconstruction from human brain activity , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[37]  Jin Liu,et al.  Artifact Removal using Improved GoogLeNet for Sparse-view CT Reconstruction , 2018, Scientific Reports.

[38]  Jin Keun Seo,et al.  Deep learning for undersampled MRI reconstruction , 2017, Physics in medicine and biology.

[39]  Jianan Cui,et al.  Deep reconstruction model for dynamic PET images , 2017, PloS one.

[40]  Yuko Nakamura,et al.  Deep Learning Reconstruction at CT: Phantom Study of the Image Characteristics. , 2020, Academic radiology.

[41]  Won-Ki Jeong,et al.  Compressed Sensing MRI Reconstruction Using a Generative Adversarial Network With a Cyclic Loss , 2017, IEEE Transactions on Medical Imaging.

[42]  Daniel S. Weller,et al.  Image quality affects deep learning reconstruction of MRI , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[43]  Michael J. Zickar,et al.  Open Science Is Robust Science , 2018, Industrial and Organizational Psychology.

[44]  John W. Paisley,et al.  Compressed Sensing MRI Using a Recursive Dilated Network , 2018, AAAI.

[45]  Marcin Lewandowski,et al.  WaveFlow-Towards Integration of Ultrasound Processing with Deep Learning , 2018, 2018 IEEE International Ultrasonics Symposium (IUS).

[46]  Guohua Shen,et al.  Deep image reconstruction from human brain activity , 2017, bioRxiv.

[47]  Zhiqiang Chen,et al.  Few-View CT reconstruction method based on deep learning , 2016, 2016 IEEE Nuclear Science Symposium, Medical Imaging Conference and Room-Temperature Semiconductor Detector Workshop (NSS/MIC/RTSD).

[48]  Marcel van Gerven,et al.  Reconstructing perceived faces from brain activations with deep adversarial neural decoding , 2017, NIPS.

[49]  Steen Moeller,et al.  Deep-Learning Methods for Parallel Magnetic Resonance Imaging Reconstruction: A Survey of the Current Approaches, Trends, and Issues , 2020, IEEE Signal Processing Magazine.

[50]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[51]  Tolga Çukur,et al.  A Transfer‐Learning Approach for Accelerated MRI Using Deep Neural Networks , 2017, Magnetic resonance in medicine.

[52]  Jongha Lee,et al.  Deep-Neural-Network-Based Sinogram Synthesis for Sparse-View CT Image Reconstruction , 2018, IEEE Transactions on Radiation and Plasma Medical Sciences.

[53]  Jian Fu,et al.  A deep learning reconstruction framework for X-ray computed tomography with incomplete data , 2019, PloS one.

[54]  Daniel Rueckert,et al.  A Deep Cascade of Convolutional Neural Networks for Dynamic MR Image Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[55]  Leslie Ying,et al.  Accelerating magnetic resonance imaging via deep learning , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[56]  Jun Xiao,et al.  Deep Learning Image Reconstruction Simulation for Electromagnetic Tomography , 2018, IEEE Sensors Journal.

[57]  Faisal Mahmood,et al.  Topographical reconstructions from monocular optical colonoscopy images via deep learning , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[58]  Ming Du,et al.  Flexible Learning-Free Segmentation and Reconstruction of Neural Volumes , 2018, Scientific Reports.

[59]  Aydogan Ozcan,et al.  Deep-Learning-Based Image Reconstruction and Enhancement in Optical Microscopy , 2020, Proceedings of the IEEE.

[60]  Vahid Ghodrati,et al.  MR image reconstruction using deep learning: evaluation of network structure and loss functions. , 2019, Quantitative imaging in medicine and surgery.

[61]  Christian Payer,et al.  Sparse-View CT Reconstruction Using Wasserstein GANs , 2018, MLMIR@MICCAI.

[62]  John W. Paisley,et al.  A deep error correction network for compressed sensing MRI , 2018, BMC Biomedical Engineering.

[63]  Jing Cheng,et al.  Deep MRI Reconstruction: Unrolled Optimization Algorithms Meet Neural Networks , 2019, ArXiv.

[64]  Yibo Zhang,et al.  Deep learning-based super-resolution in coherent imaging systems , 2018, Scientific Reports.

[65]  Kshitij Dwivedi,et al.  End-to-End Deep Image Reconstruction From Human Brain Activity , 2018, bioRxiv.

[66]  Quanzheng Li,et al.  Iterative PET Image Reconstruction Using Convolutional Neural Network Representation , 2017, IEEE Transactions on Medical Imaging.

[67]  Taeseong Kim,et al.  KIKI‐net: cross‐domain convolutional neural networks for reconstructing undersampled magnetic resonance images , 2018, Magnetic resonance in medicine.

[68]  Julien Jomier,et al.  Open science - towards reproducible research , 2017, Inf. Serv. Use.

[69]  Andreas Kamilaris,et al.  Deep learning in agriculture: A survey , 2018, Comput. Electron. Agric..

[70]  Zhaolin Chen,et al.  A Deep Learning Framework for Transforming Image Reconstruction Into Pixel Classification , 2019, IEEE Access.

[71]  Jeffrey A. Fessler,et al.  Deep dictionary-transform learning for image reconstruction , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[72]  Steve B. Jiang,et al.  Generalizability issues with deep learning models in medicine and their potential solutions: illustrated with cone-beam computed tomography (CBCT) to computed tomography (CT) image conversion , 2020, Mach. Learn. Sci. Technol..

[73]  YiNan Zhang,et al.  Deep Learning- and Transfer Learning-Based Super Resolution Reconstruction from Single Medical Image , 2017, Journal of healthcare engineering.

[74]  Bruce R. Rosen,et al.  Image reconstruction by domain-transform manifold learning , 2017, Nature.

[75]  Sasa Mutic,et al.  MRI Super Resolution Reconstruction for MRI-guided Adaptive Radiotherapy using Cascaded Deep Learning: In the Presence of Limited Training Data and Unknown Translation Model. , 2019, Medical physics.

[76]  Yuanyuan Wang,et al.  High-resolution image reconstruction for portable ultrasound imaging devices , 2019 .

[77]  Sukhjinder Singh,et al.  MEDICAL IMAGE ENHANCEMENT USING HISTOGRAM PROCESSING TECHNIQUES FOLLOWED BY MEDIAN FILTER , 2012 .

[78]  Rita Strack,et al.  Imaging: AI transforms image reconstruction , 2018, Nature Methods.

[79]  Sebastian Kozerke,et al.  Deep variational network for rapid 4D flow MRI reconstruction , 2020, Nat. Mach. Intell..

[80]  Andreas K. Maier,et al.  PYRO-NN: Python Reconstruction Operators in Neural Networks , 2019, Medical physics.

[81]  Fuyong Xing,et al.  Deep Learning in Microscopy Image Analysis: A Survey. , 2018, IEEE transactions on neural networks and learning systems.

[82]  Hee-Joung Kim,et al.  High quality imaging from sparsely sampled computed tomography data with deep learning and wavelet transform in various domains , 2018, Medical physics.

[83]  Sailesh Conjeti,et al.  Machine Learning for Medical Image Reconstruction , 2018, Lecture Notes in Computer Science.

[84]  Markus Haltmeier,et al.  A U-Nets Cascade for Sparse View Computed Tomography , 2018, MLMIR@MICCAI.

[85]  Nathan Srebro,et al.  Exploring Generalization in Deep Learning , 2017, NIPS.

[86]  Zhanxing Zhu,et al.  SIPID: A deep learning framework for sinogram interpolation and image denoising in low-dose CT reconstruction , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[87]  Ge Wang,et al.  A Perspective on Deep Imaging , 2016, IEEE Access.

[88]  Deniz Kocanaogullari,et al.  Deep Learning For Mri Reconstruction Using A Novel Projection Based Cascaded Network , 2019, 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP).

[89]  Arvid Lundervold,et al.  An overview of deep learning in medical imaging focusing on MRI , 2018, Zeitschrift fur medizinische Physik.

[90]  Jeffrey A. Fessler,et al.  Image Reconstruction is a New Frontier of Machine Learning , 2018, IEEE Transactions on Medical Imaging.

[91]  Thomas J. Fuchs,et al.  DeepPET: A deep encoder–decoder network for directly solving the PET image reconstruction inverse problem , 2018, Medical Image Anal..

[92]  Andrew J. Reader,et al.  Deep Learning for Suppression of Resolution-Recovery Artefacts in MLEM PET Image Reconstruction , 2017, 2017 IEEE Nuclear Science Symposium and Medical Imaging Conference (NSS/MIC).

[93]  Kenji Suzuki,et al.  Overview of deep learning in medical imaging , 2017, Radiological Physics and Technology.

[94]  Markus Löffler,et al.  Deep Learning-based Inaccuracy Compensation in Reconstruction of High Resolution XCT Data , 2020, Scientific Reports.

[95]  Jun Liang,et al.  Deep learning based image reconstruction algorithm for limited-angle translational computed tomography , 2020, PloS one.

[96]  Rickmer Braren,et al.  Secure, privacy-preserving and federated machine learning in medical imaging , 2020, Nature Machine Intelligence.

[97]  Feiyu Chen,et al.  Highly Scalable Image Reconstruction using Deep Neural Networks with Bandpass Filtering , 2018, ArXiv.

[98]  Jean-Luc Starck,et al.  Benchmarking MRI Reconstruction Neural Networks on Large Public Datasets , 2020, Applied Sciences.

[99]  Yibo Zhang,et al.  Phase recovery and holographic image reconstruction using deep learning in neural networks , 2017, Light: Science & Applications.

[100]  G. Klerman,et al.  Facial muscle patterning to affective imagery in depressed and nondepressed subjects , 1976, Science.

[101]  Ge Wang,et al.  Deep learning methods to guide CT image reconstruction and reduce metal artifacts , 2017, Medical Imaging.

[102]  Jean-Philippe Thiran,et al.  Deep Learning Based Ultrasound Image Reconstruction Method: A Time Coherence Study , 2019, 2019 IEEE International Ultrasonics Symposium (IUS).

[103]  Steven Guan,et al.  Limited-View and Sparse Photoacoustic Tomography for Neuroimaging with Deep Learning , 2020, Scientific Reports.

[104]  Muhammad Imran Razzak,et al.  Deep Learning for Medical Image Processing: Overview, Challenges and Future , 2017, ArXiv.

[105]  Thomas Pock,et al.  Learning a variational network for reconstruction of accelerated MRI data , 2017, Magnetic resonance in medicine.

[106]  Hu Chen,et al.  LEARN: Learned Experts’ Assessment-Based Reconstruction Network for Sparse-Data CT , 2017, IEEE Transactions on Medical Imaging.

[107]  Charles A. Bouman,et al.  DEEP BACK PROJECTION FOR SPARSE-VIEW CT RECONSTRUCTION , 2018, 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[108]  Davide Micieli,et al.  Accelerating Neutron Tomography experiments through Artificial Neural Network based reconstruction , 2019, Scientific Reports.