Generative Adversarial Networks (GAN) Powered Fast Magnetic Resonance Imaging - Mini Review, Comparison and Perspectives

Magnetic Resonance Imaging (MRI) is a vital component of medical imaging. When compared to other image modalities, it has advantages such as the absence of radiation, superior soft tissue contrast, and complementary multiple sequence information. However, one drawback of MRI is its comparatively slow scanning and reconstruction compared to other image modalities, limiting its usage in some clinical applications when imaging time is critical. Traditional compressive sensing based MRI (CS-MRI) reconstruction can speed up MRI acquisition, but suffers from a long iterative process and noise-induced artefacts. Recently, Deep Neural Networks (DNNs) have been used in sparse MRI reconstruction models to recreate relatively high-quality images from heavily undersampled k -space data, allowing for much faster MRI scanning. However, there are still some hurdles to tackle. For example, directly training DNNs based on L1/L2 distance to the target fully sampled images could result in blurry reconstruction because L1/L2 loss can only enforce overall image or patch similarity and does not take into account local information such as anatomical sharpness. It is also hard to preserve fine image details while maintaining a natural appearance. More recently, Generative Adversarial Networks (GAN) based methods are proposed to solve fast MRI with enhanced image perceptual quality. The encoder obtains a latent space for the undersampling image, and the image is reconstructed by the decoder using the GAN loss. In this chapter, we review the GAN powered fast MRI methods with a comparative study on various anatomical datasets to demonstrate the generalisability and robustness of this kind of fast MRI while providing future perspectives.

[1]  Yuan Gao,et al.  Weakly Supervised Deep Learning for COVID-19 Infection Detection and Classification From CT Images , 2020, IEEE Access.

[2]  Guang Yang,et al.  DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction , 2018, IEEE Transactions on Medical Imaging.

[3]  Abder-Rahman Ali,et al.  A Novel Fuzzy Multilayer Perceptron (F-MLP) for the Detection of Irregularity in Skin Lesion Border Using Dermoscopic Images , 2020, Frontiers in Medicine.

[4]  Terry M Peters,et al.  Stationary wavelet transform for under-sampled MRI reconstruction. , 2014, Magnetic resonance imaging.

[5]  Di Zhao,et al.  Reference-Driven Compressed Sensing MR Image Reconstruction Using Deep Convolutional Neural Networks without Pre-Training , 2020, Sensors.

[6]  Guang Yang,et al.  Holistic and Deep Feature Pyramids for Saliency Detection , 2018, BMVC.

[7]  J Hennig,et al.  RARE imaging: A fast imaging method for clinical MR , 1986, Magnetic resonance in medicine.

[8]  Guang Yang,et al.  Annealing Genetic GAN for Minority Oversampling , 2020, BMVC.

[9]  Stefano V. Albrecht,et al.  Stabilizing Generative Adversarial Networks: A Survey , 2019, 1910.00927.

[10]  Dong Liang,et al.  Deep Magnetic Resonance Image Reconstruction: Inverse Problems Meet Neural Networks , 2020, IEEE Signal Processing Magazine.

[11]  Zhangming Niu,et al.  Systematic and Comprehensive Automated Ventricle Segmentation on Ventricle Images of the Elderly Patients: A Retrospective Study , 2020, Frontiers in Aging Neuroscience.

[12]  M. Elad,et al.  Compressed Learning for Image Classification: A Deep Neural Network Approach , 2018 .

[13]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[14]  Yankun Cao,et al.  Multiparameter Synchronous Measurement With IVUS Images for Intelligently Diagnosing Coronary Cardiac Disease , 2020, IEEE Transactions on Instrumentation and Measurement.

[15]  Guang Yang,et al.  Stochastic Deep Compressive Sensing for the Reconstruction of Diffusion Tensor Cardiac MRI , 2018, MICCAI.

[16]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Shuo Li,et al.  Synthesis of gadolinium-enhanced liver tumors on nonenhanced liver MR images using pixel-level graph reinforcement learning , 2021, Medical Image Anal..

[18]  Jaejun Yoo,et al.  Deep Residual Learning for Accelerated MRI Using Magnitude and Phase Networks , 2018, IEEE Transactions on Biomedical Engineering.

[19]  Guang Yang,et al.  Deep Learning intra-image and inter-images features for Co-saliency detection , 2018, BMVC.

[20]  et al.,et al.  Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge , 2018, ArXiv.

[21]  Mohana,et al.  Feature Extraction using Convolution Neural Networks (CNN) and Deep Learning , 2018, 2018 3rd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT).

[22]  Kieren Grant Hollingsworth,et al.  Reducing acquisition time in clinical MRI by data undersampling and compressed sensing reconstruction , 2015, Physics in medicine and biology.

[23]  Dimitris N. Metaxas,et al.  MRI Reconstruction Via Cascaded Channel-Wise Attention Network , 2018, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[24]  Heye Zhang,et al.  MV-RAN: Multiview recurrent aggregation network for echocardiographic sequences segmentation and full cardiac cycle analysis , 2020, Comput. Biol. Medicine.

[25]  Guang Yang,et al.  Wavelet improved GAN for MRI reconstruction , 2021 .

[26]  Yinglan Gong,et al.  Accelerating CS-MRI Reconstruction With Fine-Tuning Wasserstein Generative Adversarial Network , 2019, IEEE Access.

[27]  Won-Ki Jeong,et al.  Compressed Sensing MRI Reconstruction Using a Generative Adversarial Network With a Cyclic Loss , 2017, IEEE Transactions on Medical Imaging.

[28]  Nicole Seiberlich,et al.  Parallel MR imaging , 2012, Journal of magnetic resonance imaging : JMRI.

[29]  Qinghao Ye,et al.  Explainable AI for COVID-19 CT Classifiers: An Initial Comparison Study , 2021, 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS).

[30]  Chengyan Wang,et al.  PIC-GAN: A Parallel Imaging Coupled Generative Adversarial Network for Accelerated Multi-Channel MRI Reconstruction , 2021, Diagnostics.

[31]  Guang Yang,et al.  ME‐Net: Multi‐encoder net framework for brain tumor segmentation , 2021, Int. J. Imaging Syst. Technol..

[32]  P. Mansfield Multi-planar image formation using NMR spin echoes , 1977 .

[33]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[34]  A. Haase,et al.  FLASH imaging: rapid NMR imaging using low flip-angle pulses. 1986. , 1986, Journal of magnetic resonance.

[35]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[36]  Zhimin Chen,et al.  Face Super-Resolution Through Wasserstein GANs , 2017, ArXiv.

[37]  Guang Yang,et al.  3D PBV-Net: An automated prostate MRI data segmentation method , 2020, Comput. Biol. Medicine.

[38]  Guang Yang,et al.  Atrial scar quantification via multi-scale CNN in the graph-cuts framework , 2019, Medical Image Anal..

[39]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[40]  Merlin J Fair,et al.  A review of 3D first-pass, whole-heart, myocardial perfusion cardiovascular magnetic resonance , 2015, Journal of Cardiovascular Magnetic Resonance.

[41]  Guang Yang,et al.  Automating in vivo cardiac diffusion tensor postprocessing with deep learning–based segmentation , 2020, Magnetic resonance in medicine.

[42]  Pietro Lio',et al.  How Can We Make Gan Perform Better in Single Medical Image Super-Resolution? A Lesion Focused Multi-Scale Approach , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[43]  Jong Chul Ye,et al.  Unpaired Deep Learning for Accelerated MRI Using Optimal Transport Driven CycleGAN , 2020, IEEE Transactions on Computational Imaging.

[44]  Taeseong Kim,et al.  KIKI‐net: cross‐domain convolutional neural networks for reconstructing undersampled magnetic resonance images , 2018, Magnetic resonance in medicine.

[45]  Guang Yang,et al.  Automatic skin lesion segmentation by coupling deep fully convolutional networks and shallow network with textons , 2019, Journal of medical imaging.

[46]  K. Mouridsen,et al.  Prediction of Tissue Outcome and Assessment of Treatment Effect in Acute Ischemic Stroke Using Deep Learning , 2018, Stroke.

[47]  Morteza Mardani,et al.  Deep Generative Adversarial Neural Networks for Compressive Sensing MRI , 2019, IEEE Transactions on Medical Imaging.

[48]  Guang Yang,et al.  Deep De-Aliasing for Fast Compressive Sensing MRI , 2017, ArXiv.

[49]  Yoram Bresler,et al.  MR Image Reconstruction From Highly Undersampled k-Space Data by Dictionary Learning , 2011, IEEE Transactions on Medical Imaging.

[50]  Qinghao Ye,et al.  Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond , 2021, Inf. Fusion.

[51]  Tammy Riklin-Raviv,et al.  Subsampled brain MRI reconstruction by generative adversarial neural networks , 2020, Medical Image Anal..

[52]  Daniel Rueckert,et al.  A Deep Cascade of Convolutional Neural Networks for Dynamic MR Image Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[53]  Junfeng Yang,et al.  A Fast Alternating Direction Method for TVL1-L2 Signal Reconstruction From Partial Fourier Data , 2010, IEEE Journal of Selected Topics in Signal Processing.

[54]  Ender M. Eksioglu,et al.  Decoupled Algorithm for MRI Reconstruction Using Nonlocal Block Matching Model: BM3D-MRI , 2016, Journal of Mathematical Imaging and Vision.

[55]  Sotirios A. Tsaftaris,et al.  DiCyc: GAN-based deformation invariant cross-domain information fusion for medical image synthesis , 2021, Inf. Fusion.

[56]  Mathews Jacob,et al.  Generalized Higher Degree Total Variation (HDTV) Regularization , 2014, IEEE Transactions on Image Processing.

[57]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[58]  D. Firmin,et al.  Fast and Automated Segmentation for the Three-Directional Multi-Slice Cine Myocardial Velocity Mapping , 2021, Diagnostics.

[59]  Qiusheng Lian,et al.  A deep unrolling network inspired by total variation for compressed sensing MRI , 2020, Digit. Signal Process..

[60]  Mai Xu,et al.  Wavelet Domain Style Transfer for an Effective Perception-Distortion Tradeoff in Single Image Super-Resolution , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[61]  Bakhtyar Ahmed Mohammed,et al.  using Deep Learning , 2020 .

[62]  Guang Yang,et al.  Lesion Focused Super-Resolution , 2018, Medical Imaging: Image Processing.

[63]  Carola-Bibiane Schönlieb,et al.  Machine learning for COVID-19 detection and prognostication using chest radiographs and CT scans: a systematic methodological review , 2020, ArXiv.

[64]  Guang Yang,et al.  Deep Attentive Wasserstein Generative Adversarial Networks for MRI Reconstruction with Recurrent Context-Awareness , 2020, MICCAI.

[65]  Xin Wang,et al.  Salient Object Detection in the Distributed Cloud-Edge Intelligent Network , 2020, IEEE Network.

[66]  Guang Yang,et al.  JAS-GAN: Generative Adversarial Network Based Joint Atrium and Scar Segmentations on Unbalanced Atrial Targets , 2021, IEEE Journal of Biomedical and Health Informatics.

[67]  Jennifer Keegan,et al.  Automated multi-channel segmentation for the 4D myocardial velocity mapping cardiac MR , 2020, Medical Imaging.

[68]  Ruijiang Li,et al.  Predicting treatment response from longitudinal images using multi-task deep learning , 2021, Nature Communications.

[69]  Guang Yang,et al.  Automatic Prostate Zonal Segmentation Using Fully Convolutional Network With Feature Pyramid Attention , 2019, IEEE Access.

[70]  Guang Yang,et al.  The Deep Poincaré Map: A Novel Approach for Left Ventricle Segmentation , 2017, MICCAI.

[71]  Daniel Rueckert,et al.  Convolutional Recurrent Neural Networks for Dynamic MR Image Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[72]  Guang Yang,et al.  SaliencyGAN: Deep Learning Semisupervised Salient Object Detection in the Fog of IoT , 2020, IEEE Transactions on Industrial Informatics.

[73]  Albert C. S. Chung,et al.  Fast Symmetric Diffeomorphic Image Registration with Convolutional Neural Networks , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[74]  Bin Dong,et al.  A Review on Deep Learning in Medical Image Reconstruction , 2019, Journal of the Operations Research Society of China.

[75]  Guang Yang,et al.  Which GAN? A comparative study of generative adversarial network-based fast MRI reconstruction , 2021, Philosophical Transactions of the Royal Society A.

[76]  Heye Zhang,et al.  Simultaneous left atrium anatomy and scar segmentations via deep learning in multiview information with attention , 2020, Future Gener. Comput. Syst..

[77]  Guang Yang,et al.  Adversarial and Perceptual Refinement for Compressed Sensing MRI Reconstruction , 2018, MICCAI.

[78]  Tinsu Pan,et al.  Fundamentals of Medical Imaging , 2010, The Journal of Nuclear Medicine.

[79]  Guang Yang,et al.  MRI Brain Tumor Segmentation and Patient Survival Prediction Using Random Forests and Fully Convolutional Networks , 2017, BrainLes@MICCAI.

[80]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[81]  Guang Yang,et al.  Automatic Brain Tumor Detection and Segmentation Using U-Net Based Fully Convolutional Networks , 2017, MIUA.

[82]  Zhong Chen,et al.  Balanced Sparse Model for Tight Frames in Compressed Sensing Magnetic Resonance Imaging , 2015, PloS one.

[83]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[84]  Dinggang Shen,et al.  Scalable High-Performance Image Registration Framework by Unsupervised Deep Feature Representations Learning , 2016, IEEE Transactions on Biomedical Engineering.

[85]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[86]  Max A. Viergever,et al.  End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network , 2017, DLMIA/ML-CDS@MICCAI.

[87]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[88]  Nan Feng Zhang,et al.  Deep Learning for Diagnosis of Chronic Myocardial Infarction on Nonenhanced Cardiac Cine MRI. , 2019, Radiology.

[89]  D. Donoho,et al.  Sparse MRI: The application of compressed sensing for rapid MR imaging , 2007, Magnetic resonance in medicine.

[90]  Thomas Pock,et al.  Learning a variational network for reconstruction of accelerated MRI data , 2017, Magnetic resonance in medicine.

[91]  G. Slabaugh,et al.  Left Atrial Scarring Segmentation from Delayed-Enhancement Cardiac MRI Images: A Deep Learning Approach , 2018, Cardiovascular Imaging and Image Analysis.

[92]  Guang Yang,et al.  Evaluation of algorithms for Multi-Modality Whole Heart Segmentation: An open-access grand challenge , 2019, Medical Image Anal..

[93]  Guang Yang,et al.  Exploring Uncertainty Measures in Bayesian Deep Attentive Neural Networks for Prostate Zonal Segmentation , 2020, IEEE Access.

[94]  Bo Wei,et al.  SARA-GAN: Self-Attention and Relative Average Discriminator Based Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction , 2020, Frontiers in Neuroinformatics.

[95]  Francesco Renna,et al.  On instabilities of deep learning in image reconstruction and the potential costs of AI , 2019, Proceedings of the National Academy of Sciences.

[96]  Jing Cheng,et al.  Deep MRI Reconstruction: Unrolled Optimization Algorithms Meet Neural Networks , 2019, ArXiv.

[97]  Zhi Liu,et al.  Deep RetinaNet for Dynamic Left Ventricle Detection in Multiview Echocardiography Classification , 2020, Sci. Program..