Wasserstein GANs for MR Imaging: From Paired to Unpaired Training

Lack of ground-truth MR images impedes the common supervised training of neural networks for image reconstruction. To cope with this challenge, this article leverages unpaired adversarial training for reconstruction networks, where the inputs are undersampled k-space and naively reconstructed images from one dataset, and the labels are high-quality images from another dataset. The reconstruction networks consist of a generator which suppresses the input image artifacts, and a discriminator using a pool of (unpaired) labels to adjust the reconstruction quality. The generator is an unrolled neural network – a cascade of convolutional and data consistency layers. The discriminator is also a multilayer CNN that plays the role of a critic scoring the quality of reconstructed images based on the Wasserstein distance. Our experiments with knee MRI datasets demonstrate that the proposed unpaired training enables diagnostic-quality reconstruction when high-quality image labels are not available for the input types of interest, or when the amount of labels is small. In addition, our adversarial training scheme can achieve better image quality (as rated by expert radiologists) compared with the paired training schemes with pixel-wise loss.

[1]  Lucas Theis,et al.  Amortised MAP Inference for Image Super-resolution , 2016, ICLR.

[2]  Zongben Xu,et al.  ADMM-CSNet: A Deep Learning Approach for Image Compressive Sensing , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Jaakko Lehtinen,et al.  Noise2Noise: Learning Image Restoration without Clean Data , 2018, ICML.

[4]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[5]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Stephen P. Boyd,et al.  Dirty Pixels: Optimizing Image Classification Architectures for Raw Sensor Data , 2017, ArXiv.

[7]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[8]  Guang Yang,et al.  DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction , 2018, IEEE Transactions on Medical Imaging.

[9]  Morteza Mardani,et al.  Neural Proximal Gradient Descent for Compressive Imaging , 2018, NeurIPS.

[10]  Tolga Çukur,et al.  A Transfer‐Learning Approach for Accelerated MRI Using Deep Neural Networks , 2017, Magnetic resonance in medicine.

[11]  Andrew L. Maas.  Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[12]  Cédric Villani,et al.  The Wasserstein distances , 2009 .

[13]  Morteza Mardani,et al.  Compressed Sensing: From Research to Clinical Practice with Data-Driven Learning , 2019, ArXiv.

[14]  John M Pauly,et al.  Data‐driven self‐calibration and reconstruction for non‐cartesian wave‐encoded single‐shot fast spin echo using deep learning , 2020, Journal of magnetic resonance imaging : JMRI.

[15]  Michael Elad,et al.  ESPIRiT—an eigenvalue approach to autocalibrating parallel MRI: Where SENSE meets GRAPPA , 2014, Magnetic resonance in medicine.

[16]  Mathews Jacob,et al.  Multi-Shot Sensitivity-Encoded Diffusion MRI Using Model-Based Deep Learning (Modl-Mussels) , 2018, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[17]  Won-Ki Jeong,et al.  Compressed Sensing MRI Reconstruction Using a Generative Adversarial Network With a Cyclic Loss , 2017, IEEE Transactions on Medical Imaging.

[18]  Jin Keun Seo,et al.  Deep learning for undersampled MRI reconstruction , 2017, Physics in medicine and biology.

[19]  Yanjun Li,et al.  Image Recovery via Transform Learning and Low-Rank Modeling: The Power of Complementary Regularizers , 2020, IEEE Transactions on Image Processing.

[20]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Cédric Villani.  Cyclical monotonicity and Kantorovich duality , 2009 .

[22]  M. Lustig,et al.  SPIRiT: Iterative self‐consistent parallel imaging reconstruction from arbitrary k‐space , 2010, Magnetic resonance in medicine.

[23]  Steen Moeller,et al.  Deep-Learning Methods for Parallel Magnetic Resonance Imaging Reconstruction: A Survey of the Current Approaches, Trends, and Issues , 2019, IEEE Signal Processing Magazine.

[24]  Jeffrey A. Fessler,et al.  Deep BCD-Net Using Identical Encoding-Decoding CNN Structures for Iterative Image Recovery , 2018, 2018 IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP).

[25]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[26]  Daniel Rueckert,et al.  A Deep Cascade of Convolutional Neural Networks for Dynamic MR Image Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[27]  Ender Konukoglu,et al.  MR image reconstruction using the learned data distribution as prior , 2017, ArXiv.

[28]  Verónica Vilaplana,et al.  Brain MRI super-resolution using 3D generative adversarial networks , 2018, ArXiv.

[29]  Jonathan I. Tamir,et al.  Variable-Density Single-Shot Fast Spin-Echo MRI with Deep Learning Reconstruction by Using Variational Networks. , 2018, Radiology.

[30]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[31]  Feiyu Chen,et al.  Highly Scalable Image Reconstruction using Deep Neural Networks with Bandpass Filtering , 2018, ArXiv.

[32]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[33]  Guido Montúfar,et al.  How Well Do WGANs Estimate the Wasserstein Metric? , 2019, ArXiv.

[34]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[35]  Daoqiang Zhang,et al.  SEGAN: Structure-Enhanced Generative Adversarial Network for Compressed Sensing MRI Reconstruction , 2019, AAAI.

[36]  Jonas Adler,et al.  Banach Wasserstein GAN , 2018, NeurIPS.

[37]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[38]  Bruce R. Rosen,et al.  Image reconstruction by domain-transform manifold learning , 2017, Nature.

[39]  Liu Yang,et al.  Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations , 2018, SIAM J. Sci. Comput..

[40]  Michael Unser,et al.  Self-Supervised Deep Active Accelerated MRI , 2019, ArXiv.

[41]  Morteza Mardani,et al.  Deep Generative Adversarial Neural Networks for Compressive Sensing MRI , 2019, IEEE Transactions on Medical Imaging.

[42]  Daniel Rueckert,et al.  Convolutional Recurrent Neural Networks for Dynamic MR Image Reconstruction , 2017, IEEE Transactions on Medical Imaging.

[43]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[44]  Jeffrey A. Fessler,et al.  Efficient Sum of Outer Products Dictionary Learning (SOUP-DIL) and Its Application to Inverse Problems , 2015, IEEE Transactions on Computational Imaging.