Inpainting Cropped Diffusion MRI Using Deep Generative Models

Minor artifacts introduced during image acquisition are often negligible to the human eye, such as a confined field of view resulting in MRI missing the top of the head. This cropping artifact, however, can cause suboptimal processing of the MRI resulting in data omission or decreasing the power of subsequent analyses. We propose to avoid data or quality loss by restoring these missing regions of the head via variational autoencoders (VAE), a deep generative model that has been previously applied to high resolution image reconstruction. Based on diffusion weighted images (DWI) acquired by the National Consortium on Alcohol and Neurodevelopment in Adolescence (NCANDA), we evaluate the accuracy of inpainting the top of the head by common autoencoder models (U-Net, VQVAE, and VAE-GAN) and a custom model proposed herein called U-VQVAE. Our results show that U-VQVAE not only achieved the highest accuracy, but also resulted in MRI processing producing lower fractional anisotropy (FA) in the supplementary motor area than FA derived from the original MRIs. Lower FA implies that inpainting reduces noise in processing DWI and thus increase the quality of the generated results. The code is available at https://github.com/RdoubleA/DWIinpainting.

[1]  Ichiro Hagiwara,et al.  An approach to surface retouching and mesh smoothing , 2003, The Visual Computer.

[2]  Victor Alves,et al.  A hitchhiker's guide to diffusion tensor imaging , 2012, Front. Neurosci..

[3]  Tony F. Chan,et al.  Nontexture Inpainting by Curvature-Driven Diffusions , 2001, J. Vis. Commun. Image Represent..

[4]  Mahmood Fathy,et al.  AVID: Adversarial Visual Irregularity Detection , 2018, ACCV.

[5]  Ali Razavi,et al.  Generating Diverse High-Fidelity Images with VQ-VAE-2 , 2019, NeurIPS.

[6]  Kilian M. Pohl,et al.  Harmonizing DTI measurements across scanners to examine the development of white matter microstructure in 803 adolescents of the NCANDA study , 2016, NeuroImage.

[7]  Jong Chul Ye,et al.  Annihilating Filter-Based Low-Rank Hankel Matrix Approach for Image Inpainting , 2015, IEEE Transactions on Image Processing.

[8]  Bruce Fischl,et al.  Accurate and robust brain image alignment using boundary-based registration , 2009, NeuroImage.

[9]  Oriol Vinyals,et al.  Neural Discrete Representation Learning , 2017, NIPS.

[10]  D. Le Bihan,et al.  Artifacts and pitfalls in diffusion MRI , 2006, Journal of magnetic resonance imaging : JMRI.

[11]  Minghui Zhang,et al.  Gradient-based low rank method and its application in image inpainting , 2018, Multimedia Tools and Applications.

[12]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Rachael D. Seidler,et al.  A simple solution for model comparison in bold imaging: the special case of reward prediction error and reward outcomes , 2013, Front. Neurosci..

[14]  Enhong Chen,et al.  Image Denoising and Inpainting with Deep Neural Networks , 2012, NIPS.

[15]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[16]  Daniel C. Alexander,et al.  Camino: Open-Source Diffusion-MRI Reconstruction and Processing , 2006 .

[17]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[18]  M. Jorge Cardoso,et al.  Neuromorphologicaly-preserving Volumetric data encoding using VQ-VAE , 2020, ArXiv.

[19]  Yilong Yin,et al.  Patch-Based Image Inpainting via Two-Stage Low Rank Approximation , 2018, IEEE Transactions on Visualization and Computer Graphics.

[20]  Omar ElHarrouss,et al.  Image Inpainting: A Review , 2019, Neural Processing Letters.

[21]  Adolf Pfefferbaum,et al.  The SRI24 multichannel atlas of normal adult human brain structure , 2009, Human brain mapping.

[22]  Ole Winther,et al.  Autoencoding beyond pixels using a learned similarity metric , 2015, ICML.

[23]  Sergios Gatidis,et al.  ipA-MedGAN: Inpainting of Arbitrary Regions in Medical Imaging , 2019, 2020 IEEE International Conference on Image Processing (ICIP).

[24]  Thomas Brox,et al.  Generating Images with Perceptual Similarity Metrics based on Deep Networks , 2016, NIPS.

[25]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Bin Yang,et al.  Adversarial Inpainting of Medical Image Modalities , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[27]  Shiguang Shan,et al.  Shift-Net: Image Inpainting via Deep Feature Rearrangement , 2018, ECCV.

[28]  Steen Moeller,et al.  The Human Connectome Project: A data acquisition perspective , 2012, NeuroImage.

[29]  Robert Oostenveld,et al.  ConnectomeDB—Sharing human brain connectivity data , 2016, NeuroImage.

[30]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[31]  Jerry L Prince,et al.  Effects of signal‐to‐noise ratio on the accuracy and reproducibility of diffusion tensor imaging–derived fractional anisotropy, mean diffusivity, and principal eigenvector measurements at 1.5T , 2007, Journal of magnetic resonance imaging : JMRI.

[32]  Saad Amin,et al.  Medical Image Inpainting with RBF Interpolation Technique , 2016 .

[33]  Sergios Gatidis,et al.  Retrospective Correction Of Rigid And Non-Rigid MR Motion Artifacts Using Gans , 2018, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).