An image interpolation approach for acquisition time reduction in navigator‐based 4D MRI

&NA; Navigated 2D multi‐slice dynamic Magnetic Resonance (MR) imaging enables high contrast 4D MR imaging during free breathing and provides in‐vivo observations for treatment planning and guidance. Navigator slices are vital for retrospective stacking of 2D data slices in this method. However, they also prolong the acquisition sessions. Temporal interpolation of navigator slices can be used to reduce the number of navigator acquisitions without degrading specificity in stacking. In this work, we propose a convolutional neural network (CNN) based method for temporal interpolation, with motion field prediction as an intermediate step. The proposed formulation incorporates the prior knowledge that a motion field underlies changes in the image intensities over time. Previous approaches that interpolate directly in the intensity space are prone to produce blurry images or even remove structures in the images. Our method avoids such problems and faithfully preserves the information in the image. Further, an important advantage of our formulation is that it provides an unsupervised estimation of bi‐directional motion fields. These motion fields can potentially be used to halve the number of registrations required during 4D reconstruction, thus substantially reducing the reconstruction time. These advantages are achieved while preserving 4D reconstruction quality as compared to that with the true navigators.

[1]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[2]  Harald Becher,et al.  Spatio-temporal (2D+T) non-rigid registration of real-time 3D echocardiography and cardiovascular MR image sequences , 2011, Physics in medicine and biology.

[3]  Gábor Székely,et al.  Influence of inter-subject correspondences on liver motion predictions from population models , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[4]  Fang-Fang Yin,et al.  Four-dimensional magnetic resonance imaging (4D-MRI) using image-based respiratory surrogate: a feasibility study. , 2011, Medical physics.

[5]  Hongdong Li,et al.  Learning Image Matching by Simply Watching Video , 2016, ECCV.

[6]  Tobias Gass,et al.  Isotropic Total Variation Regularization of Displacements in Parametric Image Registration , 2017, IEEE Transactions on Medical Imaging.

[7]  Ender Konukoglu,et al.  Temporal Interpolation of Abdominal MRIs Acquired During Free-Breathing , 2017, MICCAI.

[8]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[9]  Philippe C. Cattin,et al.  3D Organ Motion Prediction for MR-Guided High Intensity Focused Ultrasound , 2011, MICCAI.

[10]  Rae-Hong Park,et al.  Optical Flow Based Frame Interpolation of Ultrasound Images , 2006, ICIAR.

[11]  C Bert,et al.  Motion in radiotherapy: particle therapy , 2011, Physics in medicine and biology.

[12]  Xiaoou Tang,et al.  Video Frame Synthesis Using Deep Voxel Flow , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[13]  Daniel Rueckert,et al.  VTK CISG Registration Toolkit: An Open Source Software Package for Affine and Nonrigid Registration of Single- and Multimodal 3D images , 2002, Bildverarbeitung für die Medizin.

[14]  Gábor Székely,et al.  Consistency-based rectification of nonrigid registrations , 2015, Journal of medical imaging.

[15]  M. Yukawa,et al.  SSIM image quality metric for denoised images , 2010 .

[16]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[17]  K. Camphausen,et al.  Advances in 4D Medical Imaging and 4D Radiation Therapy , 2008, Technology in cancer research & treatment.

[18]  Gábor Székely,et al.  In vivo validation of spatio-temporal liver motion prediction from motion tracked on MR thermometry images , 2016, International Journal of Computer Assisted Radiology and Surgery.

[19]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[20]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[21]  Max A. Viergever,et al.  End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network , 2017, DLMIA/ML-CDS@MICCAI.

[22]  P Boesiger,et al.  4D MR imaging of respiratory organ motion and its variability , 2007, Physics in medicine and biology.

[23]  J. Lambert,et al.  Intrafractional motion during proton beam scanning , 2005, Physics in medicine and biology.

[24]  Thomas Brox,et al.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  G. Christensen,et al.  A method for the reconstruction of four-dimensional synchronized CT scans acquired during free breathing. , 2003, Medical physics.

[26]  Nitish Srivastava,et al.  Unsupervised Learning of Video Representations using LSTMs , 2015, ICML.

[27]  Gary E. Christensen,et al.  Consistent image registration , 2001, IEEE Transactions on Medical Imaging.

[28]  Konstantinos G. Derpanis,et al.  Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness , 2016, ECCV Workshops.

[29]  Marc Niethammer,et al.  Quicksilver: Fast predictive image registration – A deep learning approach , 2017, NeuroImage.

[30]  Viorica Patraucean,et al.  Spatio-temporal video autoencoder with differentiable memory , 2015, ArXiv.

[31]  Ender Konukoglu,et al.  Temporal Interpolation via Motion Field Prediction , 2018, ArXiv.

[32]  Feng Liu,et al.  Video Frame Interpolation via Adaptive Separable Convolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[33]  Rae-Hong Park,et al.  Real-time 3D ultrasound fetal image enhancment techniques using motion-compensated frame rate up-conversion , 2003, SPIE Medical Imaging.

[34]  Daniel Rueckert,et al.  Groupwise Simultaneous Manifold Alignment for High-Resolution Dynamic MR Imaging of Respiratory Motion , 2013, IPMI.

[35]  Renjie Liao,et al.  Learning to generate images with perceptual similarity metrics , 2015, 2017 IEEE International Conference on Image Processing (ICIP).

[36]  Erik Tryggestad,et al.  Respiration-based sorting of dynamic MRI to derive representative 4D-MRI for radiotherapy planning. , 2013, Medical physics.

[37]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[38]  Yann LeCun,et al.  Deep multi-scale video prediction beyond mean square error , 2015, ICLR.

[39]  Jan Kautz,et al.  Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[40]  Yann LeCun,et al.  Learning to Linearize Under Uncertainty , 2015, NIPS.

[41]  Ziwei Liu,et al.  Semantic Facial Expression Editing using Autoencoded Flow , 2016, ArXiv.

[42]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[43]  Jitendra Malik,et al.  View Synthesis by Appearance Flow , 2016, ECCV.

[44]  T. Guerrero,et al.  Acquiring 4D thoracic CT scans using a multislice helical method. , 2004, Physics in medicine and biology.

[45]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[46]  Michael J. Black,et al.  Optical Flow Estimation Using a Spatial Pyramid Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).