Temporal Interpolation via Motion Field Prediction

Navigated 2D multi-slice dynamic Magnetic Resonance (MR) imaging enables high contrast 4D MR imaging during free breathing and provides in-vivo observations for treatment planning and guidance. Navigator slices are vital for retrospective stacking of 2D data slices in this method. However, they also prolong the acquisition sessions. Temporal interpolation of navigator slices an be used to reduce the number of navigator acquisitions without degrading specificity in stacking. In this work, we propose a convolutional neural network (CNN) based method for temporal interpolation via motion field prediction. The proposed formulation incorporates the prior knowledge that a motion field underlies changes in the image intensities over time. Previous approaches that interpolate directly in the intensity space are prone to produce blurry images or even remove structures in the images. Our method avoids such problems and faithfully preserves the information in the image. Further, an important advantage of our formulation is that it provides an unsupervised estimation of bi-directional motion fields. We show that these motion fields can be used to halve the number of registrations required during 4D reconstruction, thus substantially reducing the reconstruction time.

[1]  Hongdong Li,et al.  Learning Image Matching by Simply Watching Video , 2016, ECCV.

[2]  Philippe C. Cattin,et al.  3D Organ Motion Prediction for MR-Guided High Intensity Focused Ultrasound , 2011, MICCAI.

[3]  Ender Konukoglu,et al.  Temporal Interpolation of Abdominal MRIs Acquired During Free-Breathing , 2017, MICCAI.

[4]  Rae-Hong Park,et al.  Optical Flow Based Frame Interpolation of Ultrasound Images , 2006, ICIAR.

[5]  Marc Niethammer,et al.  Quicksilver: Fast predictive image registration – A deep learning approach , 2017, NeuroImage.

[6]  C Bert,et al.  Motion in radiotherapy: particle therapy , 2011, Physics in medicine and biology.

[7]  M. Yukawa,et al.  SSIM image quality metric for denoised images , 2010 .

[8]  Konstantinos G. Derpanis,et al.  Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness , 2016, ECCV Workshops.

[9]  Ziwei Liu,et al.  Semantic Facial Expression Editing using Autoencoded Flow , 2016, ArXiv.

[10]  Gary E. Christensen,et al.  Consistent image registration , 2001, IEEE Transactions on Medical Imaging.

[11]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[12]  Rae-Hong Park,et al.  Real-time 3D ultrasound fetal image enhancment techniques using motion-compensated frame rate up-conversion , 2003, SPIE Medical Imaging.

[13]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[14]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[15]  Michael J. Black,et al.  Optical Flow Estimation Using a Spatial Pyramid Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Yann LeCun,et al.  Deep multi-scale video prediction beyond mean square error , 2015, ICLR.

[17]  Jan Kautz,et al.  Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  P Boesiger,et al.  4D MR imaging of respiratory organ motion and its variability , 2007, Physics in medicine and biology.

[19]  Nitish Srivastava,et al.  Unsupervised Learning of Video Representations using LSTMs , 2015, ICML.

[20]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[21]  Erik Tryggestad,et al.  Respiration-based sorting of dynamic MRI to derive representative 4D-MRI for radiotherapy planning. , 2013, Medical physics.

[22]  Tobias Gass,et al.  Isotropic Total Variation Regularization of Displacements in Parametric Image Registration , 2017, IEEE Transactions on Medical Imaging.

[23]  Gábor Székely,et al.  Consistency-based rectification of nonrigid registrations , 2015, Journal of medical imaging.

[24]  Harald Becher,et al.  Spatio-temporal (2D+T) non-rigid registration of real-time 3D echocardiography and cardiovascular MR image sequences , 2011, Physics in medicine and biology.

[25]  Yann LeCun,et al.  Learning to Linearize Under Uncertainty , 2015, NIPS.

[26]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[27]  Max A. Viergever,et al.  End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network , 2017, DLMIA/ML-CDS@MICCAI.

[28]  Thomas Brox,et al.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[30]  Rich Caruana,et al.  Multitask Learning , 1997, Machine-mediated learning.

[31]  Feng Liu,et al.  Video Frame Interpolation via Adaptive Separable Convolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[32]  Viorica Patraucean,et al.  Spatio-temporal video autoencoder with differentiable memory , 2015, ArXiv.

[33]  Daniel Rueckert,et al.  Groupwise Simultaneous Manifold Alignment for High-Resolution Dynamic MR Imaging of Respiratory Motion , 2013, IPMI.

[34]  Renjie Liao,et al.  Learning to generate images with perceptual similarity metrics , 2015, 2017 IEEE International Conference on Image Processing (ICIP).