Motion Inpainting by an Image-Based Geodesic AMLE Method

This work presents an automatic method for optical flow inpainting. Given a video, each frame domain is endowed with a Riemannian metric based on the video pixel values. The missing optical flow is recovered by solving the Absolutely Minimizing Lipschitz Extension (AMLE) partial differential equation on the Riemannian manifold. An efficient numerical algorithm is proposed using eikonal operators for nonlinear elliptic partial differential equations on a finite graph. The choice of the metric is discussed and the method is applied to optical flow inpainting and sparse-to-dense optical flow estimation, achieving top-tier performance in terms of End-Point-Error (EPE).

[1]  Cristian Sminchisescu,et al.  Locally Affine Sparse-to-Dense Matching for Motion and Occlusion Estimation , 2013, 2013 IEEE International Conference on Computer Vision.

[2]  J. Morel,et al.  An axiomatic approach to image interpolation. , 1998, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[3]  Adam M. Oberman,et al.  Nonlinear elliptic Partial Differential Equations and p-harmonic functions on graphs , 2012, 1212.0834.

[4]  Cordelia Schmid,et al.  EpicFlow: Edge-preserving interpolation of correspondences for optical flow , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Alexandru Telea,et al.  An Image Inpainting Technique Based on the Fast Marching Method , 2004, J. Graphics, GPU, & Game Tools.

[6]  Yann Gousseau,et al.  Interpolation of digital elevation models using AMLE and related methods , 2002, IEEE Trans. Geosci. Remote. Sens..

[7]  Gloria Haro,et al.  A Rotation-Invariant Regularization Term for Optical Flow Related Problems , 2014, ACCV.

[8]  Cordelia Schmid,et al.  DeepMatching: Hierarchical Deformable Dense Matching , 2015, International Journal of Computer Vision.

[9]  Laura Igual,et al.  An axiomatic approach to scalar data interpolation on surfaces , 2006, Numerische Mathematik.

[10]  G. Aronsson.  Extension of functions satisfying lipschitz conditions , 1967 .

[11]  Andreas Geiger,et al.  Object scene flow for autonomous vehicles , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Benjamin Berkels,et al.  Reconstructing Optical Flow Fields by Motion Inpainting , 2009, EMMCVPR.

[13]  Harry Shum,et al.  Full-frame video stabilization with motion inpainting , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[15]  Gabriele Facciolo,et al.  Linear Multiscale Analysis of Similarities between Images on Riemannian Manifolds: Practical Formula and Affine Covariant Metrics , 2015, SIAM J. Imaging Sci..

[16]  Michael J. Black,et al.  A Naturalistic Open Source Movie for Optical Flow Evaluation , 2012, ECCV.

[17]  Adam M. Oberman.  A convergent difference scheme for the infinity Laplacian: construction of absolutely minimizing Lipschitz extensions , 2004, Math. Comput..

[18]  Daniel Kondermann,et al.  Postprocessing of Optical Flows Via Surface Measures and Motion Inpainting , 2008, DAGM-Symposium.

[19]  R. Jensen.  Uniqueness of Lipschitz extensions: Minimizing the sup norm of the gradient , 1993 .

[20]  Gunnar Aronsson,et al.  On the partial differential equationux2uxx+2uxuyuxy+uy2uyy=0 , 1968 .

[21]  M. Crandall,et al.  A TOUR OF THE THEORY OF ABSOLUTELY MINIMIZING FUNCTIONS , 2004 .

[22]  Marcelo Bertalmío,et al.  Axiomatic scalar data interpolation on manifolds , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[23]  Vanel Lazcano.  Some problems in depth enjanced video processing , 2016 .

[24]  Daniel Cremers,et al.  Flow and Color Inpainting for Video Completion , 2014, GCPR.

[25]  Janusz Konrad,et al.  Occlusion-Aware Optical Flow Estimation , 2008, IEEE Transactions on Image Processing.

[26]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .