Image Sequence Interpolation Based on Optical Flow, Segmentation, and Optimal Control

When using motion fields to interpolate between two consecutive images in an image sequence, a major problem is to handle occlusions and disclusions properly. However, in most cases, one of both images contains the information that is either discluded or occluded; if the first image contains the information (i.e., the region will be occluded), forward interpolation shall be employed, while for information that is contained in the second image (i.e., the region will be discluded), one should use backward interpolation. Hence, we propose to improve an existing approach for image sequence interpolation by incorporating an automatic segmentation in the process, which decides in which region of the image forward or backward interpolation shall be used. Our approach is a combination of the optimal transport approach to image sequence interpolation and the segmentation by the Chan-Vese approach. We propose to solve the resulting optimality condition by a segregation loop, combined with a level set approach. We provide examples that illustrate the performance both in the interpolation error and in the human perception.

[1]  Daniel Cremers,et al.  An Improved Algorithm for TV-L 1 Optical Flow , 2009, Statistical and Geometrical Approaches to Visual Motion Analysis.

[2]  Dirk A. Lorenz,et al.  Image Sequence Interpolation Using Optimal Control , 2010, Journal of Mathematical Imaging and Vision.

[3]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[4]  Patrick Pérez,et al.  Dense estimation and object-based segmentation of the optical flow with robust techniques , 1998, IEEE Trans. Image Process..

[5]  John L. Barron,et al.  Determining Optical Flow for Large Motions Using Parametric Models in a Hierarchical Framework , 2007 .

[6]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[7]  Milan Sonka,et al.  "Handbook of Medical Imaging, Volume 2. Medical Image Processing and Analysis " , 2000 .

[8]  Wilfried Enkelmann,et al.  Investigations of multigrid algorithms for the estimation of optical flow fields in image sequences , 1988, Comput. Vis. Graph. Image Process..

[9]  Tony F. Chan,et al.  Active contours without edges , 2001, IEEE Trans. Image Process..

[10]  Stephen L. Keeling,et al.  Medical Image Registration and Interpolation by Optical Flow with Maximal Rigidity , 2005, Journal of Mathematical Imaging and Vision.

[11]  Marcus A. Magnor,et al.  View and Time Interpolation in Image Space , 2008, Comput. Graph. Forum.

[12]  Curtis R. Vogel,et al.  Iterative Methods for Total Variation Denoising , 1996, SIAM J. Sci. Comput..

[13]  Hans-Hellmut Nagel,et al.  An Investigation of Smoothness Constraints for the Estimation of Displacement Vector Fields from Image Sequences , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  D. Mumford,et al.  Optimal approximations by piecewise smooth functions and associated variational problems , 1989 .

[15]  K. F. Riley,et al.  Mathematical Methods for Physics and Engineering , 1998 .

[16]  C. Hirsch,et al.  Numerical Computation of Internal and External Flows. By C. HIRSCH. Wiley. Vol. 1, Fundamentals of Numerical Discretization. 1988. 515 pp. £60. Vol. 2, Computational Methods for Inviscid and Viscous Flows. 1990, 691 pp. £65. , 1991, Journal of Fluid Mechanics.

[17]  T. Chan,et al.  On the Convergence of the Lagged Diffusivity Fixed Point Method in Total Variation Image Restoration , 1999 .

[18]  G. Aubert,et al.  A mathematical study of the relaxed optical flow problem in the space BV (&Ω) , 1999 .

[19]  J. E. Glynn,et al.  Numerical Recipes: The Art of Scientific Computing , 1989 .

[20]  V. Caselles,et al.  A geometric model for active contours in image processing , 1993 .

[21]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[22]  William H. Press,et al.  Numerical Recipes 3rd Edition: The Art of Scientific Computing , 2007 .

[23]  Demetri Terzopoulos,et al.  Snakes: Active contour models , 2004, International Journal of Computer Vision.

[24]  Otmar Scherzer,et al.  Models for Image Interpolation Based on the Optical Flow , 2001, Computing.

[25]  Kazufumi Ito,et al.  Optimal Control Formulation for Determining Optical Flow , 2002, SIAM J. Sci. Comput..

[26]  Marcus A. Magnor,et al.  Perception-motivated interpolation of image sequences , 2008, TAP.

[27]  John Watkinson,et al.  MPEG Handbook , 2012 .

[28]  Xue-Cheng Tai,et al.  Image Inpainting Using a TV-Stokes Equation , 2007 .

[29]  Daniel Rueckert,et al.  Nonrigid registration using free-form deformations: application to breast MR images , 1999, IEEE Transactions on Medical Imaging.

[30]  Joachim Weickert,et al.  Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods , 2005, International Journal of Computer Vision.

[31]  Wojciech Matusik,et al.  Moving gradients: a path-based method for plausible image interpolation , 2009, ACM Trans. Graph..