Image Sequence Interpolation Using Optimal Control

The problem of finding an interpolating image between two given images in an image sequence is considered. The problem is formulated as an optimal control problem governed by a transport equation, i.e. we aim at finding a flow field which transports the first image as close as possible to the second image. This approach bears similarities with the Horn and Schunck method for optical flow calculation but in fact the model is quite different. The images are modeled in the space of functions of bounded variation and an analysis of solutions of transport equations in this space is included. Moreover, the existence of optimal controls is proven and necessary conditions are derived. Finally, two algorithms are given and numerical results are compared with existing methods. The new method is competitive with state-of-the-art methods and even outperforms several existing methods.

[1]  Atsushi Imiya,et al.  The William Harvey Code: Mathematical Analysis of Optical Flow Computation for Cardiac Motion , 2006, Human Motion.

[2]  G. Burton.  Sobolev Spaces , 2013 .

[3]  Max D. Gunzburger,et al.  An Optimal Control Formulation of an Image Registration Problem , 2009, Journal of Mathematical Imaging and Vision.

[4]  Michel Fortin,et al.  Mixed and Hybrid Finite Element Methods , 2011, Springer Series in Computational Mathematics.

[5]  Edward H. Adelson,et al.  The Laplacian Pyramid as a Compact Image Code , 1983, IEEE Trans. Commun..

[6]  Gjlles Aubert,et al.  Mathematical problems in image processing , 2001 .

[7]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[8]  Daniel Cremers,et al.  An Improved Algorithm for TV-L 1 Optical Flow , 2009, Statistical and Geometrical Approaches to Visual Motion Analysis.

[9]  C. Schnörr,et al.  Optical Stokes Flow Estimation: An Imaging‐Based Control Approach , 2006 .

[10]  John L. Barron,et al.  Determining Optical Flow for Large Motions Using Parametric Models in a Hierarchical Framework , 2007 .

[11]  Howard C. Elman,et al.  Finite Elements and Fast Iterative Solvers: with Applications in Incompressible Fluid Dynamics , 2014 .

[12]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[13]  J. Weickert,et al.  Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods , 2005 .

[14]  D. Jordan,et al.  Nonlinear Ordinary Differential Equations: An Introduction for Scientists and Engineers , 1979 .

[15]  Marcus A. Magnor,et al.  View and Time Interpolation in Image Space , 2008, Comput. Graph. Forum.

[16]  L. Ambrosio,et al.  Functions of Bounded Variation and Free Discontinuity Problems , 2000 .

[17]  Jean E. Roberts,et al.  Mixed and hybrid finite element methods , 1987 .

[18]  J. Lions.  Optimal Control of Systems Governed by Partial Differential Equations , 1971 .

[19]  L. Ambrosio.  The Flow Associated to Weakly Differentiable Vector Fields: Recent Results and Open Problems , 2011 .

[20]  William H. Press,et al.  Numerical Recipes 3rd Edition: The Art of Scientific Computing , 2007 .

[21]  I. Gavrilyuk.  Book Review: Variational analysis in Sobolev and BV spaces , 2007 .

[22]  Otmar Scherzer,et al.  Models for Image Interpolation Based on the Optical Flow , 2001, Computing.

[23]  Jeffrey Rauch,et al.  Nearly Lipschitzean Divergence Free Transport Propogates neither Continuity nor BV Regularity , 2004 .

[24]  L. Evans.  Measure theory and fine properties of functions , 1992 .

[25]  K. F. Riley,et al.  Mathematical Methods for Physics and Engineering , 1998 .

[26]  C. Hirsch,et al.  Numerical Computation of Internal and External Flows. By C. HIRSCH. Wiley. Vol. 1, Fundamentals of Numerical Discretization. 1988. 515 pp. £60. Vol. 2, Computational Methods for Inviscid and Viscous Flows. 1990, 691 pp. £65. , 1991, Journal of Fluid Mechanics.

[27]  D. Kuzmin,et al.  High-resolution FEM-TVD schemes based on a fully multidimensional flux limiter , 2004 .

[28]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[29]  P. Lions,et al.  Ordinary differential equations, transport theory and Sobolev spaces , 1989 .

[30]  Hans-Hellmut Nagel,et al.  An Investigation of Smoothness Constraints for the Estimation of Displacement Vector Fields from Image Sequences , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[32]  Vivette Girault,et al.  Finite Element Methods for Navier-Stokes Equations - Theory and Algorithms , 1986, Springer Series in Computational Mathematics.

[33]  Kazufumi Ito,et al.  Optimal Control Formulation for Determining Optical Flow , 2002, SIAM J. Sci. Comput..

[34]  Kristian Bredies.  Weak solutions of linear degenerate parabolic equations and an application in image processing , 2009 .

[35]  Zhixun Su,et al.  Learning PDEs for Image Restoration via Optimal Control , 2010, ECCV.

[36]  Barry Lee,et al.  Finite elements and fast iterative solvers: with applications in incompressible fluid dynamics , 2006, Math. Comput..

[37]  Wilfried Enkelmann,et al.  Investigations of multigrid algorithms for the estimation of optical flow fields in image sequences , 1988, Comput. Vis. Graph. Image Process..

[38]  Felix Otto,et al.  Transport Equations and Multi-D Hyperbolic Conservation Laws , 2008 .

[39]  G. Aubert,et al.  A mathematical study of the relaxed optical flow problem in the space BV (&Ω) , 1999 .

[40]  F. Tröltzsch.  Optimale Steuerung partieller Differentialgleichungen , 2005 .

[41]  Ivan P. Gavrilyuk,et al.  Variational analysis in Sobolev and BV spaces , 2007, Math. Comput..