Reliable Estimation of Dense Optical Flow Fields with Large Displacements

In this paper we show that a classic optical flow technique by Nagel and Enkelmann (1986, IEEE Trans. Pattern Anal. Mach. Intell., Vol. 8, pp. 565–593) can be regarded as an early anisotropic diffusion method with a diffusion tensor. We introduce three improvements into the model formulation that (i) avoid inconsistencies caused by centering the brightness term and the smoothness term in different images, (ii) use a linear scale-space focusing strategy from coarse to fine scales for avoiding convergence to physically irrelevant local minima, and (iii) create an energy functional that is invariant under linear brightness changes. Applying a gradient descent method to the resulting energy functional leads to a system of diffusion–reaction equations. We prove that this system has a unique solution under realistic assumptions on the initial data, and we present an efficient linear implicit numerical scheme in detail. Our method creates flow fields with 100 % density over the entire image domain, it is robust under a large range of parameter variations, and it can recover displacement fields that are far beyond the typical one-pixel limits which are characteristic for many differential methods for determining optical flow. We show that it performs better than the optical flow methods with 100 % density that are evaluated by Barron et al. (1994, Int. J. Comput. Vision, Vol. 12, pp. 43–47). Our software is available from the Internet.

[1]  B. Ripley,et al.  Pattern Recognition , 1968, Nature.

[2]  Shun-ichi Amari,et al.  A Theory of Pattern Recognition , 1968 .

[3]  H. Brezis.  Opérateurs maximaux monotones et semi-groupes de contractions dans les espaces de Hilbert , 1973 .

[4]  J. L. Webb.  OPERATEURS MAXIMAUX MONOTONES ET SEMI‐GROUPES DE CONTRACTIONS DANS LES ESPACES DE HILBERT , 1974 .

[5]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[6]  J. Galayda.  Edge Focusing , 1981, IEEE Transactions on Nuclear Science.

[7]  Hans-Hellmut Nagel,et al.  An Investigation of Smoothness Constraints for the Estimation of Displacement Vector Fields from Image Sequences , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Andrew Blake,et al.  Visual Reconstruction , 1987, Deep Learning for EEG-Based Brain–Computer Interfaces.

[9]  Hans-Hellmut Nagel,et al.  On the Estimation of Optical Flow: Relations between Different Approaches and Some New Results , 1987, Artif. Intell..

[10]  M. Bertero,et al.  Ill-posed problems in early vision , 1988, Proc. IEEE.

[11]  Wilfried Enkelmann,et al.  Investigations of multigrid algorithms for the estimation of optical flow fields in image sequences , 1988, Comput. Vis. Graph. Image Process..

[12]  Hans-Hellmut Nagel,et al.  Extending the 'Oriented Smoothness Constraint' into the Temporal Domain and the Estimation of Derivatives of Optical Flow , 1990, ECCV.

[13]  M. A. Snyder.  On the Mathematical Foundations of Smoothness Constraints for the Determination of Optical Flow and for Surface Reconstruction , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Michael J. Black,et al.  Robust dynamic motion estimation over time , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[15]  P. Lions,et al.  Image selective smoothing and edge detection by nonlinear diffusion. II , 1992 .

[16]  P. Lions,et al.  Axioms and fundamental equations of image processing , 1993 .

[17]  Patrick Bouthemy,et al.  Multimodal Estimation of Discontinuous Optical Flow using Markov Random Fields , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Paolo Nesi,et al.  Variational approach to optical flow estimation managing discontinuities , 1993, Image Vis. Comput..

[19]  Jayant Shah.  A nonlinear diffusion model for discontinuous disparity and half-occlusions in stereo , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[20]  L. Blanc-Féraud,et al.  Motion estimation involving discontinuities in multiresolution scheme , 1993 .

[21]  Christoph Schnörr.  On Functionals with Greyvalue-Controlled Smoothness Terms for Determining Optical Flow , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Luc Van Gool,et al.  Determination of Optical Flow and its Discontinuities using Non-Linear Diffusion , 1994, ECCV.

[23]  Joachim Weickert,et al.  Theoretical Foundations of Anisotropic Diffusion in Image Processing , 1994, Theoretical Foundations of Computer Vision.

[24]  Christoph Schnörr,et al.  Segmentation of visual motion by minimizing convex non-quadratic functionals , 1994, ICPR.

[25]  Luc Van Gool,et al.  Coupled Geometry-Driven Diffusion Equations for Low-Level Vision , 1994, Geometry-Driven Diffusion in Computer Vision.

[26]  Rachid Deriche,et al.  Optical-Flow Estimation while Preserving Its Discontinuities: A Variational Approach , 1995, ACCV.

[27]  Hans-Hellmut Nagel,et al.  Estimation of Optical Flow Based on Higher-Order Spatiotemporal Derivatives in Interlaced and Non-Interlaced Image Sequences , 1995, Artif. Intell..

[28]  Gary J. Balas,et al.  Optical flow: a curve evolution approach , 1995, Proceedings., International Conference on Image Processing.

[29]  Gary J. Balas,et al.  Optical flow: a curve evolution approach , 1996, IEEE Trans. Image Process..

[30]  Frédéric Guichard,et al.  Accurate estimation of discontinuous optical flow by minimizing divergence related functionals , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[31]  Joachim Weickert,et al.  Anisotropic diffusion in image processing , 1996 .

[32]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[33]  Luc Florack,et al.  Image Structure , 1997, Computational Imaging and Vision.

[34]  Tony Lindeberg,et al.  Scale-Space Theory in Computer Vision , 1993, Lecture Notes in Computer Science.

[35]  Peter Johansen,et al.  Gaussian Scale-Space Theory , 1997, Computational Imaging and Vision.

[36]  Joachim Weickert,et al.  On Discontinuity-Preserving Optic Flow , 1998 .

[37]  Bernd Jähne,et al.  Performance Characteristics of Low-level Motion Estimators in Spatiotemporal Images , 1998, Theoretical Foundations of Computer Vision.

[38]  Brendan McCane,et al.  Recovering Motion Fields: An Evaluation of Eight Optical Flow Algorithms , 1998, BMVC.

[39]  Patrick Pérez,et al.  Dense estimation and object-based segmentation of the optical flow with robust techniques , 1998, IEEE Trans. Image Process..

[40]  Joachim Weickert,et al.  A Scale-Space Approach to Nonlocal Optical Flow Calculations , 1999, Scale-Space.

[41]  Joachim Weickert,et al.  Scale-Space Theories in Computer Vision , 1999, Lecture Notes in Computer Science.

[42]  Rachid Deriche,et al.  Computing Optical Flow via Variational Techniques , 1999, SIAM J. Appl. Math..

[43]  Joachim Weickert,et al.  Räumlich-zeitliche Berechnung des optischen Flusses mit nichtlinearen flußabhängigen Glattheitstermen , 1999, DAGM-Symposium.

[44]  Julio Esclarín Monreal,et al.  A PDE model for computing the optimal flow , 1999 .

[45]  Javier Sánchez,et al.  3-D geometry reconstruction using a color image stereo pair and partial differential equations , 2000 .

[46]  J. Koenderink.  The structure of images , 2004, Biological Cybernetics.

[47]  Luc Florack,et al.  The Intrinsic Structure of Optic Flow Incorporating Measurement Duality , 1998, International Journal of Computer Vision.

[48]  Patrick Bouthemy,et al.  Computation and analysis of image motion: A synopsis of current problems and methods , 1996, International Journal of Computer Vision.

[49]  P. Anandan,et al.  A computational framework and an algorithm for the measurement of visual motion , 1987, International Journal of Computer Vision.

[50]  Christoph Schnörr,et al.  Determining optical flow for irregular domains by minimizing quadratic functionals of a certain class , 1991, International Journal of Computer Vision.

[51]  Atsushi Imiya,et al.  Linear Scale-Space has First been Proposed in Japan , 1999, Journal of Mathematical Imaging and Vision.

[52]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[53]  Willard L. Miranker.  Numerical methods of boundary layer type for stiff systems of differential equations , 2005, Computing.

[54]  I. Cohen.  Nonlinear Variational Method for Optical Flow Computation , 2006 .