Robust Optical Flow Integration

We analyze the problem of how to correctly construct dense point trajectories from optical flow fields. First, we show that simple Euler integration is unavoidably inaccurate, no matter how good is the optical flow estimator. Then, an inverse integration scheme is analyzed which is more robust to bias and input noise and shows better stability properties. Our contribution is threefold: 1) a theoretical analysis that demonstrates why and in what sense inverse integration is more accurate; 2) a rich experimental validation both on synthetic and real (image) data; and 3) an algorithm for approximate online inverse integration. This new technique is precious whether one is trying to propagate information densely available on a reference frame to the other frames in the sequence or, conversely, to assign information densely over each frame by pulling it from the reference.

[1]  Matematik,et al.  Numerical Methods for Ordinary Differential Equations: Butcher/Numerical Methods , 2005 .

[2]  Carlo Tomasi,et al.  Dense Lagrangian motion estimation with occlusions , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  C. Robinson.  Dynamical Systems: Stability, Symbolic Dynamics, and Chaos , 1994 .

[4]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[5]  Patrick Pérez,et al.  From optical flow to dense long term correspondences , 2012, 2012 19th IEEE International Conference on Image Processing.

[6]  Christophe P. Bernard,et al.  Discrete Wavelet Analysis: A New Framework for Fast Optic Flow Computation , 1998, ECCV.

[7]  Lourdes Agapito,et al.  A Variational Approach to Video Registration with Subspace Constraints , 2013, International Journal of Computer Vision.

[8]  Hua Yang,et al.  Real-Time Optical Flow Estimation Using Multiple Frame-Straddling Intervals , 2012, J. Robotics Mechatronics.

[9]  Thomas Brox,et al.  Universität Des Saarlandes Fachrichtung 6.1 – Mathematik Highly Accurate Optic Flow Computation with Theoretically Justified Warping Highly Accurate Optic Flow Computation with Theoretically Justified Warping , 2022 .

[10]  Carlo Tomasi,et al.  Simultaneous Compaction and Factorization of Sparse Image Motion Matrices , 2012, ECCV.

[11]  Cordelia Schmid,et al.  Dense Trajectories and Motion Boundary Descriptors for Action Recognition , 2013, International Journal of Computer Vision.

[12]  Patrick Pérez,et al.  Dense Estimation of Fluid Flows , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Ivan Laptev,et al.  Track to the future: Spatio-temporal video segmentation with long-range motion cues , 2011, CVPR 2011.

[14]  Kurt Keutzer,et al.  Dense Point Trajectories by GPU-Accelerated Large Displacement Optical Flow , 2010, ECCV.

[15]  Ce Liu,et al.  Towards Longer Long-Range Motion Trajectories , 2012, BMVC.

[16]  Horst Bischof,et al.  A Duality Based Approach for Realtime TV-L1 Optical Flow , 2007, DAGM-Symposium.

[17]  Carlo Tomasi,et al.  Video Motion for Every Visible Point , 2013, 2013 IEEE International Conference on Computer Vision.

[18]  Jitendra Malik,et al.  Object Segmentation by Long Term Analysis of Point Trajectories , 2010, ECCV.

[19]  Cordelia Schmid,et al.  Action recognition by dense trajectories , 2011, CVPR 2011.

[20]  Étienne Mémin,et al.  Conditional filters for image sequence-based tracking - application to point tracking , 2005, IEEE Transactions on Image Processing.

[21]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[22]  Kuo-Chin Fan,et al.  Estimating Optical Flow by Integrating Multi-Frame Information , 2008, J. Inf. Sci. Eng..

[23]  René Vidal,et al.  A Benchmark for the Comparison of 3-D Motion Segmentation Algorithms , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  C. Schmid,et al.  Recognizing activities with cluster-trees of tracklets , 2012, BMVC.

[25]  Bernd Neumann,et al.  Optical flow , 1986, Workshop on Motion.

[26]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Patrick Pérez,et al.  Multi-step flow fusion: towards accurate and dense correspondences in long video shots , 2012, BMVC.

[28]  Seth J. Teller,et al.  Particle Video: Long-Range Motion Estimation Using Point Trajectories , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[29]  Michal Irani,et al.  Multi-frame optical flow estimation using subspace constraints , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[30]  Étienne Mémin,et al.  Stochastic Uncertainty Models for the Luminance Consistency Assumption , 2012, IEEE Transactions on Image Processing.

[31]  Michael J. Black,et al.  A Naturalistic Open Source Movie for Optical Flow Evaluation , 2012, ECCV.

[32]  J. Lambert.  Numerical Methods for Ordinary Differential Equations , 1991 .

[33]  Michael J. Black,et al.  Learning Optical Flow , 2008, ECCV.

[34]  Michael J. Black,et al.  Secrets of optical flow estimation and their principles , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.