Full Flow: Optical Flow Estimation By Global Optimization over Regular Grids

We present a global optimization approach to optical flow estimation. The approach optimizes a classical optical flow objective over the full space of mappings between discrete grids. No descriptor matching is used. The highly regular structure of the space of mappings enables optimizations that reduce the computational complexity of the algorithm's inner loop from quadratic to linear and support efficient matching of tens of thousands of nodes to tens of thousands of displacements. We show that one-shot global optimization of a classical Horn-Schunck-type objective over regular grids at a single resolution is sufficient to initialize continuous interpolation and achieve state-of-the-art performance on challenging modern benchmarks.

[1]  Andreas Geiger,et al.  Object scene flow for autonomous vehicles , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Cordelia Schmid,et al.  EpicFlow: Edge-preserving interpolation of correspondences for optical flow , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Václav Hlavác,et al.  Efficient MRF Deformation Model for Non-Rigid Image Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Daniel P. Huttenlocher,et al.  Efficient Belief Propagation for Early Vision , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[5]  Richard Szeliski,et al.  A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Jiaolong Yang,et al.  Dense, accurate optical flow estimation with piecewise parametric model , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  S. Osher,et al.  Global minimization of Markov random fields with applications to optical flow , 2012 .

[8]  Didier Stricker,et al.  Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[9]  Andrew Blake,et al.  Fusion Moves for Markov Random Field Optimization , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Nassir Navab,et al.  Optical flow estimation with uncertainties through dynamic MRFs , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Daniel Cremers,et al.  Large displacement optical flow computation withoutwarping , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[12]  Jitendra Malik,et al.  Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Alok Aggarwal,et al.  Geometric applications of a matrix-searching algorithm , 1987, SCG '86.

[14]  Camillo J. Taylor,et al.  Optical Flow with Geometric Occlusion Estimation and Fusion of Multiple Frames , 2015, EMMCVPR.

[15]  Cordelia Schmid,et al.  DeepFlow: Large Displacement Optical Flow with Deep Matching , 2013, 2013 IEEE International Conference on Computer Vision.

[16]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[17]  Thomas Pock,et al.  Solving Dense Image Matching in Real-Time using Discrete-Continuous Optimization , 2016, ArXiv.

[18]  David Eppstein,et al.  Sequence Comparison with Mixed Convex and Concave Costs , 1990, J. Algorithms.

[19]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[20]  Michael J. Black,et al.  A Quantitative Analysis of Current Practices in Optical Flow Estimation and the Principles Behind Them , 2013, International Journal of Computer Vision.

[21]  Rob A. Rutenbar,et al.  FPGA acceleration of Markov Random Field TRW-S inference for stereo matching , 2013, 2013 Eleventh ACM/IEEE International Conference on Formal Methods and Models for Codesign (MEMOCODE 2013).

[22]  Christopher Zach,et al.  A Principled Approach for Coarse-to-Fine MAP Inference , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Vladlen Koltun,et al.  Robust Nonrigid Registration by Convex Optimization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[24]  Nassir Navab,et al.  TriangleFlow: Optical Flow with Triangulation-Based Higher-Order Likelihoods , 2010, ECCV.

[25]  Eric Dubois,et al.  Multigrid Bayesian Estimation Of Image Motion Using Stochastic Relaxation , 1988, [1988 Proceedings] Second International Conference on Computer Vision.

[26]  Daniel Cremers,et al.  Tight Convex Relaxations for Vector-Valued Labeling , 2013, SIAM J. Imaging Sci..

[27]  Nikos Komodakis,et al.  MRF Energy Minimization and Beyond via Dual Decomposition , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Christian Heipke,et al.  Discrete Optimization for Optical Flow , 2015, GCPR.

[29]  William H. Warren,et al.  Chapter 8 – Self-Motion: Visual Perception and Visual Control , 1995 .

[30]  Daniel Cremers,et al.  Convex Relaxation of Vectorial Problems with Coupled Regularization , 2014, SIAM J. Imaging Sci..

[31]  Sang Uk Lee,et al.  Optical flow estimation with adaptive convolution kernel prior on discrete framework , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[32]  Martin J. Wainwright,et al.  MAP estimation via agreement on trees: message-passing and linear programming , 2005, IEEE Transactions on Information Theory.

[33]  Michael J. Black,et al.  Optical Flow with Semantic Segmentation and Localized Layers , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Daniel P. Huttenlocher,et al.  Distance Transforms of Sampled Functions , 2012, Theory Comput..

[35]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[36]  Yasuyuki Matsushita,et al.  Motion detail preserving optical flow estimation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[37]  Vladimir Kolmogorov,et al.  Convergent Tree-Reweighted Message Passing for Energy Minimization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Yiannis Aloimonos,et al.  Motion segmentation using occlusions , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Pushmeet Kohli,et al.  Markov Random Fields for Vision and Image Processing , 2011 .

[40]  Patrick Bouthemy,et al.  Multimodal Estimation of Discontinuous Optical Flow using Markov Random Fields , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  Mandyam V Srinivasan,et al.  Honeybees as a model for the study of visually guided flight, navigation, and biologically inspired robotics. , 2011, Physiological reviews.

[42]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[43]  Michael J. Black,et al.  A Naturalistic Open Source Movie for Optical Flow Evaluation , 2012, ECCV.

[44]  Vladlen Koltun,et al.  Fast MRF Optimization with Application to Depth Reconstruction , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Ying Wu,et al.  Large Displacement Optical Flow from Nearest Neighbor Fields , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[47]  Konrad Schindler,et al.  An Evaluation of Data Costs for Optical Flow , 2013, GCPR.