Scene Flow Estimation: A Survey

This paper is the first to review the scene flow estimation field, which analyzes and compares methods, technical challenges, evaluation methodologies and performance of scene flow estimation. Existing algorithms are categorized in terms of scene representation, data source, and calculation scheme, and the pros and cons in each category are compared briefly. The datasets and evaluation protocols are enumerated, and the performance of the most representative methods is presented. A future vision is illustrated with few questions arisen for discussion. This survey presents a general introduction and analysis of scene flow estimation.

[1]  Edmond Boyer,et al.  Scene Flow from Depth and Color Images , 2011, BMVC.

[2]  Vincent Lepetit,et al.  DAISY: An Efficient Dense Descriptor Applied to Wide-Baseline Stereo , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[4]  Edward H. Adelson,et al.  Single Lens Stereo with a Plenoptic Camera , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Frederic Devernay,et al.  Multi-Camera Scene Flow by Tracking 3-D Points and Surfels , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[6]  Cristian Sminchisescu,et al.  Large Displacement 3D Scene Flow with Occlusion Reasoning , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[7]  R. Venkatesh Babu,et al.  Optical flow estimation using Approximate Nearest Neighbor Field fusion , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[8]  Ye Zhang,et al.  On 3-D scene flow and structure recovery from multiview image sequences , 2003, IEEE Trans. Syst. Man Cybern. Part B.

[9]  Cordelia Schmid,et al.  Human Detection Using Oriented Histograms of Flow and Appearance , 2006, ECCV.

[10]  Li Xu,et al.  Consistent Binocular Depth and Scene Flow with Chained Temporal Profiles , 2013, International Journal of Computer Vision.

[11]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[12]  Konrad Schindler,et al.  View-Consistent 3D Scene Flow Estimation over Multiple Frames , 2014, ECCV.

[13]  Roland Wilson,et al.  Estimating scene flow using an interconnected patch surface model with belief-propagation inference , 2014, Comput. Vis. Image Underst..

[14]  Cordelia Schmid,et al.  Learning realistic human actions from movies , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Andrés Bruhn,et al.  Adaptive Integration of Feature Matches into Variational Optical Flow Methods , 2012, ACCV.

[16]  Fang Liu,et al.  Disparity Estimation in Stereo Sequences using Scene Flow , 2009, BMVC.

[17]  Raquel Urtasun,et al.  Efficient Deep Learning for Stereo Matching , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[19]  Olivier D. Faugeras,et al.  Multi-View Stereo Reconstruction and Scene Flow Estimation with a Global Image-Based Matching Score , 2007, International Journal of Computer Vision.

[20]  T. Vaudrey,et al.  Differences between stereo and motion behaviour on synthetic and real-world stereo sequences , 2008, 2008 23rd International Conference Image and Vision Computing New Zealand.

[21]  Martin Jägersand,et al.  Basis constrained 3D scene flow on a dynamic proxy , 2011, 2011 International Conference on Computer Vision.

[22]  Richard Bowden,et al.  Scene Particles: Unregularized Particle-Based Scene Flow Estimation , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Karl Kunisch,et al.  Total Generalized Variation , 2010, SIAM J. Imaging Sci..

[24]  Radu Horaud,et al.  Scene flow estimation by growing correspondence seeds , 2011, CVPR 2011.

[25]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[26]  Hongyang Chao,et al.  MeshStereo: A Global Stereo Model with Mesh Alignment Regularization for View Interpolation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[27]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[28]  Richard Bowden,et al.  Scene Flow Estimation using Intelligent Cost Functions , 2014, BMVC.

[29]  Frank Dellaert,et al.  A Continuous Optimization Approach for Efficient and Accurate Scene Flow , 2016, ECCV.

[30]  Ye Zhang,et al.  Integrated 3D scene flow and structure recovery from multiview image sequences , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[31]  Richard Bowden,et al.  Kinecting the dots: Particle based scene flow from depth sensors , 2011, 2011 International Conference on Computer Vision.

[32]  Anita Sellent,et al.  GraphFlow - 6D Large Displacement Scene Flow via Graph Matching , 2015, GCPR.

[33]  Ye Zhang,et al.  On 3D scene flow and structure estimation , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[34]  Konrad Schindler,et al.  3D Scene Flow Estimation with a Piecewise Rigid Scene Model , 2015, International Journal of Computer Vision.

[35]  Ravi Ramamoorthi,et al.  Oriented Light-Field Windows for Scene Flow , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[36]  Bernd Jähne,et al.  Dense range flow from depth and intensity data , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[37]  Carsten Rother,et al.  FusionFlow: Discrete-continuous optimization for optical flow estimation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  Konrad Schindler,et al.  Piecewise Rigid Scene Flow , 2013, 2013 IEEE International Conference on Computer Vision.

[39]  Kiriakos N. Kutulakos,et al.  Multi-View Scene Capture by Surfel Sampling: From Video Streams to Non-Rigid 3D Motion, Shape and Reflectance , 2002, International Journal of Computer Vision.

[40]  Horst Bischof,et al.  CP-Census: A Novel Model for Dense Variational Scene Flow from RGB-D Data , 2014, BMVC.

[41]  Yael Moses,et al.  Multi-view Scene Flow Estimation: A View Centered Variational Approach , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[42]  In-So Kweon,et al.  A Tensor Voting Approach for Multi-view 3D Scene Flow Estimation and Refinement , 2012, ECCV.

[43]  Konrad Schindler,et al.  3D scene flow estimation with a rigid motion prior , 2011, 2011 International Conference on Computer Vision.

[44]  Roland Wilson,et al.  Multi-frame scene-flow estimation using a patch model and smooth motion prior , 2010 .

[45]  Kiriakos N. Kutulakos,et al.  Multi-view scene capture by surfel sampling: from video streams to non-rigid 3D motion, shape and reflectance , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[46]  Uwe Franke,et al.  Dense, Robust, and Accurate Motion Field Estimation from Stereo Image Sequences in Real-Time , 2010, ECCV.

[47]  Minglun Gong.  Real-time joint disparity and disparity flow estimation on programmable graphics hardware , 2009, Comput. Vis. Image Underst..

[48]  Yaser Sheikh,et al.  Photogeometric Scene Flow for High-Detail Dynamic 3D Reconstruction , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[49]  Thomas Brox,et al.  Dense Semi-rigid Scene Flow Estimation from RGBD Images , 2014, ECCV.

[50]  Daniel Cremers,et al.  A primal-dual framework for real-time dense RGB-D scene flow , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[51]  Michael J. Black,et al.  A Naturalistic Open Source Movie for Optical Flow Evaluation , 2012, ECCV.

[52]  Jitendra Malik,et al.  Depth from Combining Defocus and Correspondence Using Light-Field Cameras , 2013, 2013 IEEE International Conference on Computer Vision.

[53]  Uwe Franke,et al.  6D-Vision: Fusion of Stereo and Motion for Robust Environment Perception , 2005, DAGM-Symposium.

[54]  Jean-Philippe Pons,et al.  Dense and Accurate Spatio-temporal Multi-view Stereovision , 2009, ACCV.

[55]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[56]  Raquel Urtasun,et al.  Efficient Joint Segmentation, Occlusion Labeling, Stereo and Flow Estimation , 2014, ECCV.

[57]  Horst Bischof,et al.  A Duality Based Approach for Realtime TV-L1 Optical Flow , 2007, DAGM-Symposium.

[58]  Hatem A. Rashwan,et al.  Variational Optical Flow Estimation Based on Stick Tensor Voting , 2013, IEEE Transactions on Image Processing.

[59]  Luis Miguel Bergasa,et al.  On combining visual SLAM and dense scene flow to increase the robustness of localization and mapping in dynamic environments , 2012, 2012 IEEE International Conference on Robotics and Automation.

[60]  Richard Szeliski,et al.  High-accuracy stereo depth maps using structured light , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[61]  Dieter Fox,et al.  RGB-D flow: Dense 3-D motion estimation using color and depth , 2013, 2013 IEEE International Conference on Robotics and Automation.

[62]  Olivier D. Faugeras,et al.  Variational stereovision and 3D scene flow estimation with statistical similarity measures , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[63]  Ying Wu,et al.  Large Displacement Optical Flow from Nearest Neighbor Fields , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[64]  Bernd Jähne,et al.  Range Flow Estimation , 2002, Comput. Vis. Image Underst..

[65]  Daniel Cremers,et al.  Efficient Dense Scene Flow from Sparse or Dense Stereo Data , 2008, ECCV.

[66]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[67]  Hanno Scharr,et al.  Range Flow in Varying Illumination: Algorithms and Comparisons , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[68]  Michael Isard,et al.  Dense Motion and Disparity Estimation Via Loopy Belief Propagation , 2006, ACCV.

[69]  Joachim Weickert,et al.  Joint Estimation of Motion, Structure and Geometry from Stereo Sequences , 2010, ECCV.

[70]  Norbert Pfeifer,et al.  Motion estimation and segmentation in depth and intensity videos , 2014, Integr. Comput. Aided Eng..

[71]  Frederic Devernay,et al.  A Variational Method for Scene Flow Estimation from Stereo Sequences , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[72]  Michael Manzke,et al.  Estimating 3D Scene Flow from Multiple 2D Optical Flows , 2009, 2009 13th International Machine Vision and Image Processing Conference.

[73]  David J. Fleet,et al.  Computation of component image velocity from local phase information , 1990, International Journal of Computer Vision.

[74]  Jean Ponce,et al.  Dense 3D motion capture for human faces , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[75]  Robert B. Fisher,et al.  Colour Constrained 4D Flow , 2005, BMVC.

[76]  Daniel Cremers,et al.  Stereoscopic Scene Flow Computation for 3D Motion Understanding , 2011, International Journal of Computer Vision.

[77]  Hans-Peter Seidel,et al.  Motion capture using joint skeleton tracking and surface estimation , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[78]  Richard P. Wildes,et al.  Spatiotemporal Stereo and Scene Flow via Stequel Matching , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[79]  Horst Bischof,et al.  aTGV-SF: Dense Variational Scene Flow through Projective Warping and Higher Order Regularization , 2014, 2014 2nd International Conference on 3D Vision.

[80]  Janis Fehr,et al.  Computing Range Flow from Multi-modal Kinect Data , 2011, ISVC.

[81]  Yann LeCun,et al.  Computing the stereo matching cost with a convolutional neural network , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[82]  Rui Li,et al.  Multi-Scale 3D Scene Flow from Binocular Stereo Sequences , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[83]  Andrew W. Fitzgibbon,et al.  SphereFlow: 6 DoF Scene Flow from RGB-D Pairs , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[84]  Jörg Stückler,et al.  Motion Cooperation: Smooth Piece-wise Rigid Scene Flow from RGB-D Images , 2015, 2015 International Conference on 3D Vision.

[85]  Paria Mehrani,et al.  Superpixels and Supervoxels in an Energy Optimization Framework , 2010, ECCV.

[86]  Wenbin Li,et al.  Optical Flow Estimation Using Laplacian Mesh Energy , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[87]  Vladimir Kolmogorov,et al.  Optimizing Binary MRFs via Extended Roof Duality , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[88]  Hans-Hellmut Nagel,et al.  Optical Flow Estimation: Advances and Comparisons , 1994, ECCV.

[89]  Andreas Geiger,et al.  Object scene flow for autonomous vehicles , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[90]  Steven S. Beauchemin,et al.  The computation of optical flow , 1995, CSUR.

[91]  Andrés Bruhn,et al.  Joint trilateral filtering for multiframe optical flow , 2013, 2013 IEEE International Conference on Image Processing.

[92]  Takeo Kanade,et al.  Three-dimensional scene flow , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[93]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[94]  Yiannis Aloimonos,et al.  Spatio-Temporal Stereo Using Multi-Resolution Subdivision Surfaces , 2004, International Journal of Computer Vision.

[95]  James L. Crowley,et al.  Local/global scene flow estimation , 2013, 2013 IEEE International Conference on Image Processing.

[96]  Thomas Brox,et al.  A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[97]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[98]  Christian Theobalt,et al.  Dense Wide-Baseline Scene Flow from Two Handheld Video Cameras , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[99]  Michael J. Black,et al.  Secrets of optical flow estimation and their principles , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[100]  Nanning Zheng,et al.  Dense Scene Flow Based on Depth and Multi-channel Bilateral Filter , 2012, ACCV.

[101]  Jean Ponce,et al.  Dense 3D motion capture from synchronized video streams , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[102]  Jean Ponce,et al.  Dense 3D motion capture from synchronized video streams , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[103]  Deqing Sun,et al.  Layered RGBD scene flow estimation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[104]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[105]  Yasuyuki Matsushita,et al.  Motion detail preserving optical flow estimation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.