Stereoscopic Scene Flow Computation for 3D Motion Understanding

Building upon recent developments in optical flow and stereo matching estimation, we propose a variational framework for the estimation of stereoscopic scene flow, i.e., the motion of points in the three-dimensional world from stereo image sequences. The proposed algorithm takes into account image pairs from two consecutive times and computes both depth and a 3D motion vector associated with each point in the image. In contrast to previous works, we partially decouple the depth estimation from the motion estimation, which has many practical advantages. The variational formulation is quite flexible and can handle both sparse or dense disparity maps. The proposed method is very efficient; with the depth map being computed on an FPGA, and the scene flow computed on the GPU, the proposed algorithm runs at frame rates of 20 frames per second on QVGA images (320×240 pixels). Furthermore, we present solutions to two important problems in scene flow estimation: violations of intensity consistency between input images, and the uncertainty measures for the scene flow result.

[1]  P. Mahalanobis.  On the generalized distance in statistics , 1936 .

[2]  Louis A. Hageman,et al.  Iterative Solution of Large Linear Systems. , 1971 .

[3]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[4]  K. Stüben,et al.  Multigrid methods: Fundamental algorithms, model problem analysis and applications , 1982 .

[5]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[6]  T. Kanade,et al.  A multi-body factorization method for motion analysis , 1995, ICCV 1995.

[7]  Georgios Tziritas,et al.  Joint disparity and motion field estimation in stereoscopic image sequences , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[8]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[9]  Patrick Pérez,et al.  Dense estimation and object-based segmentation of the optical flow with robust techniques , 1998, IEEE Trans. Image Process..

[10]  Uwe Franke,et al.  Real-time stereo vision for urban traffic scene understanding , 2000, Proceedings of the IEEE Intelligent Vehicles Symposium 2000 (Cat. No.00TH8511).

[11]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[12]  Ye Zhang,et al.  On 3D scene flow and structure estimation , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[13]  Andrew Zisserman,et al.  Multiple view geometry in computer visiond , 2001 .

[14]  M. Shimizu,et al.  Precise sub-pixel estimation on area-based matching , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[15]  G. Tziritas,et al.  A Joint Motion / Disparity Estimation Method for the Construction of Stereo Interpolated Images in Stereoscopic Image Sequences , 2001 .

[16]  Kenichi Kanatani,et al.  Multi-stage Optimization for Multi-body Motion Segmentation , 2003 .

[17]  Hernán Badino,et al.  A Robust Approach for Ego-Motion Estimation Using a Mobile Stereo Platform , 2004, IWCM.

[18]  R. Zabih,et al.  What energy functions can be minimized via graph cuts , 2004 .

[19]  Vladimir Kolmogorov,et al.  An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[21]  Fridtjof Stein,et al.  Efficient Computation of Optical Flow Using the Census Transform , 2004, DAGM-Symposium.

[22]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[23]  Timo Kohlberger,et al.  Discontinuity-Preserving Computation of Variational Optic Flow in Real-Time , 2005, Scale-Space.

[24]  Takeo Kanade,et al.  Three-dimensional scene flow , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Tony F. Chan,et al.  Structure-Texture Image Decomposition—Modeling, Algorithms, and Parameter Selection , 2006, International Journal of Computer Vision.

[26]  Marc Pollefeys,et al.  A General Framework for Motion Segmentation: Independent, Articulated, Rigid, Non-rigid, Degenerate and Non-degenerate , 2006, ECCV.

[27]  J. Weickert,et al.  A Confidence Measure for Variational Optic flow Methods , 2006 .

[28]  Minglun Gong,et al.  Disparity Flow Estimation using Orthogonal Reliability-based Dynamic Programming , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[29]  Michael Isard,et al.  Dense Motion and Disparity Estimation Via Loopy Belief Propagation , 2006, ACCV.

[30]  Kwanghoon Sohn,et al.  Edge-preserving Simultaneous Joint Motion-Disparity Estimation , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[31]  Heiko Hirschmüller,et al.  Stereo Vision in Structured Environments by Consistent Semi-Global Matching , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[32]  Olivier D. Faugeras,et al.  Multi-View Stereo Reconstruction and Scene Flow Estimation with a Global Image-Based Matching Score , 2007, International Journal of Computer Vision.

[33]  Horst Bischof,et al.  A Duality Based Approach for Realtime TV-L1 Optical Flow , 2007, DAGM-Symposium.

[34]  C. Rabe,et al.  Fast detection of moving objects in complex scenarios , 2007, 2007 IEEE Intelligent Vehicles Symposium.

[35]  Frederic Devernay,et al.  A Variational Method for Scene Flow Estimation from Stereo Sequences , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[36]  H. Hirschmüller.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  Daniel Cremers,et al.  An Improved Algorithm for TV-L 1 Optical Flow , 2009, Statistical and Geometrical Approaches to Visual Motion Analysis.

[38]  Daniel Cremers,et al.  Efficient Dense Scene Flow from Sparse or Dense Stereo Data , 2008, ECCV.

[39]  René Vidal,et al.  Motion segmentation via robust subspace separation in the presence of outlying, incomplete, or corrupted trajectories , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  T. Vaudrey,et al.  Differences between stereo and motion behaviour on synthetic and real-world stereo sequences , 2008, 2008 23rd International Conference Image and Vision Computing New Zealand.

[41]  Daniel Cremers,et al.  An Evaluation Approach for Scene Flow with Decoupled Motion and Position , 2008, Statistical and Geometrical Approaches to Visual Motion Analysis.

[42]  Hans-Peter Seidel,et al.  Complementary Optic Flow , 2009, EMMCVPR.

[43]  Minglun Gong.  Real-time joint disparity and disparity flow estimation on programmable graphics hardware , 2009, Comput. Vis. Image Underst..

[44]  Daniel Cremers,et al.  Detection and Segmentation of Independently Moving Objects from Dense Scene Flow , 2009, EMMCVPR.

[45]  Horst Bischof,et al.  Motion estimation with non-local total variation regularization , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[46]  Xiaoyan Hu,et al.  Evaluation of stereo confidence indoors and outdoors , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[47]  Reinhard Klette,et al.  Generalised residual images' effect on illumination artifact removal for correspondence algorithms , 2011, Pattern Recognit..