Spatiotemporal stereo via spatiotemporal quadric element (stequel) matching

Spatiotemporal stereo is concerned with the recovery of the 3D structure of a dynamic scene from a temporal sequence of multiview images. This paper presents a novel method for computing temporally coherent disparity maps from a sequence of binocular images through an integrated consideration of image spacetime structure and without explicit recovery of motion. The approach is based on matching spatiotemporal quadric elements (stequels) between views, as it is shown that this matching primitive provides a natural way to encapsulate both local spatial and temporal structure for disparity estimation. Empirical evaluation with laboratory based imagery with ground truth and more typical natural imagery shows that the approach provides considerable benefit in comparison to alternative methods for enforcing temporal coherence in disparity estimation.

[1]  Keith J. Hanna,et al.  Combining stereo and motion analysis for direct estimation of scene structure , 1993, 1993 (4th) International Conference on Computer Vision.

[2]  Trevor Darrell,et al.  Using Multiple-Hypothesis Disparity Maps and Image Velocity for 3-D Motion Estimation , 2004, International Journal of Computer Vision.

[3]  Steven S. Beauchemin,et al.  The computation of optical flow , 1995, CSUR.

[4]  Yiannis Aloimonos,et al.  Spatio-Temporal Stereo Using Multi-Resolution Subdivision Surfaces , 2004, International Journal of Computer Vision.

[5]  Eero P. Simoncelli,et al.  Differentiation of discrete multidimensional signals , 2004, IEEE Transactions on Image Processing.

[6]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[7]  Allen M. Waxman,et al.  Binocular Image Flows: Steps Toward Stereo-Motion Fusion , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Hans Knutsson,et al.  Signal processing for computer vision , 1994 .

[9]  Konstantinos G. Derpanis,et al.  Three-dimensional nth derivative of Gaussian separable steerable filters , 2005, IEEE International Conference on Image Processing 2005.

[10]  Richard P. Wildes,et al.  Early spatiotemporal grouping with a distributed oriented energy representation , 2009, CVPR.

[11]  Olivier D. Faugeras,et al.  Three-dimensional motion computation and object segmentation in a long sequence of stereo frames , 1992, International Journal of Computer Vision.

[12]  Marc Pollefeys,et al.  Temporally Consistent Reconstruction from Multiple Video Streams Using Enhanced Belief Propagation , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[13]  Jitendra Malik,et al.  Computational framework for determining stereo correspondence from a set of linear spatial filters , 1992, Image Vis. Comput..

[14]  Changming Sun,et al.  An energy minimisation approach to stereo-temporal dense reconstruction , 2004, ICPR 2004.

[15]  Michael G. Strintzis,et al.  Model-Based Joint Motion and Structure Estimation from Stereo Images , 1997, Comput. Vis. Image Underst..

[16]  Frederic Devernay,et al.  A Variational Method for Scene Flow Estimation from Stereo Sequences , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[17]  Harpreet S. Sawhney,et al.  Correlation-based estimation of ego-motion and structure from motion and stereo , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[18]  Kanad K. Biswas,et al.  Cooperative integration of stereopsis and optic flow computation , 1995 .

[19]  D J Heeger,et al.  Model for the extraction of image flow. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[20]  Szymon Rusinkiewicz,et al.  Spacetime Stereo: A Unifying Framework for Depth from Triangulation , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[22]  Richard P. Wildes,et al.  Efficient Stereo with Accurate 3-D Boundaries , 2006, BMVC.

[23]  Michael Isard,et al.  Estimating disparity and occlusions in stereo video sequences , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[24]  Richard Szeliski,et al.  High-accuracy stereo depth maps using structured light , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[25]  Richard P. Wildes,et al.  Spatiotemporal Oriented Energy Features for Visual Tracking , 2007, ACCV.

[26]  Olivier D. Faugeras,et al.  Variational stereovision and 3D scene flow estimation with statistical similarity measures , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[27]  John K. Tsotsos,et al.  Applying temporal constraints to the dynamic stereo problem , 1986, Comput. Vis. Graph. Image Process..

[28]  Minglun Gong.  Enforcing Temporal Consistency in Real-Time Stereo Estimation , 2006, ECCV.

[29]  Eli Shechtman,et al.  Space-time behavior based correlation , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[30]  Edward H. Adelson,et al.  The Design and Use of Steerable Filters , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Paul R. Cohen,et al.  Motion and structure estimation from stereo image sequences , 1992, IEEE Trans. Robotics Autom..

[32]  Michael Isard,et al.  Dense Motion and Disparity Estimation Via Loopy Belief Propagation , 2006, ACCV.

[33]  Li Zhang,et al.  Spacetime stereo: shape recovery for dynamic scenes , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[34]  Simon Baker,et al.  Lucas-Kanade 20 Years On: A Unifying Framework , 2004, International Journal of Computer Vision.

[35]  Luc Van Gool,et al.  Motion - Stereo Integration for Depth Estimation , 2002, ECCV.

[36]  Amnon Shashua,et al.  Direct estimation of motion and extended scene structure from a moving stereo rig , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[37]  Vladimir Kolmogorov,et al.  Computing visual correspondence with occlusions using graph cuts , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[38]  Eli Shechtman,et al.  Space-Time Behavior-Based Correlation-OR-How to Tell If Two Underlying Motion Fields Are Similar Without Computing Them? , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.