Spatiotemporal stereo via spatiotemporal quadric element (stequel) matching

Spatiotemporal stereo is concerned with the recovery of the 3D structure of a dynamic scene from a temporal sequence of multiview images. This paper presents a novel method for computing temporally coherent disparity maps from a sequence of binocular images through an integrated consideration of image spacetime structure and without explicit recovery of motion. The approach is based on matching spatiotemporal quadric elements (stequels) between views, as it is shown that this matching primitive provides a natural way to encapsulate both local spatial and temporal structure for disparity estimation. Empirical evaluation with laboratory based imagery with ground truth and more typical natural imagery shows that the approach provides considerable benefit in comparison to alternative methods for enforcing temporal coherence in disparity estimation.

[1]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[2]  Allen M. Waxman,et al.  Binocular Image Flows: Steps Toward Stereo-Motion Fusion , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  John K. Tsotsos,et al.  Applying temporal constraints to the dynamic stereo problem , 1986, Comput. Vis. Graph. Image Process..

[4]  D J Heeger,et al.  Model for the extraction of image flow. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[5]  Edward H. Adelson,et al.  The Design and Use of Steerable Filters , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  Jitendra Malik,et al.  A Computational Framework for Determining Stereo Correspondence from a Set of Linear Spatial Filters , 1991, ECCV.

[7]  Paul R. Cohen,et al.  Motion and structure estimation from stereo image sequences , 1992, IEEE Trans. Robotics Autom..

[8]  Keith J. Hanna,et al.  Combining stereo and motion analysis for direct estimation of scene structure , 1993, 1993 (4th) International Conference on Computer Vision.

[9]  Kanad K. Biswas,et al.  A cooperative integration of stereopsis and optic flow computation , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[10]  Hans Knutsson,et al.  Signal processing for computer vision , 1994 .

[11]  Steven S. Beauchemin,et al.  The computation of optical flow , 1995, CSUR.

[12]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[13]  Michael G. Strintzis,et al.  Model-Based Joint Motion and Structure Estimation from Stereo Images , 1997, Comput. Vis. Image Underst..

[14]  Amnon Shashua,et al.  Direct estimation of motion and extended scene structure from a moving stereo rig , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[15]  Harpreet S. Sawhney,et al.  Correlation-based estimation of ego-motion and structure from motion and stereo , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[16]  Vladimir Kolmogorov,et al.  Computing visual correspondence with occlusions using graph cuts , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[17]  Y. Aloimonos,et al.  Spatio-Temporal Stereo Using Multi-Resolution Subdivision Surfaces , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[18]  Luc Van Gool,et al.  Motion - Stereo Integration for Depth Estimation , 2002, ECCV.

[19]  Richard Szeliski,et al.  High-accuracy stereo depth maps using structured light , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[20]  Li Zhang,et al.  Spacetime stereo: shape recovery for dynamic scenes , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[21]  Olivier D. Faugeras,et al.  Variational stereovision and 3D scene flow estimation with statistical similarity measures , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[22]  Changming Sun,et al.  An energy minimisation approach to stereo-temporal dense reconstruction , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[23]  Simon Baker,et al.  Lucas-Kanade 20 Years On: A Unifying Framework , 2004, International Journal of Computer Vision.

[24]  Eero P. Simoncelli,et al.  Differentiation of Discrete Multi-Dimensional Signals , 2004 .

[25]  Trevor Darrell,et al.  Using Multiple-Hypothesis Disparity Maps and Image Velocity for 3-D Motion Estimation , 2004, International Journal of Computer Vision.

[26]  Olivier D. Faugeras,et al.  Three-dimensional motion computation and object segmentation in a long sequence of stereo frames , 1992, International Journal of Computer Vision.

[27]  Michael Isard,et al.  Estimating disparity and occlusions in stereo video sequences , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[28]  Eli Shechtman,et al.  Space-time behavior based correlation , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[29]  Szymon Rusinkiewicz,et al.  Spacetime stereo: a unifying framework for depth from triangulation , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Konstantinos G. Derpanis,et al.  Three-dimensional nth derivative of Gaussian separable steerable filters , 2005, IEEE International Conference on Image Processing 2005.

[31]  Michael Isard,et al.  Dense Motion and Disparity Estimation Via Loopy Belief Propagation , 2006, ACCV.

[32]  Minglun Gong.  Enforcing Temporal Consistency in Real-Time Stereo Estimation , 2006, ECCV.

[33]  Richard P. Wildes,et al.  Efficient Stereo with Accurate 3-D Boundaries , 2006, BMVC.

[34]  Marc Pollefeys,et al.  Temporally Consistent Reconstruction from Multiple Video Streams Using Enhanced Belief Propagation , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[35]  Eli Shechtman,et al.  Space-Time Behavior-Based Correlation-OR-How to Tell If Two Underlying Motion Fields Are Similar Without Computing Them? , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Frederic Devernay,et al.  A Variational Method for Scene Flow Estimation from Stereo Sequences , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[37]  Richard P. Wildes,et al.  Spatiotemporal Oriented Energy Features for Visual Tracking , 2007, ACCV.

[38]  Richard P. Wildes,et al.  Spatiotemporal stereo via spatiotemporal quadric element (stequel) matching , 2009, CVPR.

[39]  R. Wildes,et al.  Early spatiotemporal grouping with a distributed oriented energy representation , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.