Surfaces with occlusions from layered stereo

We propose a new binocular stereo algorithm that estimates scene structure as a collection of smooth surface patches. The disparities within each patch are modeled by a continuous-valued spline, while the extent of each patch is represented via a pixelwise partitioning of the images. Disparities and extents are alternately estimated in an iterative, energy minimization framework. Experimental results demonstrate that, for scenes consisting of smooth surfaces, the proposed algorithm significantly improves upon the state of the art.

[1]  B. Julesz.  Binocular depth perception of computer-generated patterns , 1960 .

[2]  Lawrence G. Roberts,et al.  Machine Perception of Three-Dimensional Solids , 1963, Outstanding Dissertations in the Computer Sciences.

[3]  J. Pokorny.  Foundations of Cyclopean Perception , 1972 .

[4]  D Marr,et al.  Cooperative computation of stereo disparity. , 1976, Science.

[5]  Thomas O. Binford,et al.  Depth from Edge and Intensity Based Stereo , 1981, IJCAI.

[6]  Eric L. W. Grimson,et al.  From Images to Surfaces: A Computational Study of the Human Early Visual System , 1981 .

[7]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[8]  Martin A. Fischler,et al.  Computational Stereo , 1982, CSUR.

[9]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Tomaso Poggio,et al.  Computational vision and regularization theory , 1985, Nature.

[11]  Takeo Kanade,et al.  Stereo by Intra- and Inter-Scanline Search Using Dynamic Programming , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Demetri Terzopoulos,et al.  Regularization of Inverse Visual Problems Involving Discontinuities , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Demetri Terzopoulos,et al.  Image Analysis Using Multigrid Relaxation Methods , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Qi Tian,et al.  Algorithms for subpixel registration , 1986 .

[15]  Andrew Blake,et al.  Visual Reconstruction , 1987, Deep Learning for EEG-Based Brain–Computer Interfaces.

[16]  D. Ruppert.  Robust Statistics: The Approach Based on Influence Functions , 1987 .

[17]  David Lee,et al.  One-Dimensional Regularization with Discontinuities , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Jake K. Aggarwal,et al.  Structure from stereo-a review , 1989, IEEE Trans. Syst. Man Cybern..

[19]  Y. J. Tejwani,et al.  Robot vision , 1989, IEEE International Symposium on Circuits and Systems,.

[20]  Narendra Ahuja,et al.  Surfaces from Stereo: Integrating Feature Matching, Disparity Estimation, and Contour Detection , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  A. Pentland,et al.  Robust estimation of a multi-layered motion representation , 1991, Proceedings of the IEEE Workshop on Visual Motion.

[22]  P. Toint,et al.  Lancelot: A FORTRAN Package for Large-Scale Nonlinear Optimization (Release A) , 1992 .

[23]  David Mumford,et al.  A Bayesian treatment of the stereo correspondence problem using half-occluded regions , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[24]  Lisa M. Brown,et al.  A survey of image registration techniques , 1992, CSUR.

[25]  Peter N. Belhumeur,et al.  A binocular stereo algorithm for reconstructing sloping, creased, and broken surfaces in the presence of half-occlusion , 1993, 1993 (4th) International Conference on Computer Vision.

[26]  Edward H. Adelson,et al.  Layered representation for motion analysis , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Ramin Zabih,et al.  Non-parametric Local Transforms for Computing Visual Correspondence , 1994, ECCV.

[28]  Takeo Kanade,et al.  A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Aaron F. Bobick,et al.  Disparity-Space Images and Large Occlusion Stereo , 1994, ECCV.

[30]  Jake K. Aggarwal,et al.  Stereo Matching in the Presence of Narrow Occluding Objects Using Dynamic Disparity Search , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Alex Pentland,et al.  Cooperative Robust Estimation Using Layers of Support , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Joachim Weickert,et al.  Anisotropic diffusion in image processing , 1996 .

[33]  Ingemar J. Cox,et al.  A Maximum Likelihood Stereo Algorithm , 1996, Comput. Vis. Image Underst..

[34]  Michael J. Black,et al.  On the unification of line processes , 1996 .

[35]  Ingemar J. Cox,et al.  A maximum-flow formulation of the N-camera stereo correspondence problem , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[36]  Davi Geiger,et al.  Occlusions, Discontinuities, and Epipolar Lines in Stereo , 1998, ECCV.

[37]  Guillermo Sapiro,et al.  Robust anisotropic diffusion , 1998, IEEE Trans. Image Process..

[38]  Richard Szeliski,et al.  A layered approach to stereo reconstruction , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[39]  Max A. Viergever,et al.  Efficient and reliable schemes for nonlinear diffusion filtering , 1998, IEEE Trans. Image Process..

[40]  Carlo Tomasi,et al.  A Pixel Dissimilarity Measure That Is Insensitive to Image Sampling , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  Olga Veksler,et al.  Markov random fields with efficient approximations , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[42]  Olga Veksler,et al.  A New Algorithm for Energy Minimization with Discontinuities , 1999, EMMCVPR.

[43]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[44]  Stan Birchfield,et al.  Depth and motion discontinuities , 1999 .

[45]  Carlo Tomasi,et al.  Color edge detection with the compass operator , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[46]  R. Zabih,et al.  Efficient Graph-Based Energy Minimization Methods in Computer Vision , 1999 .

[47]  Richard Szeliski,et al.  An Experimental Comparison of Stereo Algorithms , 1999, Workshop on Vision Algorithms.

[48]  Chandra Kambhamettu,et al.  Recovery and tracking of continuous 3D surfaces from stereo data using a deformable dual-mesh , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[49]  Carlo Tomasi,et al.  Multiway cut for stereo and motion with slanted surfaces , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[50]  Ronen Basri,et al.  Fast multiscale image segmentation , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[51]  Carlo Tomasi,et al.  Alpha estimation in natural images , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[52]  Takeo Kanade,et al.  A Cooperative Algorithm for Stereo Matching and Occlusion Detection , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[53]  Hai Tao,et al.  Global matching criterion and color segmentation based stereo , 2000, Proceedings Fifth IEEE Workshop on Applications of Computer Vision.

[54]  A Global Matching Framework for Stereo Computation , 2001, ICCV.

[55]  Vladimir Kolmogorov,et al.  Computing visual correspondence with occlusions using graph cuts , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[56]  Masatoshi Okutomi,et al.  Precise Sub-Pixel Estimation on Area-Based Matching , 2001, ICCV.

[57]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[58]  Richard Szeliski,et al.  Symmetric Sub-Pixel Stereo Matching , 2002, ECCV.

[59]  Ye Zhang,et al.  Stereo Matching with Segmentation-Based Cooperation , 2002, ECCV.

[60]  Nanning Zheng,et al.  Stereo Matching Using Belief Propagation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[61]  Mariano Rivera,et al.  Adaptive Rest Condition Potentials: Second Order Edge-Preserving Regularization , 2002, ECCV.

[62]  Masatoshi Okutomi,et al.  A Simple Stereo Algorithm to Recover Precise Object Boundaries and Smooth Surfaces , 2004, International Journal of Computer Vision.

[63]  Carlo Tomasi,et al.  Surfaces with occlusions from layered stereo , 2004, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[64]  Jonathan M. Garibaldi,et al.  Real-Time Correlation-Based Stereo Vision with Reduced Border Errors , 2002, International Journal of Computer Vision.

[65]  Aaron F. Bobick,et al.  Large Occlusion Stereo , 1999, International Journal of Computer Vision.

[66]  Vladimir Kolmogorov,et al.  What energy functions can be minimized via graph cuts? , 2002, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[67]  Richard Szeliski,et al.  Stereo Matching with Nonlinear Diffusion , 1998, International Journal of Computer Vision.

[68]  Richard Szeliski,et al.  Spline-Based Image Registration , 1997, International Journal of Computer Vision.

[69]  Michael J. Black,et al.  On the unification of line processes, outlier rejection, and robust statistics with applications in early vision , 1996, International Journal of Computer Vision.

[70]  B. Ripley,et al.  Robust Statistics , 2018, Wiley Series in Probability and Statistics.

[71]  Carlo Tomasi,et al.  Depth Discontinuities by Pixel-to-Pixel Stereo , 1999, International Journal of Computer Vision.

[72]  Richard Szeliski,et al.  Stereo Matching with Transparency and Matting , 1999, International Journal of Computer Vision.

[73]  Peter N. Belhumeur,et al.  A Bayesian approach to binocular steropsis , 1996, International Journal of Computer Vision.

[74]  Stephen T. Barnard,et al.  Stochastic stereo matching over scale , 1989, International Journal of Computer Vision.

[75]  Juliang Shao,et al.  Generation of Temporally Consistent Multiple Virtual Camera Views from Stereoscopic Image Sequences , 2002, International Journal of Computer Vision.

[76]  An Luo,et al.  An intensity-based cooperative bidirectional stereo matching with simultaneous detection of discontinuities and occlusions , 1995, International Journal of Computer Vision.

[77]  Alan L. Yuille,et al.  Occlusions and binocular stereo , 1992, International Journal of Computer Vision.