Semi-global Stereo Matching with Surface Orientation Priors

Semi-Global Matching (SGM) is a widely-used efficient stereo matching technique. It works well for textured scenes, but fails on untextured slanted surfaces due to its fronto-parallel smoothness assumption. To remedy this problem, we propose a simple extension, termed SGM-P, to utilize precomputed surface orientation priors. Such priors favor different surface slants in different 2D image regions or 3D scene regions and can be derived in various ways. In this paper we evaluate plane orientation priors derived from stereo matching at a coarser resolution and show that such priors can yield significant performance gains for difficult weakly-textured scenes. We also explore surface normal priors derived from Manhattan-world assumptions, and we analyze the potential performance gains using oracle priors derived from ground-truth data. SGM-P only adds a minor computational overhead to SGM and is an attractive alternative to more complex methods employing higher-order smoothness terms.

[1]  Robert T. Collins,et al.  A space-sweep approach to true multi-image matching , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[2]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[3]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[4]  Darius Burschka,et al.  Advances in Computational Stereo , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information , 2005, CVPR.

[6]  Nahum Kiryati,et al.  Photometric stereo under perspective projection , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[7]  Robert C. Bolles,et al.  Outdoor Mapping and Navigation Using Stereo Vision , 2006, ISER.

[8]  Frank Dellaert,et al.  Line-Based Structure from Motion for Urban Environments , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[9]  Heiko Hirschmüller,et al.  Stereo Vision in Structured Environments by Consistent Semi-Global Matching , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[10]  Radim Sára,et al.  Feasibility Boundary in Dense and Semi-Dense Stereo Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Jan-Michael Frahm,et al.  Real-Time Plane-Sweeping Stereo with Multiple Sweeping Directions , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  H. Hirschmüller.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Andrew W. Fitzgibbon,et al.  Global stereo reconstruction under second order smoothness priors , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Richard Szeliski,et al.  Manhattan-world stereo , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  T. Kanade,et al.  Geometric reasoning for single image structure recovery , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Stefan K. Gehrig,et al.  A Real-Time Low-Power Stereo Vision Engine Using Semi-Global Matching , 2009, ICVS.

[17]  Reinhard Klette,et al.  Inclusion of a Second-Order Prior into Semi-Global Matching , 2009, PSIVT.

[18]  Richard Szeliski,et al.  Piecewise planar stereo for image-based rendering , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[19]  Andreas Geiger,et al.  Efficient Large-Scale Stereo Matching , 2010, ACCV.

[20]  Uwe Franke,et al.  Dense, Robust, and Accurate Motion Field Estimation from Stereo Image Sequences in Real-Time , 2010, ECCV.

[21]  Steven W. Zucker,et al.  Differential Geometric Inference in Surface Stereo , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Pushmeet Kohli,et al.  Surface stereo with soft segmentation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[23]  Pushmeet Kohli,et al.  Object stereo — Joint stereo matching and object segmentation , 2011, CVPR 2011.

[24]  Peter Pirsch,et al.  Real-time semi-global matching disparity estimation on the GPU , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[25]  Heiko Hirschmüller,et al.  Semi-Global Matching-Motivation, Developments and Applications , 2011 .

[26]  Carsten Rother,et al.  PatchMatch Stereo - Stereo Matching with Slanted Support Windows , 2011, BMVC.

[27]  Steven M. Seitz,et al.  Binocular Photometric Stereo , 2011, BMVC.

[28]  Tamir Hazan,et al.  Continuous Markov Random Fields for Robust Stereo Estimation , 2012, ECCV.

[29]  Richard Szeliski,et al.  Multiple View Object Cosegmentation Using Appearance and Stereo Cues , 2012, ECCV.

[30]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  M. Rothermel,et al.  SURE – The ifp Software for Dense Image Matching , 2013 .

[32]  Andrew W. Fitzgibbon,et al.  PMBP: PatchMatch Belief Propagation for Correspondence Field Estimation , 2014, International Journal of Computer Vision.

[33]  M. Rothermel,et al.  SURE : PHOTOGRAMMETRIC SURFACE RECONSTRUCTION FROM IMAGER Y , 2013 .

[34]  Martial Hebert,et al.  Data-Driven 3D Primitives for Single Image Understanding , 2013, 2013 IEEE International Conference on Computer Vision.

[35]  Jaishanker K. Pillai,et al.  Manhattan Junction Catalogue for Spatial Reasoning of Indoor Scenes , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Carl Olsson,et al.  In Defense of 3D-Label Stereo , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Martial Hebert,et al.  Unfolding an Indoor Origami World , 2014, ECCV.

[38]  Shai Avidan,et al.  Semi-Global Matching: A Principled Derivation in Terms of Message Passing , 2014, GCPR.

[39]  Xi Wang,et al.  High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth , 2014, GCPR.

[40]  Takeshi Naemura,et al.  Graph Cut Based Continuous Stereo Matching Using Locally Shared Labels , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Marc Pollefeys,et al.  Discriminatively Trained Dense Surface Normal Estimation , 2014, ECCV.

[42]  Richard Szeliski,et al.  Efficient High-Resolution Stereo Matching Using Local Plane Sweeps , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Richard Bowden,et al.  Exploiting High Level Scene Cues in Stereo Reconstruction , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[44]  Marc Pollefeys,et al.  Direction matters: Depth estimation with a surface normal classifier , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Enric Meinhardt,et al.  MGM: A Significantly More Global Matching for Stereovision , 2015, BMVC.

[46]  Yann LeCun,et al.  Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches , 2015, J. Mach. Learn. Res..

[47]  Hujun Bao,et al.  Robust stereo matching with surface normal prediction , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).