A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms

This paper presents a quantitative comparison of several multi-view stereo reconstruction algorithms. Until now, the lack of suitable calibrated multi-view image datasets with known ground truth (3D shape models) has prevented such direct comparisons. In this paper, we first survey multi-view stereo algorithms and compare them qualitatively using a taxonomy that differentiates their key properties. We then describe our process for acquiring and calibrating multiview image datasets with high-accuracy ground truth and introduce our evaluation methodology. Finally, we present the results of our quantitative comparison of state-of-the-art multi-view stereo reconstruction algorithms on six benchmark datasets. The datasets, evaluation details, and instructions for submitting new models are available online at http://vision.middlebury.edu/mview.

[1]  Olivier D. Faugeras,et al.  Representing stereo data with the Delaunay triangulation , 1988, Proceedings. 1988 IEEE International Conference on Robotics and Automation.

[2]  Takeo Kanade,et al.  A multiple-baseline stereo , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[3]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Thomas Fromherz,et al.  Shape from Multiple Cues: Integrating Local Brightness Information , 1995 .

[5]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[6]  Alyn P. Rockwood,et al.  Three-dimensional object reconstruction from two-dimensional images , 1997, Comput. Aided Des..

[7]  Ingemar J. Cox,et al.  A maximum-flow formulation of the N-camera stereo correspondence problem , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[8]  Olivier D. Faugeras,et al.  Variational principles, surface evolution, PDEs, level set methods, and the stereo problem , 1998, IEEE Trans. Image Process..

[9]  Takeo Kanade,et al.  Constructing virtual worlds using dense stereo , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[10]  Peter Eisert,et al.  Multi-hypothesis, volumetric reconstruction of 3-D objects from multiple calibrated camera views , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[11]  Jeremy S. De Bonet,et al.  Poxels: Probabilistic Voxelized Volume Reconstruction , 1999 .

[12]  Richard Szeliski,et al.  Prediction error as a quality metric for motion and stereo , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[13]  Richard Szeliski,et al.  A multi-view approach to motion and stereo , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[14]  Takeo Kanade,et al.  Shape reconstruction in projective grid space from large number of images , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[15]  Charles V. Stewart,et al.  Robust Parameter Estimation in Computer Vision , 1999, SIAM Rev..

[16]  David Salesin,et al.  Surface light fields for 3D photography , 2000, SIGGRAPH.

[17]  Takeo Kanade,et al.  Image-consistent surface triangulation , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[18]  Kiriakos N. Kutulakos.  Approximate N-View Stereo , 2000, ECCV.

[19]  Adrian Hilton,et al.  Reconstruction of scene models from sparse 3D structure , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[20]  Steven M. Seitz,et al.  Image-based multiresolution shape recovery by surface deformation , 2000, IS&T/SPIE Electronic Imaging.

[21]  Richard Szeliski,et al.  Handling occlusions in dense multi-view stereo , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[22]  Thomas Malzbender,et al.  Improved Voxel Coloring Via Volumetric Optimization , 2001 .

[23]  C. Dyer.  Volumetric Scene Reconstruction from Multiple Views , 2001 .

[24]  R. Cipolla,et al.  A probabilistic framework for space carving , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[25]  Jean-Yves Bouguet,et al.  Camera calibration toolbox for matlab , 2001 .

[26]  Thomas Malzbender,et al.  A Survey of Methods for Volumetric Scene Reconstruction from Photographs , 2001, VG.

[27]  Pietro Perona,et al.  Shadow Carving , 2001, ICCV.

[28]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[29]  Vladimir Kolmogorov,et al.  Multi-camera Scene Reconstruction via Graph Cuts , 2002, ECCV.

[30]  Wei-Chao Chen,et al.  Light field mapping: efficient representation and hardware rendering of surface light fields , 2002, SIGGRAPH.

[31]  O. Faugeras,et al.  Variational principles, surface evolution, PDE's, level set methods and the stereo problem , 1998, 5th IEEE EMBS International Summer School on Biomedical Imaging, 2002..

[32]  Kiriakos N. Kutulakos,et al.  A Probabilistic Theory of Occupancy and Emptiness , 2002, ECCV.

[33]  Stefano Soatto,et al.  Tales of shape and radiance in multiview stereo , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[34]  Ruigang Yang,et al.  Dealing with textureless regions and specular highlights - a progressive space carving scheme using a novel photo-consistency measure , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[35]  Peter F. Sturm,et al.  Voxel carving for specular surfaces , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[36]  Olivier D. Faugeras,et al.  Variational stereovision and 3D scene flow estimation with statistical similarity measures , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[37]  Camillo J. Taylor.  Surface reconstruction from feature based stereo , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[38]  Francis Schmitt,et al.  Silhouette and stereo fusion for 3D object modeling , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[39]  Stefano Soatto,et al.  Multi-view stereo beyond Lambert , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[40]  Stan Sclaroff,et al.  Stochastic refinement of the visual hull to satisfy photometric and silhouette consistency constraints , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[41]  Takeo Kanade,et al.  When Is the Shape of a Scene Unique Given Its Light-Field: A Fundamental Theorem of 3D Vision? , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Mark R. Stevens,et al.  Methods for Volumetric Reconstruction of Visual Scenes , 2004, International Journal of Computer Vision.

[43]  Francis Schmitt,et al.  Silhouette and stereo fusion for 3D object modeling , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[44]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[45]  Richard Szeliski,et al.  Stereo Matching with Transparency and Matting , 1999, International Journal of Computer Vision.

[46]  Ross T. Whitaker,et al.  Higher-order nonlinear priors for surface reconstruction , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Adrien Treuille,et al.  Example-Based Stereo with General BRDFs , 2004, ECCV.

[48]  Hong Qin,et al.  Shape Reconstruction from 3D and 2D Data Using PDE-Based Deformable Surfaces , 2004, ECCV.

[49]  Steven M. Seitz,et al.  Photorealistic Scene Reconstruction by Voxel Coloring , 1997, International Journal of Computer Vision.

[50]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.

[51]  Olivier D. Faugeras,et al.  Modelling dynamic scenes by registering multi-view image sequences , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[52]  Roberto Cipolla,et al.  Multi-view stereo via volumetric graph-cuts , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[53]  Pau Gargallo,et al.  Bayesian 3D modeling from images using multiple depth maps , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[54]  Sébastien Roy,et al.  Geo-consistency for wide multi-camera stereo , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[55]  Stefano Soatto,et al.  Multi-View Stereo Reconstruction of Dense Shape and Complex Appearance , 2005, International Journal of Computer Vision.

[56]  Pascal Fua,et al.  Object-centered surface reconstruction: Combining multi-image stereo and shading , 1995, International Journal of Computer Vision.

[57]  Marc Pollefeys,et al.  Multi-view reconstruction using photo-consistency and exact silhouette constraints: a maximum-flow formulation , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[58]  Long Quan,et al.  Progressive surface reconstruction from images using a local prior , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[59]  Tianli Yu,et al.  Shape and View Independent Reflectance Map from Multiple Views , 2004, International Journal of Computer Vision.

[60]  Michael Goesele,et al.  Multi-View Stereo Revisited , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[61]  Sebastian Thrun,et al.  A Bayesian method for probable surface reconstruction and decimation , 2006, TOGS.

[62]  Jean Ponce,et al.  High-fidelity image-based modeling , 2008 .

[63]  Roberto Cipolla,et al.  Reconstructing relief surfaces , 2008, Image and Vision Computing.