High-quality shape from multi-view stereo and shading under general illumination

Multi-view stereo methods reconstruct 3D geometry from images well for sufficiently textured scenes, but often fail to recover high-frequency surface detail, particularly for smoothly shaded surfaces. On the other hand, shape-from-shading methods can recover fine detail from shading variations. Unfortunately, it is non-trivial to apply shape-from-shading alone to multi-view data, and most shading-based estimation methods only succeed under very restricted or controlled illumination. We present a new algorithm that combines multi-view stereo and shading-based refinement for high-quality reconstruction of 3D geometry models from images taken under constant but otherwise arbitrary illumination. We have tested our algorithm on several scenes that were captured under several general and unknown lighting conditions, and we show that our final reconstructions rival laser range scans.

[1]  Tianli Yu,et al.  Shape and View Independent Reflectance Map from Multiple Views , 2004, ECCV.

[2]  Ping-Sing Tsai,et al.  Shape from Shading: A Survey , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Stefano Soatto,et al.  Stereoscopic shading: integrating multi-frame shape cues in a variational framework , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[4]  Stefano Soatto,et al.  Region-Based Segmentation on Evolving Surfaces with Application to 3D Reconstruction of Shape and Piecewise Constant Radiance , 2004, ECCV.

[5]  P. Hanrahan,et al.  On the relationship between radiance and irradiance: determining the illumination from images of a convex Lambertian object. , 2001, Journal of the Optical Society of America. A, Optics, image science, and vision.

[6]  Qionghai Dai,et al.  Fusing Multiview and Photometric Stereo for 3D Reconstruction under Uncalibrated Illumination , 2011, IEEE Transactions on Visualization and Computer Graphics.

[7]  Pascal Fua,et al.  Variable albedo surface reconstruction from stereo and shape from shading , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[8]  Peter F. Sturm,et al.  Joint Estimation of Shape and Reflectance using Multiple Images with Known Illumination Conditions , 2009, International Journal of Computer Vision.

[9]  David J. Kriegman,et al.  Shape from Varying Illumination and Viewpoint , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[10]  Pascal Fua,et al.  Object-centered surface reconstruction: Combining multi-image stereo and shading , 1995, International Journal of Computer Vision.

[11]  Daniel Cremers,et al.  3-D Reconstruction of Shaded Objects from Multiple Images Under Unknown Illumination , 2008, International Journal of Computer Vision.

[12]  Mubarak Shah,et al.  Integration of shape from shading and stereo , 1995, Pattern Recognit..

[13]  Roberto Cipolla,et al.  Multiview Photometric Stereo , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Diego F. Nehab,et al.  Efficiently combining positions and normals for precise 3D geometry , 2005, SIGGRAPH 2005.

[15]  Aaron F. Bobick,et al.  The direct computation of height from shading , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[16]  James T. Kajiya,et al.  The rendering equation , 1998 .

[17]  Hans-Peter Seidel,et al.  Relighting objects from image collections , 2009, CVPR 2009.

[18]  Jan Kautz,et al.  Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments , 2002 .

[19]  Tim Weyrich,et al.  Capturing Time-of-Flight data with confidence , 2011, CVPR 2011.

[20]  Thabo Beeler,et al.  High-quality single-shot capture of facial geometry , 2010, ACM Trans. Graph..

[21]  Ronen Basri,et al.  Lambertian Reflectance and Linear Subspaces , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Andrew Blake,et al.  Surface descriptions from stereo and shading , 1986, Image Vis. Comput..

[23]  Ronen Basri,et al.  Photometric stereo with general, unknown lighting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[24]  Tianli Yu,et al.  Recovering shape and reflectance model of non-lambertian objects from multiple views , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[25]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[27]  Qionghai Dai,et al.  A Point-Cloud-Based Multiview Stereo Algorithm for Free-Viewpoint Video , 2010, IEEE Transactions on Visualization and Computer Graphics.

[28]  Mark Meyer,et al.  Discrete Differential-Geometry Operators for Triangulated 2-Manifolds , 2002, VisMath.

[29]  Daniel Cremers,et al.  Shedding light on stereoscopic segmentation , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[30]  H. Bülthoff,et al.  Depth Discrimination from Shading under Diffuse Lighting , 2000, Perception.

[31]  Ira Kemelmacher-Shlizerman,et al.  Photometric Stereo with General, Unknown Lighting , 2006, International Journal of Computer Vision.

[32]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[33]  Ronen Basri,et al.  Accuracy of Spherical Harmonic Approximations for Images of Lambertian Objects under Far and Near Lighting , 2004, ECCV.