Shape and spatially-varying BRDFs from photometric stereo

This paper describes a photometric stereo method designed for surfaces with spatially-varying BRDFs, including surfaces with both varying diffuse and specular properties. Our optimization-based method builds on the observation that most objects are composed of a small number of fundamental materials by constraining each pixel to be representable by a combination of at most two such materials. This approach recovers not only the shape but also material BRDFs and weight maps, yielding accurate rerenderings under novel lighting conditions for a wide variety of objects. We demonstrate examples of interactive editing operations made possible by our approach.

[1]  Peter H. Tu,et al.  Surface reconstruction via Helmholtz reciprocity with a single image pair , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[2]  Robert L. Cook,et al.  Shade trees , 1984, SIGGRAPH.

[3]  Thomas Ertl,et al.  Computer Graphics - Principles and Practice, 3rd Edition , 2014 .

[4]  Fausto Bernardini,et al.  Cut-and-paste editing of multiresolution surfaces , 2002, SIGGRAPH.

[5]  Steven M. Seitz,et al.  Shape and materials by example: a photometric stereo approach , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[6]  Steve Marschner,et al.  Image-Based BRDF Measurement Including Human Skin , 1999, Rendering Techniques.

[7]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[8]  Andrew W. Fitzgibbon,et al.  BRDF and geometry capture from extended inhomogeneous samples using flash photography , 2005, Comput. Graph. Forum.

[9]  Tianli Yu,et al.  Recovering shape and reflectance model of non-lambertian objects from multiple views , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[10]  Stefano Soatto,et al.  Multi-view stereo beyond Lambert , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[11]  Yair Weiss,et al.  Deriving intrinsic images from image sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[12]  Wojciech Matusik,et al.  Acquisition and Rendering of Transparent and Refractive Objects , 2002, Rendering Techniques.

[13]  Thomas Malzbender,et al.  Polynomial texture maps , 2001, SIGGRAPH.

[14]  Wojciech Matusik,et al.  A data-driven reflectance model , 2003, ACM Trans. Graph..

[15]  David J. Kriegman,et al.  Beyond Lambert: reconstructing specular surfaces using color , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[16]  Jitendra Malik,et al.  Color- and texture-based image segmentation using EM and its application to content-based image retrieval , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[17]  Wojciech Matusik,et al.  Inverse shade trees for non-parametric material representation and editing , 2006, ACM Trans. Graph..

[18]  Athinodoros S. Georghiades,et al.  Recovering 3-D Shape and Reflectance From a Small Number of Photographs , 2003, Rendering Techniques.

[19]  Pat Hanrahan,et al.  A signal-processing framework for inverse rendering , 2001, SIGGRAPH.

[20]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[21]  W. Press,et al.  Numerical Recipes in C++: The Art of Scientific Computing (2nd edn)1 Numerical Recipes Example Book (C++) (2nd edn)2 Numerical Recipes Multi-Language Code CD ROM with LINUX or UNIX Single-Screen License Revised Version3 , 2003 .

[22]  Gudrun Klinker,et al.  A physical approach to color image understanding , 1989, International Journal of Computer Vision.

[23]  Katsushi Ikeuchi,et al.  Object shape and reflectance modeling from observation , 1997, SIGGRAPH.

[24]  Emanuele Trucco,et al.  Introductory techniques for 3-D computer vision , 1998 .

[25]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[26]  Takeo Kanade,et al.  Determining shape and reflectance of hybrid surfaces by photometric sampling , 1989, IEEE Trans. Robotics Autom..

[27]  Adrien Treuille,et al.  Example-Based Stereo with General BRDFs , 2004, ECCV.

[28]  Roberto Cipolla,et al.  Reconstruction in the Round Using Photometric Normals and Silhouettes. , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[29]  Stefano Soatto,et al.  Region-based segmenta-tion on evolving surfaces with application to 3D shape and radiance estimation , 2004, eccv 2004.

[30]  Greg Turk,et al.  Texture synthesis on surfaces , 2001, SIGGRAPH.

[31]  Michael D. McCool,et al.  Fast Extraction of BRDFs and Material Maps from Images , 2003, Graphics Interface.

[32]  Stefano Soatto,et al.  Tales of shape and radiance in multiview stereo , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[33]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[34]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[35]  Tatsuya Yamazaki.  Introduction of EM Algorithm into Color Image Segmentation , 1998 .

[36]  Steven M. Seitz,et al.  Example-based photometric stereo: shape reconstruction with general, varying BRDFs , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Andrew Gardner,et al.  Linear light source reflectometry , 2003, ACM Trans. Graph..

[38]  Rui J. P. de Figueiredo,et al.  A Theory of Photometric Stereo for a Class of Diffuse Non-Lambertian Surfaces , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[39]  Glenn Healey,et al.  Segmenting images using normalized color , 1992, IEEE Trans. Syst. Man Cybern..

[40]  Pat Hanrahan,et al.  Conveying shape and features with image-based relighting , 2003, IEEE Visualization, 2003. VIS 2003..

[41]  Henning Biermann,et al.  Texture and Shape Synthesis on Surfaces , 2001, Rendering Techniques.

[42]  David Salesin,et al.  Surface light fields for 3D photography , 2000, SIGGRAPH.

[43]  Hans-Peter Seidel,et al.  Image-Based Reconstruction of Spatially Varying Materials , 2001 .

[44]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[45]  Gregory J. Ward,et al.  Measuring and modeling anisotropic reflection , 1992, SIGGRAPH.

[46]  Ping-Sing Tsai,et al.  Shape from Shading: A Survey , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[47]  Hans-Peter Seidel,et al.  Image-based reconstruction of spatial appearance and geometric detail , 2003, TOGS.

[48]  Baining Guo,et al.  Synthesis of bidirectional texture functions on arbitrary surfaces , 2002, SIGGRAPH.

[49]  Wojciech Matusik,et al.  Efficient Isotropic BRDF Measurement , 2003, Rendering Techniques.

[50]  Paul Debevec,et al.  Inverse global illumination: Recovering re?ectance models of real scenes from photographs , 1998 .

[51]  Steven M. Seitz,et al.  Shape and Spatially-Varying BRDFs from Photometric Stereo , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  David Salesin,et al.  Environment matting extensions: towards higher accuracy and real-time capture , 2000, SIGGRAPH.

[53]  Marc Levoy,et al.  Texture synthesis over arbitrary manifold surfaces , 2001, SIGGRAPH.

[54]  Jean Ponce,et al.  Computer Vision: A Modern Approach , 2002 .

[55]  R. Bajcsy,et al.  Color image segmentation with detection of highlights and local illumination induced by inter-reflections , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[56]  Berthold K. P. Horn,et al.  Determining Shape and Reflectance Using Multiple Images , 1978 .