A Benchmark Dataset and Evaluation for Non-Lambertian and Uncalibrated Photometric Stereo

Recent progress on photometric stereo extends the technique to deal with general materials and unknown illumination conditions. However, due to the lack of suitable benchmark data with ground truth shapes (normals), quantitative comparison and evaluation is difficult to achieve. In this paper, we first survey and categorize existing methods using a photometric stereo taxonomy emphasizing on non-Lambertian and uncalibrated methods. We then introduce the 'DiLiGenT' photometric stereo image dataset with calibrated Directional Lightings, objects of General reflectance, and 'ground Truth' shapes (normals). Based on our dataset, we quantitatively evaluate state-of-the-art photometric stereo methods for general non-Lambertian materials and unknown lightings to analyze their strengths and limitations.

[1]  K. S. Pedersen,et al.  A Comparative Study of Interest Point Performance on a Unique Data Set , 2011 .

[2]  E. North Coleman,et al.  Obtaining 3-dimensional shape of textured and specular surfaces using four-source photometry , 1982, Comput. Graph. Image Process..

[3]  Alessio Del Bue,et al.  Multi-view Photometric Stereo Using Semi-isometric Mappings , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[4]  Steven M. Seitz,et al.  Shape and spatially-varying BRDFs from photometric stereo , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[5]  David J. Kriegman,et al.  Resolving the Generalized Bas-Relief Ambiguity by Entropy Minimization , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Katsushi Ikeuchi,et al.  Elevation Angle from Reflectance Monotonicity: Photometric Stereo for General Isotropic Reflectances , 2012, ECCV.

[7]  Yasuyuki Matsushita,et al.  Self-calibrating photometric stereo , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Katsushi Ikeuchi,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence Bi-polynomial Modeling of Low-frequency Reflectances , 2022 .

[9]  Michael Goesele,et al.  A Survey of Photometric Stereo Techniques , 2015, Found. Trends Comput. Graph. Vis..

[10]  Ronen Basri,et al.  Dense shape reconstruction of a moving object under arbitrary, unknown lighting , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[11]  Stephen Lin,et al.  Enhanced Photometric Stereo with Multispectral Images , 2013, MVA.

[12]  Pieter Peers,et al.  Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination , 2007 .

[13]  Takahiro Okabe,et al.  Shape Reconstruction Based on Similarity in Radiance Changes under Varying Illumination , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[14]  Todd E. Zickler,et al.  A projective framework for radiometric image analysis , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Yuji Iwahori,et al.  Reconstructing shape from shading images under point light source illumination , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[16]  Hideki Hayakawa.  Photometric stereo under a light source with arbitrary motion , 1994 .

[17]  Kiyoharu Aizawa,et al.  Robust photometric stereo using sparse regression , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Takahiro Okabe,et al.  Attached shadow coding: Estimating surface normals from shadows under unknown reflectance and lighting conditions , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[19]  Charlie C. L. Wang,et al.  Photometric stereo with near point lighting: A solution by mesh deformation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Jason Lawrence,et al.  A photometric approach for estimating normals and tangents , 2008, ACM Trans. Graph..

[21]  David J. Kriegman,et al.  Shape from Varying Illumination and Viewpoint , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[22]  Katsushi Ikeuchi,et al.  Temporal-color space analysis of reflection , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Yasushi Yagi,et al.  Surface Normal Deconvolution: Photometric Stereo for Optically Thick Translucent Objects , 2014, ECCV.

[24]  Ping Tan,et al.  Photometric stereo and weather estimation using internet images , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Jiaya Jia,et al.  Efficient photometric stereo on glossy surfaces with wide specular lobes , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Paolo Favaro,et al.  A New Perspective on Uncalibrated Photometric Stereo , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Ping Tan,et al.  Ring-Light Photometric Stereo , 2010, ECCV.

[28]  Simon Fuhrmann,et al.  Multi-view Photometric Stereo by Example , 2014, 2014 2nd International Conference on 3D Vision.

[29]  Yannick Hold-Geoffroy,et al.  What Is a Good Day for Outdoor Photometric Stereo? , 2015, 2015 IEEE International Conference on Computational Photography (ICCP).

[30]  Takahiro Okabe,et al.  Bispectral photometric stereo based on fluorescence , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  David J. Kriegman,et al.  Passive photometric stereo from motion , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[32]  Katsushi Ikeuchi,et al.  Photometric Stereo Using Internet Images , 2014, 2014 2nd International Conference on 3D Vision.

[33]  Kiyoharu Aizawa,et al.  Photometric Stereo Using Constrained Bivariate Regression for General Isotropic Surfaces , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Grant Schindler,et al.  Photometric Stereo via Computer Screen Lighting for Real-time Surface Reconstruction , 2008 .

[35]  Ondrej Drbohlav,et al.  Specularities Reduce Ambiguity of Uncalibrated Photometric Stereo , 2002, ECCV.

[36]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[37]  Paolo Favaro,et al.  Uncalibrated Near-Light Photometric Stereo , 2014, BMVC.

[38]  Björn Stenger,et al.  Non-rigid Photometric Stereo with Colored Lights , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[39]  Stephen Lin,et al.  Subpixel Photometric Stereo , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Simon Fuhrmann,et al.  Photometric stereo for outdoor webcams , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Henrik Aanæs,et al.  Large Scale Multi-view Stereopsis Evaluation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Chu-Song Chen,et al.  The 4-Source Photometric Stereo Under General Unknown Lighting , 2006, ECCV.

[43]  Roberto Cipolla,et al.  Multiview Photometric Stereo , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  Rin-ichiro Taniguchi,et al.  Shape and light directions from shading and polarization , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Takeshi Shakunaga,et al.  Analysis of photometric factors based on photometric linearization. , 2007, Journal of the Optical Society of America. A, Optics, image science, and vision.

[46]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[47]  Yasuyuki Matsushita,et al.  Multiview Photometric Stereo Using Planar Mesh Parameterization , 2013, 2013 IEEE International Conference on Computer Vision.

[48]  Katsushi Ikeuchi,et al.  Median Photometric Stereo as Applied to the Segonko Tumulus and Museum Objects , 2009, International Journal of Computer Vision.

[49]  David J. Kriegman,et al.  Color Subspaces as Photometric Invariants , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[50]  Aswin C. Sankaranarayanan,et al.  A Dictionary-Based Approach for Estimating Shape and Spatially-Varying Reflectance , 2015, 2015 IEEE International Conference on Computational Photography (ICCP).

[51]  Alfred M. Bruckstein,et al.  A Direct Differential Approach to Photometric Stereo with Perspective Viewing , 2014, SIAM J. Imaging Sci..

[52]  Robert Pless,et al.  Heliometric Stereo: Shape from Sun Position , 2012, ECCV.

[53]  Thomas Malzbender,et al.  Surface enhancement using real-time photometric stereo and reflectance transformation , 2006, EGSR '06.

[54]  Moshe Ben-Ezra,et al.  Photometric Stereo for Dynamic Surface Orientations , 2010, ECCV.

[55]  Imari Sato,et al.  A Microfacet-Based Reflectance Model for Photometric Stereo with Highly Specular Surfaces , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[56]  Katsushi Ikeuchi,et al.  Photometric stereo under unknown light sources using robust SVD with missing data , 2010, 2010 IEEE International Conference on Image Processing.

[57]  Ravi Ramamoorthi,et al.  On Differential Photometric Reconstruction for Unknown, Isotropic BRDFs , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Tien-Tsin Wong,et al.  Dense Photometric Stereo: A Markov Random Field Approach , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Katsushi Ikeuchi,et al.  Extracting the Shape and Roughness of Specular Lobe Objects Using Four Light Photometric Stereo , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[60]  In-So Kweon,et al.  One-day outdoor photometric stereo via skylight estimation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[61]  Edward H. Adelson,et al.  Shape estimation in natural illumination , 2011, CVPR 2011.

[62]  Christian Wöhler,et al.  An introduction to image-based 3D surface reconstruction and a survey of photometric stereo methods , 2011 .

[63]  Li Zhang,et al.  Shape and motion under varying illumination: unifying structure from motion, photometric stereo, and multiview stereo , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[64]  Ko Nishino,et al.  Multiview Shape and Reflectance from Natural Illumination , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[65]  Alessio Del Bue,et al.  Direct Differential Photometric Stereo Shape Recovery of Diffuse and Specular Surfaces , 2016, Journal of Mathematical Imaging and Vision.

[66]  Ramesh Raskar,et al.  Fast separation of direct and global components of a scene using high frequency illumination , 2006, SIGGRAPH 2006.

[67]  Wojciech Matusik,et al.  A data-driven reflectance model , 2003, ACM Trans. Graph..

[68]  Michael Goesele,et al.  Removing the Example from Example-Based Photometric Stereo , 2010, ECCV Workshops.

[69]  Todd E. Zickler,et al.  Passive Reflectometry , 2008, ECCV.

[70]  Pieter Peers,et al.  Dynamic shape capture using multi-view photometric stereo , 2009, ACM Trans. Graph..

[71]  Zhe Wu,et al.  Calibrating Photometric Stereo by Holistic Reflectance Symmetry Analysis , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[72]  Qing Zhang,et al.  Edge-preserving photometric stereo via depth fusion , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[73]  Daniel Snow,et al.  Shape and albedo from multiple images using integrability , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[74]  Roberto Cipolla,et al.  A Single-Lobe Photometric Stereo Approach for Heterogeneous Material , 2016, SIAM J. Imaging Sci..

[75]  Steven M. Seitz,et al.  Binocular Photometric Stereo , 2011, BMVC.

[76]  Paul E. Debevec,et al.  Multiview face capture using polarized spherical gradient illumination , 2011, ACM Trans. Graph..

[77]  Diego F. Nehab,et al.  Efficiently combining positions and normals for precise 3D geometry , 2005, SIGGRAPH 2005.

[78]  Yoichi Sato,et al.  Uncalibrated photometric stereo based on elevation angle recovery from BRDF symmetry of isotropic materials , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[79]  Björn Stenger,et al.  Color photometric stereo for multicolored surfaces , 2011, 2011 International Conference on Computer Vision.

[80]  David J. Kriegman,et al.  Isotropy, Reciprocity and the Generalized Bas-Relief Ambiguity , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[81]  Tai-Pang Wu,et al.  Photometric Stereo via Expectation Maximization , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[82]  Sang Wook Lee,et al.  Photometric Stereo from Maximum Feasible Lambertian Reflections , 2010, ECCV.

[83]  David J. Kriegman,et al.  Shape from Fluorescence , 2012, ECCV.

[84]  Yizhou Wang,et al.  What Object Motion Reveals about Shape with Unknown BRDF and Lighting , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[85]  Nahum Kiryati,et al.  Photometric stereo under perspective projection , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[86]  Roberto Scopigno,et al.  Image‐to‐Geometry Registration: a Mutual Information Method exploiting Illumination‐related Geometric Properties , 2009, Comput. Graph. Forum.

[87]  Henrik Aanæs,et al.  Interesting Interest Points , 2011, International Journal of Computer Vision.

[88]  Paul Graham,et al.  Acquiring reflectance and shape from continuous spherical harmonic illumination , 2013, ACM Trans. Graph..

[89]  Manmohan Krishna Chandraker,et al.  What Camera Motion Reveals about Shape with Unknown BRDF , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[90]  Maria E. Angelopoulou,et al.  Backscatter Compensated Photometric Stereo with 3 Sources , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[91]  Takahiro Okabe,et al.  Uncalibrated Photometric Stereo for Unknown Isotropic Reflectances , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[92]  Michael S. Brown,et al.  A 3D Imaging Framework Based on High-Resolution Photometric-Stereo and Low-Resolution Depth , 2013, International Journal of Computer Vision.

[93]  Ira Kemelmacher-Shlizerman,et al.  Photometric Stereo with General, Unknown Lighting , 2006, International Journal of Computer Vision.

[94]  Maria Petrou,et al.  The 4-Source Photometric Stereo Technique for Three-Dimensional Surfaces in the Presence of Highlights and Shadows , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[95]  Tony F. Chan,et al.  Outdoor photometric stereo , 2013, IEEE International Conference on Computational Photography (ICCP).

[96]  Takayuki Okatani,et al.  Optimal integration of photometric and geometric surface measurements using inaccurate reflectance/illumination knowledge , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[97]  Yongtian Wang,et al.  Robust Photometric Stereo via Low-Rank Matrix Completion and Recovery , 2010, ACCV.

[98]  Anders Bjorholm Dahl,et al.  Large-Scale Data for Multiple-View Stereopsis , 2016, International Journal of Computer Vision.

[99]  Athinodoros S. Georghiades,et al.  Incorporating the Torrance and Sparrow model of reflectance in uncalibrated photometric stereo , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[100]  James J. Clark.  Photometric stereo using LCD displays , 2010, Image Vis. Comput..

[101]  Paolo Favaro,et al.  A Closed-Form, Consistent and Robust Solution to Uncalibrated Photometric Stereo Via Local Diffuse Reflectance Maxima , 2013, International Journal of Computer Vision.

[102]  Venu Madhav Govindu,et al.  High Quality Photometric Reconstruction Using a Depth Camera , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[103]  Steven M. Seitz,et al.  Example-based photometric stereo: shape reconstruction with general, varying BRDFs , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[104]  Yannick Hold-Geoffroy,et al.  x-Hour Outdoor Photometric Stereo , 2015, 2015 International Conference on 3D Vision.

[105]  Boxin Shi,et al.  Photometric stereo for general isotropic reflectances by spherical linear interpolation , 2015 .

[106]  Alessio Del Bue,et al.  Bilinear Factorization via Augmented Lagrange Multipliers , 2010, ECCV.

[107]  Mike J. Chantler,et al.  Can two specular pixels calibrate photometric stereo? , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[108]  Tai-Pang Wu,et al.  Normal Estimation of a Transparent Object Using a Video , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[109]  David J. Kriegman,et al.  Toward Reconstructing Surfaces With Arbitrary Isotropic Reflectance : A Stratified Photometric Stereo Approach , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[110]  David J. Kriegman,et al.  ShadowCuts: Photometric Stereo with Shadows , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[111]  Paul E. Debevec,et al.  Single-shot photometric stereo by spectral multiplexing , 2011, ICCP.

[112]  Hanspeter Pfister,et al.  Visibility Subspaces: Uncalibrated Photometric Stereo with Shadows , 2010, ECCV.

[113]  Ping Tan,et al.  A Benchmark Dataset and Evaluation for Non-Lambertian and Uncalibrated Photometric Stereo , 2016, CVPR.

[114]  Bo Dong,et al.  Scattering Parameters and Surface Normals from Homogeneous Translucent Materials Using Photometric Stereo , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[115]  David J. Kriegman,et al.  Reflections on the generalized bas-relief ambiguity , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[116]  Ronen Basri,et al.  A two-frame theory of motion, lighting and shape , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[117]  Edward H. Adelson,et al.  Microgeometry capture using an elastomeric sensor , 2011, ACM Trans. Graph..

[118]  Venu Madhav Govindu,et al.  Photometric refinement of depth maps for multi-albedo objects , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[119]  David J. Kriegman,et al.  What Shadows Reveal about Object Structure , 1998, ECCV.

[120]  Luc Van Gool,et al.  Photometric stereo with coherent outlier handling and confidence estimation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[121]  Takeo Kanade,et al.  Determining shape and reflectance of hybrid surfaces by photometric sampling , 1989, IEEE Trans. Robotics Autom..

[122]  Yasuyuki Matsushita,et al.  A hand-held photometric stereo camera for 3-D modeling , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[123]  David J. Kriegman,et al.  Photometric stereo with non-parametric and spatially-varying reflectance , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[124]  Katsushi Ikeuchi,et al.  Determining a Depth Map Using a Dual Photometric Stereo , 1987 .

[125]  Long Quan,et al.  The Geometry of Reflectance Symmetries , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[126]  Stefan Roth,et al.  Discriminative shape from shading in uncalibrated illumination , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[127]  David J. Kriegman,et al.  The Bas-Relief Ambiguity , 2004, International Journal of Computer Vision.

[128]  James J. Clark.  Active photometric stereo , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[129]  Jian Wang,et al.  Photometric Stereo with Small Angular Variations , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[130]  Zhe Wu,et al.  Multi-view Photometric Stereo with Spatially Varying Isotropic Materials , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[131]  Yasuyuki Matsushita,et al.  Photometric Stereo in the Wild , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[132]  Katsushi Ikeuchi,et al.  Consensus photometric stereo , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[133]  Hans-Peter Seidel,et al.  Mesostructure from Specularity , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).