Semi-Calibrated Photometric Stereo

While conventional calibrated photometric stereo methods assume that light intensities and sensor exposures are known or unknown but identical across observed images, this assumption easily breaks down in practical settings due to individual light bulb's characteristics and limited control over sensors. This paper studies the effect of unknown and possibly non-uniform light intensities and sensor exposures among observed images on the shape recovery based on photometric stereo. This leads to the development of a “semi-calibrated” photometric stereo method, where the light directions are known but light intensities (and sensor exposures) are unknown. We show that the semi-calibrated photometric stereo becomes a bilinear problem, whose general form is difficult to solve, but in the photometric stereo context, there exists a unique solution for the surface normal and light intensities (or sensor exposures). We further show that there exists a linear solution method for the problem, and develop efficient and stable solution methods. The semi-calibrated photometric stereo is advantageous over conventional calibrated photometric stereo in accurate determination of surface normal, because it relaxes the assumption of known light intensity ratios/sensor exposures. The experimental results show superior accuracy of the semi-calibrated photometric stereo in comparison to conventional methods in practical settings.

[1]  Robert Pless,et al.  Heliometric Stereo: Shape from Sun Position , 2012, ECCV.

[2]  Thomas Malzbender,et al.  Surface enhancement using real-time photometric stereo and reflectance transformation , 2006, EGSR '06.

[3]  Dmitry B. Goldgof,et al.  A Simple Strategy for Calibrating the Geometry of Light Sources , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Hui-Liang Shen,et al.  Calibrating light sources by using a planar mirror , 2011, J. Electronic Imaging.

[5]  Zhe Wu,et al.  Calibrating Photometric Stereo by Holistic Reflectance Symmetry Analysis , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Kwan-Yee Kenneth Wong,et al.  Camera and light calibration from reflections on a sphere , 2013, Comput. Vis. Image Underst..

[7]  Ronen Basri,et al.  Dense shape reconstruction of a moving object under arbitrary, unknown lighting , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[8]  Daniel Cremers,et al.  LED-Based Photometric Stereo: Modeling, Calibration and Numerical Solution , 2017, Journal of Mathematical Imaging and Vision.

[9]  Hideki Hayakawa.  Photometric stereo under a light source with arbitrary motion , 1994 .

[10]  Paolo Favaro,et al.  Uncalibrated Near-Light Photometric Stereo , 2014, BMVC.

[11]  Simon Fuhrmann,et al.  Photometric stereo for outdoor webcams , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  R. Woodham,et al.  Neural network to reconstruct specular surface shape from its three shading images , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[13]  Tai-Pang Wu,et al.  Photometric Stereo via Expectation Maximization , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Yasuyuki Matsushita,et al.  Calibrating a Non-isotropic Near Point Light Source Using a Plane , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Phillipp Kaestner,et al.  Linear And Nonlinear Programming , 2016 .

[16]  Tony F. Chan,et al.  Outdoor photometric stereo , 2013, IEEE International Conference on Computational Photography (ICCP).

[17]  John Wright,et al.  Toward Guaranteed Illumination Models for Non-convex Objects , 2013, 2013 IEEE International Conference on Computer Vision.

[18]  Yongtian Wang,et al.  Robust Photometric Stereo via Low-Rank Matrix Completion and Recovery , 2010, ACCV.

[19]  David J. Kriegman,et al.  Isotropy, Reciprocity and the Generalized Bas-Relief Ambiguity , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Behrooz Kamgar-Parsi,et al.  Evaluation of quantization error in computer vision , 1988, Proceedings CVPR '88: The Computer Society Conference on Computer Vision and Pattern Recognition.

[21]  David J. Kriegman,et al.  What shadows reveal about object structure. , 2001 .

[22]  Yuji Iwahori,et al.  Reconstructing shape from shading images under point light source illumination , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[23]  James C. Bezdek,et al.  Convergence of Alternating Optimization , 2003, Neural Parallel Sci. Comput..

[24]  Kwan-Yee Kenneth Wong,et al.  Recovering Light Directions and Camera Poses from a Single Sphere , 2008, ECCV.

[25]  Daniel Cremers,et al.  A Non-convex Variational Approach to Photometric Stereo under Inaccurate Lighting , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Paolo Favaro,et al.  A New Perspective on Uncalibrated Photometric Stereo , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Ondrej Drbohlav,et al.  Specularities Reduce Ambiguity of Uncalibrated Photometric Stereo , 2002, ECCV.

[28]  Katsushi Ikeuchi,et al.  Extracting the Shape and Roughness of Specular Lobe Objects Using Four Light Photometric Stereo , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  David J. Kriegman,et al.  Resolving the Generalized Bas-Relief Ambiguity by Entropy Minimization , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Zhe Wu,et al.  A Benchmark Dataset and Evaluation for Non-Lambertian and Uncalibrated Photometric Stereo , 2019, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Yasuyuki Matsushita,et al.  Self-calibrating photometric stereo , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[32]  Katsushi Ikeuchi,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence Bi-polynomial Modeling of Low-frequency Reflectances , 2022 .

[33]  Luc Van Gool,et al.  Photometric stereo with coherent outlier handling and confidence estimation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Charlie C. L. Wang,et al.  Surface-from-Gradients: An Approach Based on Discrete Geometry Processing , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Mike J. Chantler,et al.  Can two specular pixels calibrate photometric stereo? , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[36]  Yasuyuki Matsushita,et al.  Photometric Stereo Under Non-uniform Light Intensities and Exposures , 2016, ECCV.

[37]  Yasuyuki Matsushita,et al.  A hand-held photometric stereo camera for 3-D modeling , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[38]  Charlie C. L. Wang,et al.  Photometric stereo with near point lighting: A solution by mesh deformation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Katsushi Ikeuchi,et al.  Radiometric Calibration by Rank Minimization , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Athinodoros S. Georghiades,et al.  Incorporating the Torrance and Sparrow model of reflectance in uncalibrated photometric stereo , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[41]  In-So Kweon,et al.  Partial Sum Minimization of Singular Values in RPCA for Low-Level Vision , 2013, 2013 IEEE International Conference on Computer Vision.

[42]  Ronen Basri,et al.  Photometric stereo with general, unknown lighting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[43]  Daniel Snow,et al.  Determining Generative Models of Objects Under Varying Illumination: Shape and Albedo from Multiple Images Using SVD and Integrability , 1999, International Journal of Computer Vision.

[44]  Sang Wook Lee,et al.  Estimation of diffuse and specular appearance , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[45]  Daniel Snow,et al.  Shape and albedo from multiple images using integrability , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[46]  David J. Kriegman,et al.  ShadowCuts: Photometric Stereo with Shadows , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[47]  P. Holland,et al.  Robust regression using iteratively reweighted least-squares , 1977 .

[48]  David J. Kriegman,et al.  The Bas-Relief Ambiguity , 2004, International Journal of Computer Vision.

[49]  Robert L. Cook,et al.  A Reflectance Model for Computer Graphics , 1987, TOGS.

[50]  Hanspeter Pfister,et al.  Visibility Subspaces: Uncalibrated Photometric Stereo with Shadows , 2010, ECCV.

[51]  David J. Kriegman,et al.  Reflections on the generalized bas-relief ambiguity , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[52]  Kiyoharu Aizawa,et al.  Photometric Stereo Using Sparse Bayesian Regression for General Diffuse Surfaces , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[53]  Tien-Tsin Wong,et al.  Dense Photometric Stereo: A Markov Random Field Approach , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Berthold K. P. Horn,et al.  Determining Shape and Reflectance Using Multiple Images , 1978 .

[55]  Fumihiko Sakaue,et al.  A new approach of photometric stereo from linear image representation under close lighting , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[56]  Kiyoharu Aizawa,et al.  Photometric Stereo Using Constrained Bivariate Regression for General Isotropic Surfaces , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[57]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[58]  Chu-Song Chen,et al.  The 4-Source Photometric Stereo Under General Unknown Lighting , 2006, ECCV.

[59]  Takeshi Shakunaga,et al.  Analysis of photometric factors based on photometric linearization. , 2007, Journal of the Optical Society of America. A, Optics, image science, and vision.

[60]  Roberto Cipolla,et al.  Semi-Calibrated Near Field Photometric Stereo , 2017, CVPR 2017.

[61]  Daniel Cremers,et al.  Semi-calibrated Near-Light Photometric Stereo , 2017, SSVM.