Depth Estimation for Glossy Surfaces with Light-Field Cameras

Light-field cameras have now become available in both consumer and industrial applications, and recent papers have demonstrated practical algorithms for depth recovery from a passive single-shot capture. However, current light-field depth estimation methods are designed for Lambertian objects and fail or degrade for glossy or specular surfaces. Because light-field cameras have an array of micro-lenses, the captured data allows modification of both focus and perspective viewpoints. In this paper, we develop an iterative approach to use the benefits of light-field data to estimate and remove the specular component, improving the depth estimation. The approach enables light-field data depth estimation to support both specular and diffuse scenes. We present a physically-based method that estimates one or multiple light source colors. We show our method outperforms current state-of-the-art diffuse and specular separation and depth estimation algorithms in multiple real world scenarios.

[1]  Stefano Soatto,et al.  Multi-view stereo beyond Lambert , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[2]  Jae Byung Park Efficient color representation for image segmentation under nonwhite illumination , 2003, SPIE Optics East.

[3]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[4]  Michael J. Black,et al.  Secrets of optical flow estimation and their principles , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[5]  Steven A. Shafer,et al.  Depth from focusing and defocusing , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Sven Wanner,et al.  Globally consistent depth labeling of 4D light fields , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Stephen Lin,et al.  Separation of Highlight Reflections on Textured Surfaces , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[8]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[9]  Yael Pritch,et al.  Scene reconstruction from high spatio-angular resolution light fields , 2013, ACM Trans. Graph..

[10]  K. Ikeuchi,et al.  Color constancy through inverse-intensity chromaticity space. , 2004, Journal of the Optical Society of America. A, Optics, image science, and vision.

[11]  Katsushi Ikeuchi,et al.  Separating Reflection Components of Textured Surfaces Using a Single Image , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  In-So Kweon,et al.  Fast Separation of Reflection Components using a Specularity-Invariant Image Representation , 2006, 2006 International Conference on Image Processing.

[13]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[14]  Minh N. Do,et al.  Joint Histogram-Based Cost Aggregation for Stereo Matching , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[16]  Sang Wook Lee,et al.  Detection of diffuse and specular interface reflections and inter-reflections by color image segmentation , 1996, International Journal of Computer Vision.

[17]  David J. Kriegman,et al.  Beyond Lambert: reconstructing specular surfaces using color , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[18]  Katsushi Ikeuchi,et al.  Determining Reflectance Parameters and Illumination Distribution from a Sparse Set of Images for View-dependent Image Synthesis , 2001, ICCV.

[19]  Dmitry Chetverikov,et al.  A Survey of Specularity Removal Methods , 2011, Comput. Graph. Forum.

[20]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[21]  Katsushi Ikeuchi,et al.  Temporal-color space analysis of reflection , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Jonathan T. Barron,et al.  A category-level 3-D object dataset: Putting the Kinect to work , 2011, ICCV Workshops.

[23]  In-So Kweon,et al.  Specular Reflection Separation Using Dark Channel Prior , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Shree K. Nayar,et al.  Rational Filters for Passive Depth from Defocus , 1998, International Journal of Computer Vision.

[25]  Katsushi Ikeuchi,et al.  Temporal-color space analysis of reflection , 1994 .

[26]  Steven A. Shafer,et al.  Using color to separate reflection components , 1985 .

[27]  Narendra Ahuja,et al.  Real-Time Specular Highlight Removal Using Bilateral Filtering , 2010, ECCV.

[28]  Greg Humphreys,et al.  Physically Based Rendering: From Theory to Implementation , 2004 .

[29]  Jitendra Malik,et al.  Depth from Combining Defocus and Correspondence Using Light-Field Cameras , 2013, 2013 IEEE International Conference on Computer Vision.

[30]  David J. Kriegman,et al.  Specularity Removal in Images and Videos: A PDE Approach , 2006, ECCV.

[31]  Stephen Lin,et al.  Diffuse-Specular Separation and Depth Recovery from Image Sequences , 2002, ECCV.

[32]  Gerald Schaefer,et al.  Solving for Colour Constancy using a Constrained Dichromatic Reflection Model , 2001, International Journal of Computer Vision.