Stereoscopic Flash and No-Flash Photography for Shape and Albedo Recovery

We present a minimal imaging setup that harnesses both geometric and photometric approaches for shape and albedo recovery. We adopt a stereo camera and a flashlight to capture a stereo image pair and a flash/no-flash pair. From the stereo image pair, we recover a rough shape that captures low-frequency shape variation without high-frequency details. From the flash/no-flash pair, we derive an image formation model for Lambertian objects under natural lighting, based on which a fine normal map is obtained and fused with the rough shape. Further, we use the flash/no-flash pair for cast shadow detection and albedo canceling, making the shape recovery robust against shadows and albedo variation. We verify the effectiveness of our approach on both synthetic and real-world data.

[1]  Harry Shum,et al.  Flash Cut: Foreground Extraction with Flash and No-flash Image Pairs , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Edward H. Adelson,et al.  Shape estimation in natural illumination , 2011, CVPR 2011.

[3]  Jaakko Lehtinen,et al.  Reflectance modeling by neural texture synthesis , 2016, ACM Trans. Graph..

[4]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Alfred M. Bruckstein,et al.  RGBD-fusion: Real-time high precision depth recovery , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Shahram Izadi,et al.  Real-time shading-based refinement for consumer depth cameras , 2014, ACM Trans. Graph..

[7]  Ruigang Yang,et al.  Detailed Surface Geometry and Albedo Recovery from RGB-D Video under Natural Illumination , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[8]  Matthias Nießner,et al.  Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[9]  Qing Zhang,et al.  Edge-preserving photometric stereo via depth fusion , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Venu Madhav Govindu,et al.  High Quality Photometric Reconstruction Using a Depth Camera , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Ronen Basri,et al.  Lambertian reflectance and linear subspaces , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[12]  Daniel Cremers,et al.  A Variational Approach to Shape-from-Shading Under Natural Illumination , 2017, EMMCVPR.

[13]  Hans-Peter Seidel,et al.  Shading-based dynamic shape refinement from multi-view video under general illumination , 2011, 2011 International Conference on Computer Vision.

[14]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[15]  Adrien Bartoli,et al.  Dense Non-rigid Structure-from-Motion and Shading with Unknown Albedos , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[16]  Jaakko Lehtinen,et al.  Two-shot SVBRDF capture for stationary materials , 2015, ACM Trans. Graph..

[17]  Kari Pulli,et al.  Robust stereo with flash and no-flash image pairs , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Stephen Lin,et al.  Shading-Based Shape Refinement of RGB-D Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Harry Shum,et al.  Flash matting , 2006, ACM Trans. Graph..

[20]  Carl Olsson,et al.  Combining Depth Fusion and Photometric Stereo for Fine-Detailed 3D Models , 2019, SCIA.

[21]  In-So Kweon,et al.  High Quality Shape from a Single RGB-D Image under Uncalibrated Natural Illumination , 2013, 2013 IEEE International Conference on Computer Vision.

[22]  Kalyan Sunkavalli,et al.  Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image , 2018, ECCV.

[23]  Yasuyuki Matsushita,et al.  High-quality shape from multi-view stereo and shading under general illumination , 2011, CVPR 2011.

[24]  Michael Goesele,et al.  Modulation transfer function of patch-based stereo systems , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  F. Durand,et al.  Flash photography enhancement via intrinsic relighting , 2004, ACM Trans. Graph..

[26]  Kalyan Sunkavalli,et al.  Learning to reconstruct shape and spatially-varying reflectance from a single image , 2018, ACM Trans. Graph..

[27]  Michael Breuß,et al.  Combining Shape from Shading and Stereo: A Joint Variational Method for Estimating Depth, Illumination and Albedo , 2018, International Journal of Computer Vision.

[28]  Lizhen Wang,et al.  DDRNet: Depth Map Denoising and Refinement for Consumer Depth Cameras Using Cascaded CNNs , 2018, ECCV.

[29]  Tony DeRose,et al.  Surface reconstruction from unorganized points , 1992, SIGGRAPH.

[30]  Szymon Rusinkiewicz,et al.  Efficiently combining positions and normals for precise 3D geometry , 2005, ACM Trans. Graph..

[31]  Adrien Bousseau,et al.  Single-image SVBRDF capture with a rendering-aware deep network , 2018, ACM Trans. Graph..

[32]  Ramesh Raskar,et al.  Discontinuity preserving stereo with small baseline multi-flash illumination , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[33]  Ira Kemelmacher-Shlizerman,et al.  Photometric Stereo with General, Unknown Lighting , 2006, International Journal of Computer Vision.

[34]  Matthias Nießner,et al.  Shading-based refinement on volumetric signed distance functions , 2015, ACM Trans. Graph..

[35]  In-So Kweon,et al.  Exploiting Shading Cues in Kinect IR Images for Geometry Refinement , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Rynson W. H. Lau,et al.  Saliency Detection with Flash and No-flash Image Pairs , 2014, ECCV.

[37]  Michael F. Cohen,et al.  Digital photography with flash and no-flash image pairs , 2004, ACM Trans. Graph..

[38]  Berthold K. P. Horn.  SHAPE FROM SHADING: A METHOD FOR OBTAINING THE SHAPE OF A SMOOTH OPAQUE OBJECT FROM ONE VIEW , 1970 .