AppFusion: Interactive Appearance Acquisition Using a Kinect Sensor

We present an interactive material acquisition system for average users to capture the spatially varying appearance of daily objects. While an object is being scanned, our system estimates its appearance on‐the‐fly and provides quick visual feedback. We build the system entirely on low‐end, off‐the‐shelf components: a Kinect sensor, a mirror ball and printed markers. We exploit the Kinect infra‐red emitter/receiver, originally designed for depth computation, as an active hand‐held reflectometer, to segment the object into clusters of similar specular materials and estimate the roughness parameters of BRDFs simultaneously. Next, the diffuse albedo and specular intensity of the spatially varying materials are rapidly computed in an inverse rendering framework, using data from the Kinect RGB camera. We demonstrate captured results of a range of materials, and physically validate our system.

[1]  Pat Hanrahan,et al.  All-frequency shadows using non-linear wavelet lighting approximation , 2003, ACM Trans. Graph..

[2]  Frédo Durand,et al.  Experimental analysis of BRDF models , 2005, EGSR '05.

[3]  Steven M. Seitz,et al.  Shape and materials by example: a photometric stereo approach , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[4]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[5]  Tim Weyrich,et al.  Principles of appearance acquisition and representation , 2007, SIGGRAPH '08.

[6]  Shree K. Nayar,et al.  Reflectance and texture of real-world surfaces , 1999, TOGS.

[7]  Pieter Peers,et al.  Estimating Specular Roughness and Anisotropy from Second Order Spherical Gradient Illumination , 2009, Comput. Graph. Forum.

[8]  Jaakko Lehtinen,et al.  Practical SVBRDF capture in the frequency domain , 2013, ACM Trans. Graph..

[9]  Hans-Peter Seidel,et al.  Relighting objects from image collections , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Roberto Scopigno,et al.  A Statistical Method for SVBRDF Approximation from Video Sequences in General Lighting Conditions , 2012, Comput. Graph. Forum.

[11]  Gregory J. Ward,et al.  Measuring and modeling anisotropic reflection , 1992, SIGGRAPH.

[12]  Daniel Cohen-Or,et al.  Bilateral mesh denoising , 2003 .

[13]  Todd E. Zickler,et al.  Passive Reflectometry , 2008, ECCV.

[14]  Hans-Peter Seidel,et al.  Image-based reconstruction of spatial appearance and geometric detail , 2003, TOGS.

[15]  David J. Kriegman,et al.  Photometric stereo with non-parametric and spatially-varying reflectance , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  John M. Snyder,et al.  Modeling anisotropic surface reflectance with example-based microfacet synthesis , 2008, SIGGRAPH 2008.

[17]  Wojciech Matusik,et al.  Efficient Isotropic BRDF Measurement , 2003, Rendering Techniques.

[18]  M. Gross,et al.  Analysis of human faces using a measurement-based skin reflectance model , 2006, ACM Trans. Graph..

[19]  James T. Kajiya,et al.  The rendering equation , 1986, SIGGRAPH.

[20]  Julie Dorsey,et al.  Digital Modeling of Material Appearance , 2007 .

[21]  In-So Kweon,et al.  Exploiting Shading Cues in Kinect IR Images for Geometry Refinement , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  John M. Snyder,et al.  Manifold bootstrapping for SVBRDF capture , 2010, ACM Trans. Graph..

[23]  Andrew Gardner,et al.  Linear light source reflectometry , 2003, ACM Trans. Graph..

[24]  Hugues Hoppe,et al.  Spherical parametrization and remeshing , 2003, ACM Trans. Graph..

[25]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[26]  Rui Wang,et al.  Fast, realistic lighting and material design using nonlinear cut approximation , 2008, SIGGRAPH Asia '08.

[27]  Paul Graham,et al.  Acquiring reflectance and shape from continuous spherical harmonic illumination , 2013, ACM Trans. Graph..

[28]  Steve Marschner,et al.  Image-Based BRDF Measurement Including Human Skin , 1999, Rendering Techniques.

[29]  Sander Oude Elberink,et al.  Accuracy and Resolution of Kinect Depth Data for Indoor Mapping Applications , 2012, Sensors.

[30]  Ravi Ramamoorthi,et al.  Reflectance sharing: predicting appearance from a sparse set of images of a known shape , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Anselmo Lastra,et al.  A generalized surface appearance representation for computer graphics , 2002 .

[32]  Baining Guo,et al.  Pocket reflectometry , 2011, SIGGRAPH 2011.

[33]  Qionghai Dai,et al.  Capturing Relightable Human Performances under General Uncontrolled Illumination , 2013, Comput. Graph. Forum.

[34]  Gabriel Taubin,et al.  The ball-pivoting algorithm for surface reconstruction , 1999, IEEE Transactions on Visualization and Computer Graphics.