Photorealistic Reproduction with Anisotropic Reflection on Mobile Devices using Densely Sampled Images

Photorealistic reproduction of real objects with complexi ti s of geometry and optics on mobile devices has been a long-standing challenge in augmented reality owing t o the difficulties of modeling and rendering the real object faithfully. Although image-based rendering, w hich does not require the objects to be modeled, has been proposed, it still fails to photorealistically rep roduce the object’s complete appearance containing complex optical properties such as anisotropic reflection. We propose a novel system for use on mobile devices capable of reproducing real objects photorealisti cally from all angles based on new view generation using densely sampled images. In order to realize the propos ed system, we developed a method of selecting the image closest to a given camera view from densely sampled ima ges by quantifying the similarity of two rays, performed rigid geometric transformation to preserve the v ertical direction for stable viewing, and introduced color correction for consistency of color between the gener ated view and the real world. Through experiments, we confirmed that our proposed system can reproduce real obje cts with complex optical properties more photorealistically compared with conventional augmented reality.

[1]  Katsumi Yamashita,et al.  Real-time color correction for a photo-based augmented reality system , 2013, 2013 IEEE 2nd Global Conference on Consumer Electronics (GCCE).

[2]  Yasushi Yagi,et al.  Rapid BRDF Measurement Using an Ellipsoidal Mirror and a Projector , 2009, IPSJ Trans. Comput. Vis. Appl..

[3]  Naotaka Fujii,et al.  Substitutional Reality System: A Novel Experimental Platform for Experiencing Alternative Reality , 2012, Scientific Reports.

[4]  David Salesin,et al.  Surface light fields for 3D photography , 2000, SIGGRAPH.

[5]  Katsumi Yamashita,et al.  Data compression for photo-based augmented reality on a tablet , 2016, 2016 IEEE 5th Global Conference on Consumer Electronics.

[6]  Adrien Bartoli,et al.  Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces , 2013, BMVC.

[7]  Evan Suma Rosenberg,et al.  Virtual Content Creation Using Dynamic Omnidirectional Texture Synthesis , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[8]  Robert Bregovic,et al.  Image based rendering technique via sparse representation in shearlet domain , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[9]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[10]  Gaurav Sharma,et al.  Digital color imaging , 1997, IEEE Trans. Image Process..

[11]  Paul Graham,et al.  Acquiring reflectance and shape from continuous spherical harmonic illumination , 2013, ACM Trans. Graph..

[12]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[13]  M. Ohta,et al.  A photo-based augmented reality system with HTML5/JavaScript , 2012, The 1st IEEE Global Conference on Consumer Electronics 2012.

[14]  I. Rock,et al.  The effect of retinal and phenomenal orientation on the perception of form. , 1957, The American journal of psychology.

[15]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[16]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[17]  Francisco José Madrid-Cuevas,et al.  Automatic generation and detection of highly reliable fiducial markers under occlusion , 2014, Pattern Recognit..

[18]  Michael Goesele,et al.  Image-based rendering for scenes with reflections , 2012, ACM Trans. Graph..

[19]  D. Ruderman,et al.  Statistics of cone responses to natural images: implications for visual coding , 1998 .

[20]  F. E. Nicodemus,et al.  Geometrical considerations and nomenclature for reflectance , 1977 .

[21]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[22]  Gary R. Bradski,et al.  ORB: An efficient alternative to SIFT or SURF , 2011, 2011 International Conference on Computer Vision.

[23]  Nicolas Bonneel,et al.  Reducing Anisotropic BSDF Measurement to Common Practice , 2014, Material Appearance Modeling.

[24]  Harry Shum,et al.  Pop-up light field: An interactive image-based modeling and rendering system , 2004, TOGS.