Projection mapping based on BRDF reconstruction from single RGBD image

There have been many researches on projection mapping focused on target objects tracking, geometric shape recovering or virtual materials simulating such as clothes. However, few people pay attention to the material of target object which actually influences the visual results of projection. We present a new projection mapping framework based on BRDF reconstruction for the goal of more real projection results by enhancing the effects of Augmented Reality. In the framework, 3D computer vision method is used to reconstruct the BRDF of target object with a single RGBD image. A new algorithm is proposed using two Convolutional Neural Networks(CNN) which can predict both normal map and reflectance map of the target surface simultaneously with the RGBD image. The predicted maps are used to render the content to be projected onto the target object. Our BRDF reconstruction algorithm can recover several materials in one scene correctly in use of just one image. Experimental results show our framework has impressive performance and relatively accurate consequence.

[1]  Ping Tan,et al.  A Benchmark Dataset and Evaluation for Non-Lambertian and Uncalibrated Photometric Stereo , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Ramesh Raskar,et al.  Dynamic shader lamps : painting on movable objects , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[3]  Woontack Woo,et al.  Mirror Mirror: An On-Body T-shirt Design System , 2016, CHI.

[4]  Anselm Grundhöfer,et al.  Augmenting physical avatars using projector-based illumination , 2013, ACM Trans. Graph..

[5]  Steve Marschner,et al.  Estimating dual-scale properties of glossy surfaces from step-edge lighting , 2011, ACM Trans. Graph..

[6]  Greg Welch,et al.  A general approach for closed-loop registration in AR , 2012, 2013 IEEE Virtual Reality (VR).

[7]  Takeshi Naemura,et al.  Continuous 3D Label Stereo Matching Using Local Expansion Moves , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Jannik Boll Nielsen,et al.  Minimal BRDF sampling for two-shot near-field reflectance acquisition , 2016, ACM Trans. Graph..

[9]  Dieter Fox,et al.  DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Kosuke Sato,et al.  Diminishable visual markers on fabricated projection object for dynamic spatial augmented reality , 2015, SIGGRAPH Asia Emerging Technologies.

[11]  Ko Nishino,et al.  Shape and Reflectance Estimation in the Wild , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Todd E. Zickler,et al.  Blind Reflectometry , 2010, ECCV.

[14]  Ramesh Raskar,et al.  Modern approaches to augmented reality: introduction to current approaches , 2006, SIGGRAPH Courses.

[15]  Jeremy R. Cooperstock,et al.  Digital facial augmentation for interactive entertainment , 2015, 2015 7th International Conference on Intelligent Technologies for Interactive Entertainment (INTETAIN).

[16]  Greg Welch,et al.  Animatronic shader lamps avatars , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[17]  Geoffrey E. Hinton,et al.  Deep Lambertian Networks , 2012, ICML.

[18]  Ko Nishino,et al.  Reflectance and Natural Illumination from a Single Image , 2012, ECCV.

[19]  Xiang Cao,et al.  Interactive Environment-Aware Handheld Projectors for Pervasive Computing Spaces , 2012, Pervasive.

[20]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[21]  Christoph Resch,et al.  Sticky projections — A new approach to interactive shader lamp tracking , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[22]  Zhe Wu,et al.  A Benchmark Dataset and Evaluation for Non-Lambertian and Uncalibrated Photometric Stereo , 2019, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Alex Kendall,et al.  End-to-End Learning of Geometry and Context for Deep Stereo Regression , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[24]  Shigeo Morishima,et al.  Retexturing under self-occlusion using hierarchical markers , 2017, SIGGRAPH Posters.

[25]  Xiao Li,et al.  Modeling surface appearance from a single photograph using self-augmented convolutional neural networks , 2017, ACM Trans. Graph..

[26]  Atsuo Takanishi,et al.  Development and evaluation of face robot to express various face shape , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[27]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[28]  Justus Thies,et al.  Real-time pixel luminance optimization for dynamic multi-projection mapping , 2015, ACM Trans. Graph..

[29]  Mario Fritz,et al.  Deep Reflectance Maps , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Ichiro Sakuma,et al.  Augmented Reality Navigation With Automatic Marker-Free Image Registration Using 3-D Image Overlay for Dental Surgery , 2014, IEEE Transactions on Biomedical Engineering.

[31]  Jonathan T. Barron,et al.  Scene Intrinsics and Depth from a Single Image , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[32]  Jaakko Lehtinen,et al.  Reflectance modeling by neural texture synthesis , 2016, ACM Trans. Graph..

[33]  Yasuyuki Matsushita,et al.  Robust Multiview Photometric Stereo Using Planar Mesh Parameterization , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Alexei A. Efros,et al.  SVBRDF-Invariant Shape and Reflectance Estimation from Light-Field Cameras , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Ko Nishino,et al.  Reflectance and Illumination Recovery in the Wild , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Anselm Grundhöfer,et al.  Makeup Lamps: Live Augmentation of Human Faces via Projection , 2017, Comput. Graph. Forum.

[37]  Yasuyuki Matsushita,et al.  Deep Photometric Stereo Network , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[38]  Xu Chen,et al.  Pmomo: Projection Mapping on Movable 3D Object , 2016, CHI.

[39]  Eyal Ofek,et al.  IllumiRoom: peripheral projected illusions for interactive experiences , 2013, SIGGRAPH '13.

[40]  Gordon Wetzstein,et al.  The Visual Computing of Projector‐Camera Systems , 2008, SIGGRAPH '08.