Towards Mobile Diminished Reality

We present a diminished reality application running live on consumer mobile devices. In our pre-observation-based approach, the clean 3D scene, free of undesired objects, is scanned beforehand and reconstructed as a high resolution textured 3D model. At runtime, objects added in a region of interest are efficiently removed by projecting the previously captured background. Differences of illumination conditions between scan time and run-time are compensated to obtain seamless results. The proposed approach requires no segmentation or manual input other than the definition of the 3D region of interest to be diminished, and is not based on any particular assumption on the background geometry. We show the potential of our approach by processing a variety of challenging unknown 3D scenes including textured backgrounds, dynamic illumination conditions and foreground objects partially occluding the diminished region. We provide details on our compute shader implementation to make as easy as possible the reimplementation by the community.

[1]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[2]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[3]  Anil C. Kokaram,et al.  Interpolation of missing data in image sequences , 1995, IEEE Trans. Image Process..

[4]  Hideo Saito,et al.  Real-Time Diminished Reality for Dynamic Scenes , 2015, 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops.

[5]  Naokazu Yokoya,et al.  Diminished Reality Based on Image Inpainting Considering Background Geometry , 2016, IEEE Transactions on Visualization and Computer Graphics.

[6]  Hideo Saito,et al.  A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects , 2017, IPSJ Transactions on Computer Vision and Applications.

[7]  Matthieu Fradet,et al.  A Multi-resolution Approach for Color Correction of Textured Meshes , 2018, 2018 International Conference on 3D Vision (3DV).

[8]  H. Saito,et al.  Diminished Reality using Multiple Handheld Cameras , 2007 .

[9]  Patrick Pérez,et al.  Video Inpainting of Complex Scenes , 2014, SIAM J. Imaging Sci..

[10]  Yehu Shen,et al.  A simple and fast image cloning algorithm , 2016, 2016 12th World Congress on Intelligent Control and Automation (WCICA).

[11]  Fumihisa Shibata,et al.  Efficient Use of Textured 3D Model for Pre-observation-based Diminished Reality , 2015, 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops.

[12]  Michael F. Cohen,et al.  Emptying, refurnishing, and relighting indoor spaces , 2016, ACM Trans. Graph..

[13]  Eli Shechtman,et al.  Space-Time Completion of Video , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Gang Ren,et al.  SceneCtrl: Mixed Reality Enhancement via Efficient Scene Editing , 2017, UIST.

[15]  Zhuwen Li,et al.  Diminished reality using appearance and 3D geometry of internet photo collections , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[16]  Matthias Nießner,et al.  ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[17]  Wolfgang Broll,et al.  High-Quality Real-Time Video Inpaintingwith PixMix , 2014, IEEE Transactions on Visualization and Computer Graphics.

[18]  Zeev Farbman,et al.  Coordinates for instant image cloning , 2009, ACM Trans. Graph..

[19]  Nassir Navab,et al.  Multiview paraperspective projection model for diminished reality , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[20]  Fabio Bruno,et al.  Augmented touch without visual obtrusion , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.