Detour Light Field Rendering for Diminished Reality Using Unstructured Multiple Views

Instructor's perspective videos are useful for presenting intuitive visual instructions for trainees in medical and industrial settings. In such videos, the instructor's arms often obstruct the trainee's view of the work area. In this article, we present a diminished reality method for visualizing the work area hidden by an instructor's arms by capturing the work area with multiple cameras. To achieve such diminished reality, we propose detour light field rendering (DLFR), in which light rays avoid passing through penalty points set in the unstructured light fields reconstructed from multiple viewpoint images. In DLFR, the camera blending field used in an existing freeviewpoint image generation method known as unstructured lumigraph is re-designed based on our use cases. In this re-design, lesser weights are given to light rays as they pass close to given penalty points. Experimental results demonstrate that using DLFR, the appearance of an undesirable object can be removed from an image in real time.

[1]  Nassir Navab,et al.  Multiview paraperspective projection model for diminished reality , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[2]  Marc Levoy,et al.  Using plane + parallax for calibrating dense camera arrays , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[3]  Mark Segal,et al.  Fast shadows and lighting effects using texture mapping , 1992, SIGGRAPH.

[4]  Fumihisa Shibata,et al.  Efficient Use of Textured 3D Model for Pre-observation-based Diminished Reality , 2015, 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops.

[5]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[6]  Hideo Saito,et al.  Real-Time Diminished Reality for Dynamic Scenes , 2015, 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops.

[7]  Naokazu Yokoya,et al.  [Paper] Diminished Reality for AR Marker Hiding Based on Image Inpainting with Reflection of Luminance Changes , 2013 .

[8]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[9]  Marc Levoy,et al.  Light Fields and Computational Imaging , 2006, Computer.

[10]  Zhuwen Li,et al.  Diminished reality using appearance and 3D geometry of internet photo collections , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).