A methodology for remote virtual interaction in teleimmersive environments

Though the quality of imaging devices, the accuracy of algorithms that construct 3D data, and the hardware available to render such data have all improved, the algorithms available to calibrate, reconstruct, and then visualize such data are difficult to use, extremely noise sensitive, and unreasonably slow. In this paper, we describe a multi-camera system that creates a highly accurate (on the order of a centimeter), 3D reconstruction of an environment in real time (under 30 ms) that allows for remote interaction between users. The paper addresses the aforementioned deficiencies by featuring an overview of the technology and algorithms used to calibrate, reconstruct, and render objects in the system. The algorithm produces partial 3D meshes, instead of dense point clouds, which are combined on the renderer to create a unified model of the environment. The chosen representation of the data allows for high compression ratios for transfer to remote sites. We demonstrate the accuracy and speed of our results on a variety of benchmarks and data collected from our own system.

[1]  Tomás Svoboda,et al.  A Convenient Multicamera Self-Calibration for Virtual Environments , 2005, Presence: Teleoperators & Virtual Environments.

[2]  Ruzena Bajcsy,et al.  Immersive 3D Environment for Remote Collaboration and Training of Physical Activities , 2008, 2008 IEEE Virtual Reality Conference.

[3]  Stefano Soatto,et al.  3D shape from anisotropic diffusion , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[4]  Philipp Slusallek,et al.  Wide area camera calibration using virtual calibration objects , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[5]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[6]  J. Bailenson,et al.  The Effect of Interactivity on Learning Physical Actions in Virtual Reality , 2008 .

[7]  Andreas Klaus,et al.  Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[8]  Zheng Zhi.  A Region Based Stereo Matching Algorithm Using Cooperative Optimization , 2009 .

[9]  Ruzena Bajcsy,et al.  The Effects of Fully Immersive Virtual Reality on the Learning of Physical Tasks , 2006 .

[10]  John F. Canny,et al.  MultiView: spatially faithful group video conferencing , 2005, CHI.

[11]  Jim X. Chen,et al.  Rendering avatars in virtual reality: integrating a 3D model with 2D images , 2002, Comput. Sci. Eng..

[12]  Richard Szeliski,et al.  High-accuracy stereo depth maps using structured light , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[13]  Margrit Gelautz,et al.  A layered stereo matching algorithm using image segmentation and global visibility constraints , 2005 .

[14]  Luc Van Gool,et al.  Blue-c: a spatially immersive display and 3D video portal for telepresence , 2003, IPT/EGVE.

[15]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[16]  Zeyu Li,et al.  Wide-area external multi-camera calibration using vision graphs and virtual calibration object , 2008, 2008 Second ACM/IEEE International Conference on Distributed Smart Cameras.

[17]  Ruzena Bajcsy,et al.  A Framework for Constructing Real-time Immersive Environments for Training Physical Activities , 2006, J. Multim..

[18]  Marcus A. Magnor,et al.  External Camera Calibration for Synchronized Multi-video Systems , 2004, WSCG.

[19]  Klara Nahrstedt,et al.  Advancing interactive collaborative mediums through tele-immersive dance (TED): a symbiotic creativity and design environment for art and computer science , 2008, ACM Multimedia.

[20]  Kostas Daniilidis,et al.  Real time trinocular stereo for tele-immersion , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[21]  Yoshihiko Nomura,et al.  Error analysis and optimization of camera calibration , 1991, Proceedings IROS '91:IEEE/RSJ International Workshop on Intelligent Robots and Systems '91.

[22]  Joseph M. Maubach,et al.  Local bisection refinement for $n$-simplicial grids generated by reflection , 2017 .

[23]  Yuichi Ohta,et al.  Mixed Reality: Merging Real and Virtual Worlds , 1999 .

[24]  Daniel Thalmann,et al.  Real-Time Animation of Realistic Virtual Humans , 1998, IEEE Computer Graphics and Applications.

[25]  Marc Pollefeys,et al.  IEEE Computer Society Conference on Computer Vision and Pattern Recognition workshops, 2008 , 2008, CVPR 2008.

[26]  Nassir Navab,et al.  Efficient visual hull computation for real-time 3D reconstruction using CUDA , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[27]  M. Hirose.  Mixed Reality - Merging Real and Virtual Worlds , 1999 .

[28]  Katherine Mezur.  Symbiosis of Tele-Immersive Environments with Creative Choreography , 2007 .

[29]  François X. Sillion,et al.  A Real-Time System for Full Body Interaction with Virtual Worlds , 2004, EGVE.

[30]  Thomas Malzbender,et al.  The Coliseum Immersive Teleconferencing System , 2002 .