A Framework for Collaborative Real-Time 3D Teleimmersion in a Geographically Distributed Environment

In this paper, we present a framework for immersive 3D video conferencing and geographically distributed collaboration. Our multi-camera system performs a full-body 3D reconstruction of users in real time and renders their image in a virtual space allowing remote interaction between users and the virtual environment. The paper features an overview of the technology and algorithms used for calibration, capturing, and reconstruction. We introduce stereo mapping using adaptive triangulation which allows for fast (under 25 ms) and robust real-time 3D reconstruction. The chosen representation of the data provides high compression ratios for transfer to a remote site. The algorithm produces partial 3D meshes, instead of dense point clouds, which are combined on the renderer to create a unified model of the user. We have successfully demonstrated the use of our system in various applications such as remote dancing and immersive Tai Chi learning.

[1]  Daniel Thalmann,et al.  Real-Time Animation of Realistic Virtual Humans , 1998, IEEE Computer Graphics and Applications.

[2]  ThalmannDaniel,et al.  Real-Time Animation of Realistic Virtual Humans , 1998 .

[3]  Thomas Malzbender,et al.  The Coliseum Immersive Teleconferencing System , 2002 .

[4]  Kostas Daniilidis,et al.  Real time trinocular stereo for tele-immersion , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[5]  Heinrich Niemann,et al.  Finite element method for determination of optical flow , 1992, Pattern Recognit. Lett..

[6]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[7]  Joseph M. Maubach,et al.  Local bisection refinement for $n$-simplicial grids generated by reflection , 2017 .

[8]  Jim X. Chen,et al.  Rendering avatars in virtual reality: integrating a 3D model with 2D images , 2002, Comput. Sci. Eng..

[9]  Ruzena Bajcsy,et al.  ViewCast: view dissemination and management for multi-party 3d tele-immersive environments , 2007, ACM Multimedia.

[10]  Tomás Svoboda,et al.  A Convenient Multicamera Self-Calibration for Virtual Environments , 2005, Presence: Teleoperators & Virtual Environments.

[11]  Ruzena Bajcsy,et al.  Immersive 3D Environment for Remote Collaboration and Training of Physical Activities , 2008, 2008 IEEE Virtual Reality Conference.

[12]  Ruzena Bajcsy,et al.  A Framework for Constructing Real-time Immersive Environments for Training Physical Activities , 2006, J. Multim..

[13]  Marcus A. Magnor,et al.  External Camera Calibration for Synchronized Multi-video Systems , 2004, WSCG.

[14]  Zeyu Li,et al.  Wide-area external multi-camera calibration using vision graphs and virtual calibration object , 2008, 2008 Second ACM/IEEE International Conference on Distributed Smart Cameras.

[15]  J. Bailenson,et al.  The Effect of Interactivity on Learning Physical Actions in Virtual Reality , 2008 .

[16]  Katherine Mezur.  Symbiosis of Tele-Immersive Environments with Creative Choreography , 2007 .

[17]  John F. Canny,et al.  MultiView: spatially faithful group video conferencing , 2005, CHI.

[18]  Luc Van Gool,et al.  Blue-c: a spatially immersive display and 3D video portal for telepresence , 2003, IPT/EGVE.

[19]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[20]  Philipp Slusallek,et al.  Wide area camera calibration using virtual calibration objects , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[21]  Ruzena Bajcsy,et al.  The Effects of Fully Immersive Virtual Reality on the Learning of Physical Tasks , 2006 .

[22]  M. Hirose.  Mixed Reality - Merging Real and Virtual Worlds , 1999 .

[23]  François X. Sillion,et al.  A Real-Time System for Full Body Interaction with Virtual Worlds , 2004, EGVE.