Virtual Reality Based Immersive Telepresence System for Remote Conversation and Collaboration

We developed a Virtual Reality (VR) based telepresence system providing novel immersive experience for remote conversation and collaboration. By wearing VR headsets, all the participants can be gathered into a same virtual space, with 3D cartoon Avatars representing them. The 3D VR Avatars can realistically emulate the head postures, facial expressions and hand motions of the participants, enabling them to conduct enjoyable group-to-group conversations with people spatially isolated from them. Moreover, our VR telepresence system offers conspicuously new manners for remote collaboration. For example, users can play PPT slides or watch videos together, or they can cooperate on solving a math problem by calculating on a virtual blackboard, all of which can be hardly achieved using conventional video-based telepresence system. Experiments show that our system can provide unprecedented immersive experience for tele-conversation and new possibilities for remote collaboration.

[1]  Robin Wolff,et al.  A Mixed Reality Telepresence System for Collaborative Space Operation , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[2]  Philip A. Chou,et al.  Viewport: A Distributed, Immersive Teleconferencing System with Infrared Dot Pattern , 2013, IEEE MultiMedia.

[3]  Xin Zhou,et al.  Rapid DCT-based LipSync generation algorithm for game making , 2016, SIGGRAPH Asia Posters.

[4]  Fernando De la Torre,et al.  Supervised Descent Method and Its Applications to Face Alignment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Yu Zhou,et al.  Leveraging Two Kinect Sensors for Accurate Full-Body Motion Capture , 2015, Sensors.

[6]  Fuchun Sun,et al.  A novel data glove using inertial and magnetic sensors for motion capture and robotic arm-hand teleoperation , 2017, Ind. Robot.

[7]  Eugene Fiume,et al.  JALI , 2016, ACM Trans. Graph..

[8]  Kun Zhou,et al.  3D shape regression for real-time facial animation , 2013, ACM Trans. Graph..

[9]  Henry Fuchs,et al.  Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[10]  Kazuhiro Otsuka MMSpace: Kinetically-augmented telepresence for small group-to-group conversations , 2016, 2016 IEEE Virtual Reality (VR).

[11]  Zhong Zhou,et al.  Real-time stereo-vision system for 3D teleimmersive collaboration , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[12]  Keita Higuchi,et al.  ImmerseBoard: Immersive Telepresence Experience using a Digital Whiteboard , 2015, CHI.

[13]  Wang Yin Chai,et al.  Development of Real-Time Lip Sync Animation Framework Based On Viseme Human Speech , 2014 .

[14]  Zhe Zhu,et al.  3D modeling and motion parallax for improved videoconferencing , 2016, Computational Visual Media.

[15]  Mark Pauly,et al.  Dynamic 3D avatar creation from hand-held video input , 2015, ACM Trans. Graph..

[16]  Andreas Aristidou,et al.  FABRIK: A fast, iterative solver for the Inverse Kinematics problem , 2011, Graph. Model..