Volumetric Mixed Reality Telepresence for Real-time Cross Modality Collaboration

Mixed-reality telepresence allows local and remote users feel as if they are present together in the same space. In this paper we report on a mixed-reality volumetric telepresence system that is adaptable, multi-user and cross-modal, i.e. combining augmented and virtual reality technologies with face-to-face interactions. The system extends state-of-art by creating full-body and environmental volumetric renderings in real-time over local enterprise networks. We report findings of an evaluation in a training scenario which was adapted for remote delivery and led by an industry professional. Analysis of interviews and observed behaviours identify varying attitudes towards virtually mediated full-body experiences and highlight the impact of volumetric mixed-reality telepresence to facilitate personal experiences of co-presence and to ground communication with interlocutors.

[1]  Paulo Dias,et al.  A Conceptual Model and Taxonomy for Collaborative Augmented Reality , 2021, IEEE Transactions on Visualization and Computer Graphics.

[2]  T. Koskela,et al.  "Nice to see you virtually": Thoughtful design and evaluation of virtual avatar of the other user in AR and VR based telexistence systems , 2021, Entertain. Comput..

[3]  Mark Billinghurst,et al.  eyemR-Vis: Using Bi-Directional Gaze Behavioural Cues to Improve Mixed Reality Remote Collaboration , 2021, CHI Extended Abstracts.

[4]  Jenny Waycott,et al.  Avatar-Mediated Communication in Social VR: An In-depth Exploration of Older Adult Interaction in an Emerging Communication Platform , 2021, CHI.

[5]  Maurício Sousa,et al.  Adventures in Hologram Space: Exploring the Design Space of Eye-to-eye Volumetric Telepresence , 2019, VRST.

[6]  Paul E. Debevec,et al.  The relightables , 2019, ACM Trans. Graph..

[7]  Tovi Grossman,et al.  Loki: Facilitating Remote Instruction of Physical Tasks Using Bi-Directional Mixed-Reality Telepresence , 2019, UIST.

[8]  Daniel Jackson,et al.  Conveying spatial awareness cues in xR collaborations , 2019, IEEE Transactions on Visualization and Computer Graphics.

[9]  Matt Adcock,et al.  Mixed Reality Remote Collaboration Combining 360 Video and 3D Reconstruction , 2019, CHI.

[10]  Simon Richir,et al.  From Robot to Virtual Doppelganger: Impact of Visual Fidelity of Avatars Controlled in Third-Person Perspective on Embodiment and Behavior in Immersive Virtual Environments , 2019, Front. Robot. AI.

[11]  Arindam Dey,et al.  The Effects of Sharing Awareness Cues in Collaborative Mixed Reality , 2019, Front. Robot. AI.

[12]  Frank Steinicke,et al.  Floor-Projected Guidance Cues for Collaborative Exploration of Spatial Augmented Reality Setups , 2018, ISS.

[13]  Jörg Müller,et al.  Velt: A Framework for Multi RGB-D Camera Systems , 2018, ISS.

[14]  Robert W. Lindeman,et al.  Mini-Me: An Adaptive Avatar for Mixed Reality Remote Collaboration , 2018, CHI.

[15]  David Lindlbauer,et al.  Remixed Reality: Manipulating Space and Time in Augmented Reality , 2018, CHI.

[16]  Marc Erich Latoschik,et al.  Any “Body” There? Avatar Visibility Effects in a Virtual Reality Game , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[17]  Mark Billinghurst,et al.  CoVAR: a collaborative virtual and augmented reality system for remote collaboration , 2017, SIGGRAPH ASIA Emerging Technologies.

[18]  Shahram Izadi,et al.  Motion2fusion , 2017, ACM Trans. Graph..

[19]  Daniel Mendes,et al.  Creepy Tracker Toolkit for Context-aware Interfaces , 2017, ISS.

[20]  Andy Wilson,et al.  MeetAlive: Room-Scale Omni-Directional Display System for Multi-User Content and Control Sharing , 2017, ISS.

[21]  Holger Regenbrecht,et al.  Mixed Voxel Reality: Presence and Embodiment in Low Fidelity, Visually Coherent, Mixed Reality Environments , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[22]  Ross T. Smith,et al.  Cognitive Cost of Using Augmented Reality Displays , 2017, IEEE Transactions on Visualization and Computer Graphics.

[23]  Robin Wolff,et al.  A Mixed Reality Telepresence System for Collaborative Space Operation , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[24]  Hrvoje Benko,et al.  Projected Augmented Reality with the RoomAlive Toolkit , 2016, ISS.

[25]  Charles T. Loop,et al.  Holoportation: Virtual 3D Teleportation in Real-time , 2016, UIST.

[26]  Pushmeet Kohli,et al.  Fusion4D , 2016, ACM Trans. Graph..

[27]  Eyal Ofek,et al.  Room2Room: Enabling Life-Size Telepresence in a Projected Augmented Reality Environment , 2016, CSCW.

[28]  Yuxi Wang,et al.  SoD-Toolkit: A Toolkit for Interactively Prototyping and Developing Multi-Sensor, Multi-Device Environments , 2015, ITS.

[29]  Marek Kowalski,et al.  Livescan3D: A Fast and Inexpensive 3D Data Acquisition System for Multiple Kinect v2 Sensors , 2015, 2015 International Conference on 3D Vision.

[30]  Alvaro Collet,et al.  High-quality streamable free-viewpoint video , 2015, ACM Trans. Graph..

[31]  Keita Higuchi,et al.  ImmerseBoard: Immersive Telepresence Experience using a Digital Whiteboard , 2015, CHI.

[32]  Shahram Izadi,et al.  3D-board: a whole-body remote collaborative whiteboard , 2014, UIST.

[33]  Blair MacIntyre,et al.  RoomAlive: magical experiences enabled by scalable, adaptive projector-camera units , 2014, UIST.

[34]  Bruce H. Thomas,et al.  Visualization of off-surface 3D viewpoint locations in spatial augmented reality , 2013, SUI '13.

[35]  Bernd Fröhlich,et al.  Immersive Group-to-Group Telepresence , 2013, IEEE Transactions on Visualization and Computer Graphics.

[36]  Henry Fuchs,et al.  Real-time volumetric 3D capture of room-sized scenes for telepresence , 2012, 2012 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[37]  Massimo Bergamasco,et al.  Beaming: An Asymmetric Telepresence System , 2012, IEEE Computer Graphics and Applications.

[38]  Kun Peng,et al.  Enhanced personal autostereoscopic telepresence system using commodity depth cameras , 2012, Comput. Graph..

[39]  Henry Fuchs,et al.  Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[40]  Nicolai Marquardt,et al.  The proximity toolkit: prototyping proxemic interactions in ubiquitous computing ecologies , 2011, UIST.

[41]  Jun Rekimoto,et al.  Peripheral vision annotation: noninterference information presentation method for mobile augmented reality , 2011, AH '11.

[42]  Saul Greenberg,et al.  Proxemic interaction: designing for a proximity and orientation-aware environment , 2010, ITS '10.

[43]  Ruzena Bajcsy,et al.  Immersive 3D Environment for Remote Collaboration and Training of Physical Activities , 2008, 2008 IEEE Virtual Reality Conference.

[44]  Masayuki Tanimoto,et al.  Multiview Imaging and 3DTV , 2007, IEEE Signal Processing Magazine.

[45]  Robert E. Kraut,et al.  Action as language in a shared visual space , 2004, CSCW.

[46]  Hirokazu Kato,et al.  Collaborative augmented reality , 2002, CACM.

[47]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[48]  Greg Welch,et al.  The office of the future: a unified approach to image-based modeling and spatially immersive displays , 1998, SIGGRAPH.

[49]  F. Biocca,et al.  Communication in the age of virtual reality , 1995 .

[50]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[51]  Frank Biocca,et al.  Virtual Reality Technology: A Tutorial , 1992 .

[52]  Robert David Sack,et al.  Human Territoriality: A Theory , 1983 .

[53]  A. Bottino,et al.  Assessing Visual Cues for Improving Awareness in Collaborative Augmented Reality , 2022, XR.

[54]  Mark Billinghurst,et al.  AR/MR Remote Collaboration on Physical Tasks: A Review , 2021, Robotics Comput. Integr. Manuf..

[55]  Jeremy N. Bailenson,et al.  Virtual Doppelgangers: Psychological Effects of Avatars Who Ignore Their Owners , 2010, Online Worlds: Convergence of the Real and the Virtual.

[56]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[57]  Herbert H. Clark,et al.  Grounding in communication , 1991, Perspectives on socially shared cognition.

[58]  V. Braun,et al.  Please Scroll down for Article Qualitative Research in Psychology Using Thematic Analysis in Psychology , 2022 .