Affordance Analysis of Virtual and Augmented Reality Mediated Communication

Virtual and augmented reality communication platforms are seen as promising modalities for next-generation remote face-to-face interactions. Our study attempts to explore non-verbal communication features in relation to their conversation context for virtual and augmented reality mediated communication settings. We perform a series of user experiments, triggering nine conversation tasks in 4 settings, each containing corresponding non-verbal communication features. Our results indicate that conversation types which involve less emotional engagement are more likely to be acceptable in virtual reality and augmented reality settings with low-fidelity avatar representation, compared to scenarios that involve high emotional engagement or intellectually difficult discussions. We further systematically analyze and rank the impact of low-fidelity representation of micro-expressions, body scale, head pose, and hand gesture in affecting the user experience in one-on-one conversations, and validate that preserving micro-expression cues plays the most effective role in improving bi-directional conversations in future virtual and augmented reality settings.

[1]  Michael J. Black,et al.  Detailed Full-Body Reconstructions of Moving People from Monocular RGB-D Sequences , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  Robert E. Kraut,et al.  Visual Information as a Conversational Resource in Collaborative Physical Tasks , 2003, Hum. Comput. Interact..

[3]  Henry Fuchs,et al.  Immersive 3D Telepresence , 2014, Computer.

[4]  Michael Neff,et al.  Communication Behavior in Embodied Virtual Reality , 2018, CHI.

[5]  Hirokazu Kato,et al.  THE EFFECT OF SPATIAL CUES IN AUGMENTED REALITY VIDEO CONFERENCING , 2001 .

[6]  Steve Benford,et al.  Embodiments, avatars, clones and agents for multi-user, multi-sensory virtual worlds , 1997, Multimedia Systems.

[7]  Robert E. Kraut,et al.  Action as language in a shared visual space , 2004, CSCW.

[8]  Bilge Mutlu,et al.  Handheld or Handsfree?: Remote Collaboration via Lightweight Head-Mounted Displays and Handheld Devices , 2015, CSCW.

[9]  H. H. Clark,et al.  Speaking while monitoring addressees for understanding , 2004 .

[10]  Herbert H. Clark,et al.  Grounding in communication , 1991, Perspectives on socially shared cognition.

[11]  J. Haxby,et al.  Human neural systems for face recognition and social communication , 2002, Biological Psychiatry.

[12]  Ralph Schroeder,et al.  Small-Group Behavior in a Virtual and Real Environment: A Comparative Study , 2000, Presence: Teleoperators & Virtual Environments.

[13]  A. Kendon.  Some functions of gaze-direction in social interaction. , 1967, Acta psychologica.

[14]  John Short,et al.  The social psychology of telecommunications , 1976 .

[15]  Robert W. Lindeman,et al.  Mini-Me: An Adaptive Avatar for Mixed Reality Remote Collaboration , 2018, CHI.

[16]  Tom Rodden,et al.  Turn it this way: grounding collaborative action with remote gestures , 2007, CHI.

[17]  P. Blanche,et al.  Holographic three-dimensional telepresence using large-area photorefractive polymer , 2010, Nature.

[18]  Mel Slater,et al.  The impact of avatar realism and eye gaze control on perceived quality of communication in a shared immersive virtual environment , 2003, CHI '03.

[19]  Bernd Fröhlich,et al.  Immersive Group-to-Group Telepresence , 2013, IEEE Transactions on Visualization and Computer Graphics.

[20]  Henry Fuchs,et al.  Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[21]  Henry Fuchs,et al.  Real-time volumetric 3D capture of room-sized scenes for telepresence , 2012, 2012 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[22]  Gwyneth Doherty-Sneddon,et al.  Face-to-face and video mediated communication: a comparison of dialogue structure and task performance , 1997 .

[23]  J. Haxby,et al.  The distributed human neural system for face perception , 2000, Trends in Cognitive Sciences.

[24]  J. Streeck.  Gesture as communication I: Its coordination with gaze and speech , 1993 .

[25]  Susan R. Fussell,et al.  Gestures Over Video Streams to Support Remote Collaboration on Physical Tasks , 2004, Hum. Comput. Interact..

[26]  Ronald E. Rice,et al.  Evaluating video as a technology for informal communication , 1992, CHI.

[27]  Ivan Poupyrev,et al.  Experiments with Face-To-Face Collaborative AR Interfaces , 2002, Virtual Reality.

[28]  Anoop Gupta,et al.  Communication behaviors of co-located users in collaborative AR interfaces , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[29]  Pushmeet Kohli,et al.  Simultaneous Segmentation and Pose Estimation of Humans Using Dynamic Graph Cuts , 2008, International Journal of Computer Vision.