Technologies for Social Augmentations in User-Embodied Virtual Reality

Technologies for Virtual, Mixed, and Augmented Reality (VR, MR, and AR) allow to artificially augment social interactions and thus to go beyond what is possible in real life. Motivations for the use of social augmentations are manifold, for example, to synthesize behavior when sensory input is missing, to provide additional affordances in shared environments, or to support inclusion and training of individuals with social communication disorders. We review and categorize augmentation approaches and propose a software architecture based on four data layers. Three components further handle the status analysis, the modification, and the blending of behaviors. We present a prototype (injectX) that supports behavior tracking (body motion, eye gaze, and facial expressions from the lower face), status analysis, decision-making, augmentation, and behavior blending in immersive interactions. Along with a critical reflection, we consider further technical and ethical aspects.

[1]  Michitaka Hirose,et al.  Brain-Computer Interfaces, Virtual Reality, and Videogames , 2008, Computer.

[2]  Maja Pantic,et al.  Social signal processing: Survey of an emerging domain , 2009, Image Vis. Comput..

[3]  Charles T. Loop,et al.  Holoportation: Virtual 3D Teleportation in Real-time , 2016, UIST.

[4]  Jaron Lanier,et al.  Reality built for two: a virtual reality tool , 1990, I3D '90.

[5]  M. Turk,et al.  Transformed social interaction, augmented gaze, and social influence in immersive virtual environments , 2005 .

[6]  Marc Erich Latoschik,et al.  Brain 2 Communicate: EEG-based Affect Recognition to Augment Virtual Social Interactions , 2019, Mensch & Computer Workshopband.

[7]  T. Chartrand,et al.  The Chameleon Effect as Social Glue: Evidence for the Evolutionary Significance of Nonconscious Mimicry , 2003 .

[8]  Robin Wolff,et al.  Communicating Eye-gaze Across a Distance: Comparing an Eye-gaze enabled Immersive Collaborative Virtual Environment, Aligned Video Conferencing, and Being Together , 2009, 2009 IEEE Virtual Reality Conference.

[9]  Jeremy N. Bailenson,et al.  The Effect of Behavioral Realism and Form Realism of Real-Time Avatar Faces on Verbal Disclosure, Nonverbal Disclosure, Emotion Recognition, and Copresence in Dyadic Interaction , 2006, PRESENCE: Teleoperators and Virtual Environments.

[10]  Mel Slater,et al.  How to Build an Embodiment Lab: Achieving Body Representation Illusions in Virtual Reality , 2014, Front. Robot. AI.

[11]  Michael Neff,et al.  Communication Behavior in Embodied Virtual Reality , 2018, CHI.

[12]  Mel Slater,et al.  Grand Challenges in Virtual Environments , 2014, Front. Robot. AI.

[13]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[14]  Katherine Isbister,et al.  Understanding Emerging Design Practices for Avatar Systems in the Commercial Social VR Ecology , 2019, Conference on Designing Interactive Systems.

[15]  Steve Benford,et al.  User embodiment in collaborative virtual environments , 1995, CHI '95.

[16]  Albert A. Rizzo,et al.  FAAST: The Flexible Action and Articulated Skeleton Toolkit , 2011, 2011 IEEE Virtual Reality Conference.

[17]  Marc Erich Latoschik,et al.  Beyond Replication: Augmenting Social Behaviors in Multi-User Virtual Realities , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[18]  Tobias Baur,et al.  The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time , 2013, ACM Multimedia.

[19]  Mel Slater,et al.  Small Group Behavior Experiments in the Coven Project , 1998, IEEE Computer Graphics and Applications.

[20]  Takeo Kanade,et al.  Virtual Space Teleconferencing Using a Sea of Cameras , 1994 .

[21]  Ross T. Smith,et al.  Emotion Sharing and Augmentation in Cooperative Virtual Reality Games , 2018, CHI PLAY.

[22]  Steed Anthony,et al.  Spatial Interfaces-Beaming: An Asymmetric Telepresence System , 2012 .

[23]  Marc Erich Latoschik,et al.  The effect of avatar realism in immersive social virtual realities , 2017, VRST.

[24]  Michael J. Hove,et al.  It's all in the timing: Interpersonal synchrony increases affiliation , 2009 .

[25]  L. Tickle-Degnen,et al.  The Nature of Rapport and Its Nonverbal Correlates , 1990 .

[26]  S. Liversedge,et al.  Oxford handbook of eye movements , 2011 .

[27]  Kazuhiro Otsuka.  MMSpace: Kinetically-augmented telepresence for small group-to-group conversations , 2016, 2016 IEEE Virtual Reality (VR).

[28]  Hirokazu Kato,et al.  Collaborative Mixed Reality , 1999 .

[29]  Soo Youn Oh,et al.  Let the Avatar Brighten Your Smile: Effects of Enhancing Facial Expressions in Virtual Environments , 2016, PloS one.

[30]  Robert W. Lindeman,et al.  Mini-Me: An Adaptive Avatar for Mixed Reality Remote Collaboration , 2018, CHI.

[31]  Bernd Fröhlich,et al.  Immersive Group-to-Group Telepresence , 2013, IEEE Transactions on Visualization and Computer Graphics.

[32]  James D. Hollan,et al.  Beyond being there , 1992, CHI.

[33]  H. Fuchs,et al.  A First Look at a Telepresence System with Room-Sized Real-Time 3 D Capture and Life-Sized Tracked Display Wall , 2011 .

[34]  Jaron Lanier,et al.  Homuncular Flexibility: The Human Ability to Inhabit Nonhuman Avatars , 2015 .

[35]  J. Cappella.  Mutual influence in expressive behavior: adult--adult and infant--adult dyadic interaction. , 1981, Psychological bulletin.

[36]  Mel Slater,et al.  Meeting people vitually: experiments in shared virtual environments , 2002 .

[37]  Mark Billinghurst,et al.  Sharing and Augmenting Emotion in Collaborative Mixed Reality , 2018, 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct).

[38]  Emmanuel Frécon,et al.  DIVE: a scaleable network architecture for distributed virtual environments , 1998, Distributed Syst. Eng..

[39]  Robin Wolff,et al.  Eye Tracking for Avatar Eye Gaze Control During Object-Focused Multiparty Interaction in Immersive Collaborative Virtual Environments , 2009, 2009 IEEE Virtual Reality Conference.

[40]  Jeremy N. Bailenson,et al.  Transformed Social Interaction: Decoupling Representation from Behavior and Form in Collaborative Virtual Environments , 2004, Presence: Teleoperators & Virtual Environments.

[41]  Arjun Nagendran,et al.  A Unified Framework for Individualized Avatar-Based Interactions , 2014, PRESENCE: Teleoperators and Virtual Environments.

[42]  Chongyang Ma,et al.  Facial performance sensing head-mounted display , 2015, ACM Trans. Graph..

[43]  Andreas Aristidou,et al.  FABRIK: A fast, iterative solver for the Inverse Kinematics problem , 2011, Graph. Model..

[44]  Jeffrey R. Spies,et al.  Effects of damping head movement and facial expression in dyadic conversation using real–time facial expression tracking and synthesized avatars , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[45]  L. Stark,et al.  Most naturally occurring human saccades have magnitudes of 15 degrees or less. , 1975, Investigative ophthalmology.

[46]  Maria V. Sanchez-Vives,et al.  First Person Experience of Body Transfer in Virtual Reality , 2010, PloS one.

[47]  Stacy Marsella,et al.  Virtual Rapport , 2006, IVA.

[48]  Massimo Bergamasco,et al.  Beaming: An Asymmetric Telepresence System , 2012, IEEE Computer Graphics and Applications.

[49]  Marc Erich Latoschik,et al.  Fast generation of realistic virtual humans , 2017, VRST.

[50]  André Kunert,et al.  Virtual Valcamonica: Collaborative Exploration of Prehistoric Petroglyphs and Their Surrounding Environment in Multi-User Virtual Reality , 2017, PRESENCE: Teleoperators and Virtual Environments.

[51]  Marc Erich Latoschik,et al.  Effects of Hybrid and Synthetic Social Gaze in Avatar-Mediated Interactions , 2018, 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct).

[52]  Marc Erich Latoschik,et al.  Hybrid Avatar-Agent Technology – A Conceptual Step Towards Mediated “Social” Virtual Reality and its Respective Challenges , 2015, i-com.

[53]  David B. Kaber,et al.  Telepresence , 1998, Hum. Factors.

[54]  Marc Erich Latoschik,et al.  Not Alone Here?! Scalability and User Experience of Embodied Ambient Crowds in Distributed Social Virtual Reality , 2019, IEEE Transactions on Visualization and Computer Graphics.

[55]  Jessica K. Hodgins,et al.  Interactive control of avatars animated with human motion data , 2002, SIGGRAPH.

[56]  Marc Erich Latoschik,et al.  Injecting Nonverbal Mimicry with Hybrid Avatar-Agent Technologies: A Naïve Approach , 2018, SUI.

[57]  N. Badler,et al.  Eyes Alive Eyes Alive Eyes Alive Figure 1: Sample Images of an Animated Face with Eye Movements , 2022 .

[58]  Marc Erich Latoschik,et al.  A simplified inverse kinematic approach for embodied VR applications , 2016, 2016 IEEE Virtual Reality (VR).

[59]  Karl F. MacDorman,et al.  Revisiting the uncanny valley theory: Developing and validating an alternative to the Godspeed indices , 2010, Comput. Hum. Behav..

[60]  Marc Erich Latoschik,et al.  The Impact of Avatar Personalization and Immersion on Virtual Body Ownership, Presence, and Emotional Response , 2018, IEEE Transactions on Visualization and Computer Graphics.

[61]  Lucas Kovar,et al.  Motion Graphs , 2002, ACM Trans. Graph..

[62]  Marc Erich Latoschik,et al.  Avatar realism and social interaction quality in virtual reality , 2016, 2016 IEEE Virtual Reality (VR).

[63]  David G. Rand,et al.  Why We Cooperate , 2014 .

[64]  Michael F. Cohen,et al.  Efficient generation of motion transitions using spacetime constraints , 1996, SIGGRAPH.

[65]  Oleg Spakov.  Comparison of eye movement filters used in HCI , 2012, ETRA '12.

[66]  Mel Slater,et al.  An Evaluation of Self-Avatar Eye Movement for Virtual Embodiment , 2013, IEEE Transactions on Visualization and Computer Graphics.

[67]  G. Bente,et al.  Avatar-Mediated Networking: Increasing Social Presence and Interpersonal Trust in Net-Based Collaborations , 2008 .

[68]  Dave J. Hobbs,et al.  Continuous Presence in Collaborative Virtual Environments: Towards a Hybrid Avatar-Agent Model for User Representation , 2001, IVA.

[69]  俞基勋,et al.  Head-mounted display with facial expression detecting capability , 2016 .

[70]  Marc Erich Latoschik,et al.  SIAMC: a socially immersive avatar mediated communication platform , 2016, VRST.

[71]  J. Bailenson,et al.  Transformed social interaction in mediated interpersonal communication , 2008 .