Enhancing User Engagement in Immersive Games through Multisensory Cues

In this article, we report on a user study investigating the effects of multisensory cues on triggering the emotional response in immersive games. Yet, isolating the effect of a specific sensory cue on the emotional state is a difficult feat. The performed experiment is a first of a series that aims at producing usable guidelines that can be applied to reproducing similar emotional responses, as well as the methods to measure the effects. As such, we are interested in methodologies to both design effective stimuli, and assess the quality and effect thereof. We start with identifying main challenges and the followed methodology. Thereafter, we closely analyze the study results to address some of the challenges, and identify where the potential is for improving the induced stimuli (cause) and effect, as well as the analytical methods used to pinpoint the extent of the effect.

[1]  Tom Carey,et al.  Engagement as a Design Concept for Multimedia , 1995 .

[2]  R. Sekuler,et al.  Sound alters visual motion perception , 1997, Nature.

[3]  Holger Regenbrecht,et al.  Measuring the Sense of Presence and its Relations to Fear of Heights in Virtual Environments , 1998, Int. J. Hum. Comput. Interact..

[4]  Ernst Kruijff,et al.  Human-potential driven design of 3D user interfaces , 2013, 2013 23rd International Conference on Artificial Reality and Telexistence (ICAT).

[5]  Gabriella Kazai,et al.  Towards a science of user engagement. , 2011 .

[6]  Mel Slater,et al.  Depth of Presence in Virtual Environments , 1994, Presence: Teleoperators & Virtual Environments.

[7]  Ville Pulkki,et al.  Spatial sound generation and perception by amplitude panning techniques , 2001 .

[8]  Brian McDonald,et al.  Intelligent Biofeedback using an Immersive Competitive Environment , 2001 .

[9]  Rosalind W. Picard.  The Galvactivator: A glove that senses and communicates skin conductivity , 2001 .

[10]  Steffi Beckhaus,et al.  How to not hit a virtual wall: aural spatial awareness for collision avoidance in virtual environments , 2011, AM '11.

[11]  Ellen Yi-Luen Do,et al.  Simulating the sensation of taste for immersive experiences , 2013, ImmersiveMe '13.

[12]  Jesse Schell,et al.  Designing Interactive Theme Park Rides , 2001, IEEE Computer Graphics and Applications.

[13]  Michael Meehan,et al.  Physiological measures of presence in stressful virtual environments , 2002, SIGGRAPH.

[14]  Dacher Keltner,et al.  The nature of awe: Elicitors, appraisals, and effects on self-concept , 2007 .

[15]  Jodi Forlizzi,et al.  Understanding experience in interactive systems , 2004, DIS '04.

[16]  S. Shimojo,et al.  Sensory modalities are not separate modalities: plasticity and interactions , 2001, Current Opinion in Neurobiology.

[17]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[18]  Robert W. Lindeman,et al.  A classification scheme for multi-sensory augmented reality , 2007, VRST '07.

[19]  Kenith V. Sobel,et al.  PSYCHOLOGICAL SCIENCE Research Article Neural Synergy Between Kinetic Vision and Touch , 2022 .

[20]  Sidney K. D'Mello,et al.  A Review and Meta-Analysis of Multimodal Affect Detection Systems , 2015, ACM Comput. Surv..

[21]  Wendy S. Ark,et al.  The Emotion Mouse , 1999, HCI.

[22]  Ernst Kruijff,et al.  Unconventional human computer interfaces , 2004, SIGGRAPH '04.

[23]  Yong-Moo Kwon,et al.  The making of Kyongju VR Theatre , 2002, Proceedings IEEE Virtual Reality 2002.

[24]  Robert W. Lindeman,et al.  Haptic ChairIO: A system to study the effect of wind and floor vibration feedback on spatial orientation in VEs , 2015, 2015 IEEE Symposium on 3D User Interfaces (3DUI).

[25]  Rosalind W. Picard.  Affective Computing , 1997 .

[26]  George Drettakis,et al.  Perceptual audio rendering of complex virtual environments , 2004, ACM Trans. Graph..

[27]  Jeffrey Heer,et al.  Balancing exertion experiences , 2012, CHI.

[28]  Lennart E. Nacke,et al.  Flow and immersion in first-person shooters: measuring the player's gameplay experience , 2008, Future Play.

[29]  Mel Slater,et al.  Immersion, presence and performance in virtual environments: an experiment with tri-dimensional chess , 1996, VRST.

[30]  Felix Hülsmann,et al.  Wind and warmth in virtual reality: implementation and evaluation , 2014, VRIC.

[31]  Takuji Narumi,et al.  Augmented reality flavors: gustatory display based on edible marker and cross-modal interaction , 2011, CHI.

[32]  C. Lenay,et al.  SENSORY SUBSTITUTION: LIMITS AND PERSPECTIVES , 2003 .

[33]  Timo Partala,et al.  Affective Information in Human-Computer Interaction , 2009 .

[34]  Dinesh K. Pai,et al.  Multisensory Interaction: Real and Virtual , 2003, ISRR.

[35]  Doug A. Bowman,et al.  Evaluating Display Fidelity and Interaction Fidelity in a Virtual Reality Game , 2012, IEEE Transactions on Visualization and Computer Graphics.

[36]  Francesco Ricci,et al.  User Modeling, Adaptation, and Personalization , 2013, Lecture Notes in Computer Science.

[37]  Markus H. Gross,et al.  Spatialized audio rendering for immersive virtual environments , 2002, VRST '02.

[38]  Mounia Lalmas,et al.  Models of user engagement , 2012, UMAP.

[39]  Lennart E. Nacke,et al.  Player Typology in Theory and Practice , 2011, DiGRA Conference.

[40]  Gail E. Kaiser,et al.  Proceedings of the Second International Workshop on Games and Software Engineering: Realizing User Engagement with Game Engineering Techniques , 2012, ICSE 2012.

[41]  Stephen Chi-fai Chan,et al.  Physiological mouse: toward an emotion-aware mouse , 2014, 2014 IEEE 38th International Computer Software and Applications Conference Workshops.

[42]  Simon Brown,et al.  Affective gaming: measuring emotion through the gamepad , 2003, CHI Extended Abstracts.

[43]  Janet M. Weisenberger,et al.  Multisensory roughness perception of virtual surfaces: effects of correlated cues , 2004, 12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2004. HAPTICS '04. Proceedings..

[44]  D. R. Heise,et al.  Structure of Emotions , 1988 .

[45]  Alistair G. Sutcliffe.  Multimedia and virtual reality - designing multisensory user interfaces , 2003 .

[46]  C. Spence,et al.  Multisensory Integration: Maintaining the Perception of Synchrony , 2003, Current Biology.

[47]  Peta Wyeth,et al.  GameFlow: a model for evaluating player enjoyment in games , 2005, CIE.

[48]  Steven P. Dow,et al.  Understanding user engagement in immersive and interactive stories , 2008 .

[49]  S. Lederman,et al.  Perception of texture by vision and touch: multidimensionality and intersensory integration. , 1986, Journal of experimental psychology. Human perception and performance.

[50]  M. Ernst,et al.  Feeling what you hear: auditory signals can modulate tactile tap perception , 2005, Experimental Brain Research.

[51]  Jennifer Aaker,et al.  Awe Expands People’s Perception of Time, Alters Decision Making, and Enhances Well-Being , 2012, Psychological science.

[52]  Joseph J. LaViola,et al.  Altering gameplay behavior using stereoscopic 3D vision-based video game design , 2014, CHI.

[53]  Maic Masuch,et al.  Formalizing the potential of stereoscopic 3D user experience in interactive entertainment , 2015, Electronic Imaging.

[54]  Paul A. Cairns,et al.  A grounded investigation of game immersion , 2004, CHI EA '04.

[55]  Shinjiro Kawato,et al.  A nose-tracked, personal olfactory display , 2003, SIGGRAPH '03.

[56]  Kenny Mitchell,et al.  Surround Haptics: sending shivers down your spine , 2011, SIGGRAPH '11.

[57]  Mark Mine.  Towards virtual reality for the masses: 10 years of research at Disney's VR studio , 2003 .

[58]  Jane Fulton Suri,et al.  Experience prototyping , 2000, DIS '00.

[59]  Takuji Narumi,et al.  Meta cookie , 2010, SIGGRAPH '10.

[60]  Joseph Nathaniel Kaye,et al.  Symbolic olfactory display , 2001 .

[61]  Ramya B. Nagaraja.  Converging Technologies for Improving Human Performance , 2004 .

[62]  Kostas Karpouzis,et al.  Estimation of behavioral user state based on eye gaze and head pose—application in an e-learning environment , 2009, Multimedia Tools and Applications.

[63]  Elaine Toms,et al.  Examining the generalizability of the User Engagement Scale (UES) in exploratory search , 2013, Inf. Process. Manag..

[64]  Jonathan Steuer,et al.  Defining virtual reality: dimensions determining telepresence , 1992 .