Sensory Alignment in Immersive Entertainment

When we use digital systems to stimulate the senses, we typically stimulate only a subset of users' senses, leaving other senses stimulated by the physical world. This creates potential for misalignment between senses, where digital and physical stimulation give conflicting signals to users. We synthesize knowledge from HCI, traditional entertainments, and underlying sensory science research relating to how senses work when given conflicting signals. Using this knowledge we present a design dimension of sensory alignment, and show how this dimension presents opportunities for a range of creative strategies ranging from full alignment of sensory stimulation, up to extreme conflict between senses.

[1]  Larissa Hjorth,et al.  TastyBeats: Designing Palatable Representations of Physical Activity , 2015, CHI.

[2]  Jared Medina,et al.  Influence of the Body Schema on Multisensory Integration: Evidence from the Mirror Box Illusion , 2017, Scientific Reports.

[3]  S. Shimojo,et al.  Visual illusion induced by sound. , 2002, Brain research. Cognitive brain research.

[4]  Mary C. Whitton,et al.  Redirected touching: The effect of warping space on task performance , 2012, 2012 IEEE Symposium on 3D User Interfaces (3DUI).

[5]  Brian L Day,et al.  Probing the human vestibular system with galvanic stimulation. , 2004, Journal of applied physiology.

[6]  Joe Marshall,et al.  Abstract Machines: Overlaying Virtual Worlds on Physical Rides , 2019, CHI.

[7]  Gerd Bruder,et al.  Estimation of Detection Thresholds for Redirected Walking Techniques , 2010, IEEE Transactions on Visualization and Computer Graphics.

[8]  V. Jousmäki,et al.  Parchment-skin illusion: sound-biased touch , 1998, Current Biology.

[9]  Hans-Werner Gellersen,et al.  Substitutional Reality: Using the Physical Environment to Design Virtual Reality Experiences , 2015, CHI.

[10]  Stephen A. Brewster,et al.  I Am The Passenger: How Visual Motion Cues Can Influence Sickness For In-Car VR , 2017, CHI.

[11]  Ellen Yi-Luen Do,et al.  Simulating the sensation of taste for immersive experiences , 2013, ImmersiveMe '13.

[12]  B. Green,et al.  Studying taste as a cutaneous sense , 2003 .

[13]  Hiroyuki Kajimoto,et al.  Effect of Electrical Stimulation Haptic Feedback on Perceptions of Softness-Hardness and Stickiness While Touching a Virtual Object , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[14]  Ellen Yi-Luen Do,et al.  Virtual ingredients for food and beverages to create immersive taste experiences , 2015, Multimedia Tools and Applications.

[15]  P. Mamassian,et al.  Multisensory processing in review: from physiology to behaviour. , 2010, Seeing and perceiving.

[16]  Patrick Haggard,et al.  Thermal referral: evidence for a thermoceptive uniformity illusion without touch , 2016, Scientific Reports.

[17]  R. W. Wood,et al.  The 'Haunted Swing' illusion. , 1895 .

[18]  M. Heller.  Visual and tactual texture perception: Intersensory cooperation , 1982, Perception & psychophysics.

[19]  D. Valentin,et al.  Taste-odour interactions in sweet taste perception , 2006 .

[20]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.

[21]  Fumio Kishino,et al.  Augmented reality: a class of displays on the reality-virtuality continuum , 1995, Other Conferences.

[22]  Carlos Velasco,et al.  Digitizing the chemical senses: Possibilities & pitfalls , 2017, Int. J. Hum. Comput. Stud..

[23]  G. Kenyon.  Six scales for assessing attitude toward physical activity. , 1968, Research quarterly.

[24]  Pattie Maes,et al.  VMotion: Designing a Seamless Walking Experience in VR , 2018, Conference on Designing Interactive Systems.

[25]  Daniel Västfjäll,et al.  Perception of Self-motion and Presence in Auditory Virtual Environments , 2004 .

[26]  R. Dolan,et al.  The Nose Smells What the Eye Sees Crossmodal Visual Facilitation of Human Olfactory Perception , 2003, Neuron.

[27]  Stefan Rennick Egglestone,et al.  Performing thrill: designing telemetry systems and spectator interfaces for amusement rides , 2008, CHI.

[28]  D. Dubourdieu,et al.  The Color of Odors , 2001, Brain and Language.

[29]  Julie Greensmith,et al.  Personalizing the theme park: psychometric profiling and physiological monitoring , 2011, UMAP'11.

[30]  Sang-Youn Kim,et al.  RealWalk: Feeling Ground Surfaces While Walking in Virtual Reality , 2018, CHI Extended Abstracts.

[31]  Thomas H. Massie,et al.  The PHANToM Haptic Interface: A Device for Probing Virtual Objects , 1994 .

[32]  Dennis Proffitt,et al.  Quantifying immersion in virtual reality , 1997, SIGGRAPH.

[33]  Takuji Narumi,et al.  Walking uphill and downhill: redirected walking in the vertical direction , 2017, SIGGRAPH Posters.

[34]  C. Spence,et al.  Does Food Color Influence Taste and Flavor Perception in Humans? , 2010 .

[35]  Tao Zhang,et al.  Smelling directions: Olfaction modulates ambiguous visual motion perception , 2014, Scientific Reports.

[36]  Eyal Ofek,et al.  Sparse Haptic Proxy: Touch Feedback in Virtual Environments Using a General Passive Prop , 2017, CHI.

[37]  Pedro Lopes,et al.  Proprioceptive Interaction , 2015, CHI.

[38]  T. Stoffregen,et al.  Control of a virtual vehicle influences postural activity and motion sickness. , 2011, Journal of experimental psychology. Applied.

[39]  Tony P. Pridmore,et al.  Deception and magic in collaborative interaction , 2010, CHI.

[40]  Andreas Georgiou,et al.  Holographic near-eye displays for virtual and augmented reality , 2017, ACM Trans. Graph..

[41]  R. P. Power,et al.  Dominance of Touch by Vision: Generalization of the Hypothesis to a Tactually Experienced Population , 1976, Perception.

[42]  C. Spence.  Eating with our ears: assessing the importance of the sounds of consumption on our perception and enjoyment of multisensory flavour experiences , 2015, Flavour.

[43]  Eyal Ofek,et al.  Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences , 2016, CHI.

[44]  Blake S. Wilson,et al.  Cochlear implants: A remarkable past and a brilliant future , 2008, Hearing Research.

[45]  Mary C. Whitton,et al.  Passive haptics significantly enhances virtual environments , 2001 .

[46]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.

[47]  Henry Been-Lirn Duh,et al.  “Conflicting” Motion Cues to the Visual and Vestibular Self- Motion Systems Around 0.06 Hz Evoke Simulator Sickness , 2004, Hum. Factors.

[48]  K. Durand,et al.  The Nose Tells it to the Eyes: Crossmodal Associations between Olfaction and Vision , 2010, Perception.

[49]  M. Zuckerman,et al.  Sensation Seeking And Risky Behavior , 2006 .

[50]  N. Troje,et al.  Influence of bone-conducted vibration on simulator sickness in virtual reality , 2018, PloS one.

[51]  Anjul Patney,et al.  Towards virtual reality infinite walking , 2018, ACM Trans. Graph..

[52]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[53]  M. Reiner,et al.  Sensory dominance in combinations of audio, visual and haptic stimuli , 2009, Experimental Brain Research.

[54]  C. Spence.  Multisensory Flavor Perception , 2015, Cell.

[55]  Pedro Lopes,et al.  Impacto: Simulating Physical Impact by Combining Tactile Stimulation with Electrical Muscle Stimulation , 2015, UIST.

[56]  Hiroyuki Kajimoto,et al.  Development of a wearable haptic device that presents haptics sensation of the finger pad to the forearm , 2018, 2018 IEEE Haptics Symposium (HAPTICS).

[57]  Charles B. Owen,et al.  Review on cybersickness in applications and visual displays , 2016, Virtual Reality.

[58]  V. C. Coffey.  Vision Accomplished: The Bionic Eye , 2017 .

[59]  Sriram Subramanian,et al.  UltraHaptics: multi-point mid-air haptic feedback for touch surfaces , 2013, UIST.

[60]  Sharif Razzaque,et al.  Redirected Walking , 2001, Eurographics.

[61]  Joe Marshall,et al.  Balance Ninja: Towards the Design of Digital Vertigo Games via Galvanic Vestibular Stimulation , 2016, CHI PLAY.

[62]  P. Bertelson,et al.  The After-Effects of Ventriloquism , 1974, The Quarterly journal of experimental psychology.

[63]  Joe Marshall,et al.  AR Fighter: Using HMDs to create Vertigo Play Experiences , 2018, CHI PLAY.

[64]  Ali Israr,et al.  Sensing the future of HCI , 2016, Interactions.

[65]  Joe Marshall,et al.  The Challenges of Visual-Kinaesthetic Experience , 2017, Conference on Designing Interactive Systems.

[66]  Sarah Sharples,et al.  Virtual reality induced symptoms and effects (VRISE): Comparison of head mounted display (HMD), desktop and projection display systems , 2008, Displays.

[67]  D. Small,et al.  Odor/taste integration and the perception of flavor , 2005, Experimental Brain Research.

[68]  T SHIPLEY,et al.  Auditory Flutter-Driving of Visual Flicker , 1964, Science.

[69]  Stefan Rennick Egglestone,et al.  Breath control of amusement rides , 2011, CHI.

[70]  Joe Marshall,et al.  Uncomfortable interactions , 2012, CHI.

[71]  Ali Israr,et al.  AIREAL: interactive tactile experiences in free air , 2013, ACM Trans. Graph..

[72]  R. Fitzpatrick,et al.  Resolving Head Rotation for Human Bipedalism , 2006, Current Biology.

[73]  Eric Burns,et al.  Combining passive haptics with redirected walking , 2005, ICAT '05.