Can Multisensory Cues in VR Help Train Pattern Recognition to Citizen Scientists?

As the internet of things (IoT) has integrated physical and digital technologies, designing for multiple sensory media (mulsemedia) has become more attainable. Designing technology for multiple senses has the capacity to improve virtual realism, extend our ability to process information, and more easily transfer knowledge between physical and digital environments. HCI researchers are beginning to explore the viability of integrating multimedia into virtual experiences, however research has yet to consider whether mulsemedia truly enhances realism, immersion and knowledge transfer. My work developing StreamBED, a VR training platform to train citizen science water monitors plans to consider the role of mulsemedia in immersion and learning goals. Future findings about the role of mulsemedia in learning contexts will potentially allow learners to experience, connect to, learn from spaces that are impossible to experience firsthand.

[1]  Weisi Lin,et al.  Mulsemedia: State of the Art, Perspectives, and Challenges , 2014, TOMM.

[2]  Jennifer Preece,et al.  StreamBED: Training Citizen Scientists to Make Qualitative Judgments Using Embodied Virtual Reality Training , 2016, CHI Extended Abstracts.

[3]  Frederick P. Brooks,et al.  Project GROPEHaptic displays for scientific visualization , 1990, SIGGRAPH.

[4]  Mel Slater,et al.  Drumming in Immersive Virtual Reality: The Body Shapes the Way We Play , 2013, IEEE Transactions on Visualization and Computer Graphics.

[5]  Michael Haverkamp,et al.  Application of Synesthetic Design as multi-sensory Approach on Sound Quality , 2009 .

[6]  Chris Dede,et al.  Multisensory Immersion as a Modeling Environment for Learning Complex Scientific Concepts , 1999 .

[7]  Ryan S. Elder,et al.  Feminine to smell but masculine to touch? Multisensory congruence and its effect on the aesthetic experience , 2010 .

[8]  Ali Israr,et al.  Feel Effects , 2014, ACM Trans. Appl. Percept..

[9]  Gabriel-Miro Muntean,et al.  Can Multisensorial Media Improve Learner Experience? , 2017, MMSys.

[10]  Orit Shaer,et al.  Reality-based interaction: a framework for post-WIMP interfaces , 2008, CHI.

[11]  George Ghinea,et al.  Multisensory games-based learning - lessons learnt from olfactory enhancement of a digital board game , 2018, Multimedia Tools and Applications.

[12]  Aaron R. Seitz,et al.  Benefits of multisensory learning , 2008, Trends in Cognitive Sciences.

[13]  Charles Spence,et al.  Olfactory cues modulate facial attractiveness. , 2007, Chemical senses.

[14]  Charles Spence,et al.  Multisensory Presence in Virtual Reality: Possibilities & Limitations. , 2011 .

[15]  Midori Tokita,et al.  Perception of the material properties of wood based on vision, audition, and touch , 2015, Vision Research.

[16]  Ju-Hwan Lee,et al.  Assessing the benefits of multimodal feedback on dual-task performance under demanding conditions , 2008, BCS HCI.

[17]  Ramiro Gonçalves,et al.  A multisensory virtual experience model for thematic tourism: A Port wine tourism application proposal , 2017 .

[18]  Carlos Velasco,et al.  Digitizing the chemical senses: Possibilities & pitfalls , 2017, Int. J. Hum. Comput. Stud..

[19]  T. Stanford,et al.  Multisensory integration: current issues from the perspective of the single neuron , 2008, Nature Reviews Neuroscience.

[20]  Ellen Yi-Luen Do,et al.  Ambiotherm: Enhancing Sense of Presence in Virtual Reality by Simulating Real-World Environmental Conditions , 2017, CHI.

[21]  Kevin Crowston,et al.  Mechanisms for Data Quality and Validation in Citizen Science , 2011, 2011 IEEE Seventh International Conference on e-Science Workshops.

[22]  Kenneth R. Koedinger,et al.  Adding Physical Objects to an Interactive Game Improves Learning and Enjoyment , 2016, ACM Trans. Comput. Hum. Interact..

[23]  Neff Walker,et al.  Evaluating the importance of multi-sensory input on memory and the sense of presence in virtual environments , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[24]  Ali Israr,et al.  Sensing the future of HCI , 2016, Interactions.

[25]  Joseph Psotka,et al.  Immersive training systems: Virtual reality and education and training , 1995 .

[26]  Stephen R. Gulliver,et al.  Multiple Sensorial Media Advances and Applications: New Developments in MulSeMedia , 2011 .

[27]  Pedro Lopes,et al.  Impacto: Simulating Physical Impact by Combining Tactile Stimulation with Electrical Muscle Stimulation , 2015, UIST.

[28]  Chris Dede,et al.  Virtual Reality as an Immersive Medium for Authentic Simulations , 2017 .

[29]  Hiroo Iwata,et al.  Food simulator: a haptic interface for biting , 2004, IEEE Virtual Reality 2004.

[30]  Muawiyath Shujau,et al.  Analysing the Quality of Experience of multisensory media from measurements of physiological responses , 2014, 2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX).

[31]  Lynda Joy Gerry,et al.  Paint with Me: Stimulating Creativity and Empathy While Painting with a Painter in Virtual Reality , 2017, IEEE Transactions on Visualization and Computer Graphics.

[32]  R. Bonney,et al.  Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy , 2009 .