Multisensory Experiences in Virtual Reality and Augmented Reality Interaction Paradigms

Multisensory stimuli can be integrated in systems that make use of different paradigms, such as Virtual Reality (VR), Augmented Reality (AR) or, in a wider sense, Mixed Reality (MR), enhancing user experiences within the virtual content. However, despite the many technological solutions that exist (both hardware and software), only visual and sonic stimuli can be considered as highly integrated in consumer-grade applications. This chapter addresses the current state of the art in multisensory experiences, taking also in consideration the aforementioned interaction paradigms, and brings the benefits and challenges. As an example, authors introduce ROMOT, a RObotic 3D-MOvie Theatre, that supports and integrates various types of displays and interactive applications, providing users with multisensory experiences. Multisensory Experiences in Virtual Reality and Augmented Reality Interaction Paradigms

[1]  M. Murray,et al.  The role of multisensory memories in unisensory object discrimination. , 2005, Brain research. Cognitive brain research.

[2]  Ramiro Gonçalves,et al.  A multisensory virtual experience model for thematic tourism: A Port wine tourism application proposal , 2017 .

[3]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[4]  Farzad Towhidkhah,et al.  Comparison of visual and proprioceptive training on multisensory perception using a new designed setup , 2013, 2013 21st Iranian Conference on Electrical Engineering (ICEE).

[5]  Teresa Galvão Dias,et al.  Average Speed of Public Transport Vehicles Based on Smartcard Data , 2020 .

[6]  R. Baños,et al.  Treating cockroach phobia with augmented reality. , 2010, Behavior therapy.

[7]  Yu Guo,et al.  A Virtual Try-On System for Prescription Eyeglasses , 2017, IEEE Computer Graphics and Applications.

[8]  Ali Israr,et al.  Sensing the future of HCI , 2016, Interactions.

[9]  Sergio Casas,et al.  On a First Evaluation of ROMOT - A RObotic 3D MOvie Theatre - For Driving Safety Awareness , 2017, Multimodal Technol. Interact..

[10]  Aaron R. Seitz,et al.  Benefits of Stimulus Congruency for Multisensory Facilitation of Visual Learning , 2008, PloS one.

[11]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[12]  Adrian David Cheok,et al.  Digital taste and smell communication , 2011, BODYNETS.

[13]  Louise P. Kirsch,et al.  Balancing body ownership: Visual capture of proprioception and affectivity during vestibular stimulation , 2018, Neuropsychologia.

[14]  Lennart E. Nacke,et al.  Physiological acrophobia evaluation through in vivo exposure in a VR CAVE , 2014, 2014 IEEE Games Media Entertainment.

[15]  Adrian David Cheok,et al.  Welcome to MTI—A New Open Access Journal Dealing with Blue Sky Research and Future Trends in Multimodal Technologies and Interaction , 2016 .

[16]  Nilanjan Dey,et al.  Motion Cueing Algorithms: A Review: Algorithms, Evaluation and Tuning , 2017 .

[17]  David Aaron,et al.  A New Concept , 1974 .

[18]  Takuji Narumi,et al.  Augmented reality flavors: gustatory display based on edible marker and cross-modal interaction , 2011, CHI.

[19]  Frank Weichert,et al.  Analysis of the Accuracy and Robustness of the Leap Motion Controller , 2013, Sensors.

[20]  Morton Leonard Heilig,et al.  EL Cine del Futuro: The Cinema of the Future , 1992, Presence: Teleoperators & Virtual Environments.

[21]  Katashi Nagao,et al.  The world through the computer: computer augmented interaction with real world environments , 1995, UIST '95.

[22]  Hiroshi Ishii,et al.  Iterative design of seamless collaboration media , 1994, CACM.

[23]  Sergio Casas,et al.  On the Use of ROMOT - A RObotized 3D-MOvie Theatre - To Enhance Romantic Movie Scenes , 2017, Multimodal Technol. Interact..

[24]  Raffaele Di Fuccio,et al.  Digital and Multisensory Storytelling: Narration with Smell, Taste and Touch , 2016, EC-TEL.

[25]  Cristina Portalés,et al.  Sound and Movement Visualization in the AR-Jazz Scenario , 2009, ICEC.

[26]  Carolina Cruz-Neira,et al.  Virtual Reality and Games , 2018, Multimodal Technol. Interact..

[27]  Aaron R. Seitz,et al.  Benefits of multisensory learning , 2008, Trends in Cognitive Sciences.

[28]  Sergio Casas,et al.  MIME: A Mixed-Space Collaborative System with Three Immersion Levels and Multiple Users , 2018, 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct).

[29]  Min-Chun Hu,et al.  Immersive virtual reality with multimodal interaction and streaming technology , 2016, ICMI.

[30]  Inmaculada Coma,et al.  Combining traditional and indirect augmented reality for indoor crowded environments. A case study on the Casa Batlló museum , 2017, Comput. Graph..

[31]  Sergio Casas,et al.  Interacting with Augmented Reality Mirrors , 2016 .

[32]  Warren Robinett.  Interactivity and individual viewpoint in shared virtual worlds: the big screen vs. networked personal displays , 1994, COMG.

[33]  A. Paivio.  Dual coding theory: Retrospect and current status. , 1991 .

[34]  Inmaculada Coma,et al.  Towards a simulation-based tuning of motion cueing algorithms , 2016, Simul. Model. Pract. Theory.

[35]  Rob Comber,et al.  Temporal, affective, and embodied characteristics of taste experiences: a framework for design , 2014, CHI.

[36]  Brian Yecies.  Transnational collaboration of the multisensory kind: Exploiting Korean 4D cinema in China , 2016 .

[37]  Cristina Portalés,et al.  AR-Immersive Cinema at the Aula Natura Visitors Center , 2010, IEEE MultiMedia.

[38]  Dieter Kranzlmuller,et al.  State of the art of virtual reality technology , 2016, 2016 IEEE Aerospace Conference.

[39]  Inmaculada Coma,et al.  Augmented Mirror: Interactive Augmented Reality System Based on Kinect , 2011, INTERACT.

[40]  Pedro Morillo,et al.  On the Design of a Mixed-Reality Annotations Tool for the Inspection of Pre-fab Buildings , 2018, CEIG.

[41]  Sergio Casas,et al.  A particle swarm approach for tuning washout algorithms in vehicle simulators , 2018, Appl. Soft Comput..

[42]  Rhonda Poynter,et al.  The Night Café , 2009 .

[43]  Jesús Gimeno,et al.  Multiuser Augmented Reality System for Indoor Exhibitions , 2011, INTERACT.

[44]  Didier Stricker,et al.  Archeoguide: An Augmented Reality Guide for Archaeological Sites , 2002, IEEE Computer Graphics and Applications.

[45]  Weiliang Xu,et al.  Multimodal Haptic Display for Virtual Reality: A Survey , 2020, IEEE Transactions on Industrial Electronics.

[46]  Ajune Wanis Ismail,et al.  Designing an Augmented Reality Multimodal Interface for 6DOF Manipulation Techniques - Multimodal Fusion Using Gesture and Speech Input for AR , 2018, IntelliSys.