Effects of mixed-reality on players' behaviour and immersion in a cultural tourism game: A cognitive processing perspective

Abstract Mixed-reality environments introduce innovative human-computer interaction paradigms assisted by enhanced visual content presentation which require from end-users to perform excessive cognitive tasks related to visual attention, search, processing, and comprehension. In such visually enriched interaction realms, individual differences in perception and visual information processing might affect users’ behaviour and immersion, given that such effects are known to exist in conventional computer environments, like desktop or mobile. In an attempt to shed light on whether, how, and why such effects persist within mixed-reality contexts, we conducted a between-subjects eye-tracking study (N=73) in which users interacted within either a conventional or a mixed-reality technological context, and adopted an accredited cognitive style theory to interpret the derived results. Analysis of results yielded that mixed-reality interaction realms amplified the effects of human cognitive style towards game-specific interaction behaviour and visual behaviour. Findings further support the added value of incorporating human cognitive factors in both design and run-time, aiming to provide adaptive and personalised features to end-users within mixed-reality interaction contexts. Such practical implications are also discussed in this paper.

[1]  PietroniEva,et al.  Interacting with Virtual Reconstructions in Museums , 2014 .

[2]  Tanguy Coenen,et al.  MuseUs: Case study of a pervasive cultural heritage serious game , 2013, JOCCH.

[3]  Andrew Y. C. Nee,et al.  Augmented reality for assembly guidance using a virtual interactive tool , 2008 .

[4]  C. Angeli,et al.  Examining the effects of text-only and text-and-visual instructional materials on the achievement of field-dependent and field-independent learners during problem-solving with modeling software , 2004 .

[5]  Gary F. Templeton A Two-Step Approach for Transforming Continuous Variables to Normal: Implications and Recommendations for IS Research , 2011, Commun. Assoc. Inf. Syst..

[6]  Conrad S. Tucker,et al.  Investigating the Impact of Interactive Immersive Virtual Reality Environments in Enhancing Task Performance in Online Engineering Design Activities , 2015 .

[7]  Josef Froschauer,et al.  A serious heritage game for art history: Design and evaluation of ThIATRO , 2012, 2012 18th International Conference on Virtual Systems and Multimedia.

[8]  Fotis Liarokapis,et al.  Developing serious games for cultural heritage: a state-of-the-art review , 2010, Virtual Reality.

[9]  Andrew Laghos,et al.  Field Dependence-Independence and Eye Movement Patterns: Investigating Users' Differences Through an Eye Tracking Study , 2016, Interact. Comput..

[10]  Naokazu Yokoya,et al.  Mixed-Reality World Exploration Using Image-Based Rendering , 2015, ACM Journal on Computing and Cultural Heritage.

[11]  Sherry Y. Chen,et al.  The influences of an interactive group-based videogame: Cognitive styles vs. prior ability , 2015, Comput. Educ..

[12]  Marios Belk,et al.  Using Eye Gaze Data and Visual Activities to Infer Human Cognitive Styles: Method and Feasibility Studies , 2017, UMAP.

[13]  Giorgos Lepouras,et al.  Personalized augmented reality experiences in museums using Google Cardboards , 2017, 2017 12th International Workshop on Semantic and Social Media Adaptation and Personalization (SMAP).

[14]  Rudy McDaniel,et al.  Evaluating the Relationship between Cognitive Style and Pre-Service Teachers' Preconceived Notions about Adopting Console Video Games for Use in Future Classrooms , 2013, Int. J. Game Based Learn..

[15]  Pejman Mirza-Babaei,et al.  More than Meets the Eye: The Benefits of Augmented Reality and Holographic Displays for Digital Cultural Heritage , 2017, JOCCH.

[16]  C. A. Moore,et al.  FIELD‐DEPENDENT AND FIELD‐INDEPENDENT COGNITIVE STYLES AND THEIR EDUCATIONAL IMPLICATIONS , 1975 .

[17]  R. Riding,et al.  Cognitive Styles—an overview and integration , 1991 .

[18]  James L. C. Wijnen,et al.  An eye movement analysis system (EMAS) for the identification of cognitive processes on figural tasks , 1984 .

[19]  Stephen R. Serge,et al.  Mixed Reality Training of Military Tasks: Comparison of Two Approaches Through Reactions from Subject Matter Experts , 2016, HCI.

[20]  Andrew K. Przybylski,et al.  The Motivational Pull of Video Games: A Self-Determination Theory Approach , 2006 .

[21]  Mehmet Bahar,et al.  The Relationship Between Some Psychological Factors and their Effect on the Performance of Grid Questions and Word Association Tests , 2000 .

[22]  Nikolaos M. Avouris,et al.  Using Eye Tracking to Identify Cognitive Differences: A Brief Literature Review , 2016, PCI.

[23]  Nikolaos M. Avouris,et al.  Cultural Heritage Gaming: Effects of Human Cognitive Styles on Players' Performance and Visual Behavior , 2017, UMAP.

[24]  Anton Bogdanovych,et al.  Case-based planning for large virtual agent societies , 2017, VRST.

[25]  Eugene Ch'ng,et al.  Experiential archaeology: Is virtual time travel possible? , 2009 .

[26]  Joseph R. Fanfarelli,et al.  Individual Differences in Digital Badging: Do Learner Characteristics Matter? , 2015 .

[27]  Marios Belk,et al.  Influences of Human Cognition and Visual Behavior on Password Strength during Picture Password Composition , 2018, CHI.

[28]  Tsvi Kuflik,et al.  Exploring the potential of a mobile eye tracker as an intuitive indoor pointing device: A case study in cultural heritage , 2018, Future Gener. Comput. Syst..

[29]  Samuel S. Silva,et al.  Head-mounted display versus desktop for 3D navigation in virtual reality: a user study , 2008, Multimedia Tools and Applications.

[30]  Cristina Conati,et al.  User-adaptive information visualization: using eye gaze data to infer visualization tasks and user cognitive abilities , 2013, IUI '13.

[31]  Fotis Liarokapis,et al.  The Herbert Virtual Museum , 2013, J. Electr. Comput. Eng..

[32]  R. Mayer,et al.  Three Facets of Visual and Verbal Learners: Cognitive Ability, Cognitive Style, and Learning Preference. , 2003 .

[33]  Frank Biocca,et al.  Comparative effectiveness of augmented reality in object assembly , 2003, CHI '03.

[34]  Panagiotis Petridis,et al.  Learning cultural heritage by serious games , 2014 .

[35]  D. Chapman,et al.  Validation of learning style measures: implications for medical education practice , 2006, Medical education.

[36]  Xuqun You,et al.  Analysis of Mental Workload in Online Shopping: Are Augmented and Virtual Reality Consistent? , 2017, Front. Psychol..

[37]  Julian Frommel,et al.  Evaluating VR Driving Simulation from a Player Experience Perspective , 2017, CHI Extended Abstracts.

[38]  Gastón Ares,et al.  Influence of cognitive style on information processing and selection of yogurt labels: Insights from an eye-tracking study. , 2015, Food research international.

[39]  Miriam Reiner,et al.  Field Dependency and the Sense of Object-Presence in Haptic Virtual Environments , 2007, Cyberpsychology Behav. Soc. Netw..

[40]  Tsvi Kuflik,et al.  Towards Using Mobile, Head-Worn Displays in Cultural Heritage: User Requirements and a Research Agenda , 2016, IUI.

[41]  Yoones A. Sekhavat,et al.  Sense of Immersion in Computer Games Using Single and Stereoscopic Augmented Reality , 2018, Int. J. Hum. Comput. Interact..

[42]  Arif Altun,et al.  Undergraduate Students\u2019 Academic Achievement, Field Dependent/Independent Cognitive Styles and Attitude toward Computers , 2006, J. Educ. Technol. Soc..

[43]  Jon-Chao Hong,et al.  Effects of cognitive style on digital jigsaw puzzle performance: A GridWare analysis , 2012, Comput. Hum. Behav..

[44]  Sarah Sharples,et al.  Virtual reality induced symptoms and effects (VRISE): Comparison of head mounted display (HMD), desktop and projection display systems , 2008, Displays.

[45]  Louise Moody,et al.  The feasibility of a mixed reality surgical training environment , 2008, Virtual Reality.

[46]  Gang Hu,et al.  Doing While Thinking: Physical and Cognitive Engagement and Immersion in Mixed Reality Games , 2016, Conference on Designing Interactive Systems.

[47]  W. Wirth,et al.  More than meets the eye , 2007 .

[48]  Siam Charoenseang,et al.  Augmented reality for skill transfer in assembly task , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[49]  Cristina Conati,et al.  Te, Te, Hi, Hi: Eye Gaze Sequence Analysis for Informing User-Adaptive Information Visualizations , 2014, UMAP.

[50]  Judi Repman,et al.  The Relationship of Learning, Behavior, and Cognitive Style in Hypermedia-Based Instruction: Implications for Design of HBI , 1995, Planning Continuing Professional Development.

[51]  M. Kozhevnikov Cognitive styles in the context of modern psychology: toward an integrated framework of cognitive style. , 2007, Psychological bulletin.

[52]  Francis M. Dwyer,et al.  The Effect of Online Gaming, Cognition and Feedback Type in Facilitating Delayed Achievement of Different Learning Objectives , 2005 .

[53]  Paul A. Kirschner,et al.  Field dependence-independence and instructional-design effects on learners' performance with a computer-modeling tool , 2009, Comput. Hum. Behav..

[54]  Tuck Wah Leong,et al.  Exploring Gameplay Experiences on the Oculus Rift , 2015, CHI PLAY.

[55]  Kasia Muldner,et al.  Diving Head-First into Virtual Reality: Evaluating HMD Control Schemes for VR Games , 2015, FDG.

[56]  Gavriel Salvendy,et al.  Effect of evaluators' cognitive style on heuristic evaluation: Field dependent and field independent evaluators , 2009, Int. J. Hum. Comput. Stud..

[57]  Nikolaos M. Avouris,et al.  Differences of Field Dependent/Independent Gamers on Cultural Heritage Playing: Preliminary Findings of an Eye-Tracking Study , 2016, EuroMed.

[58]  H WittenIan,et al.  The WEKA data mining software , 2009 .

[59]  Martin White,et al.  Exploring the relationship between presence and enjoyment in a virtual museum , 2010, Int. J. Hum. Comput. Stud..

[60]  Yoones A. Sekhavat,et al.  Enhancing the Sense of Immersion and Quality of Experience in Mobile Games Using Augmented Reality , 2017 .

[61]  Edward E. Cureton,et al.  The upper and lower twenty-seven per cent rule , 1957 .

[62]  Yi-Ping Hung,et al.  Interacting with the past: Creating a time perception journey experience using kinect-based breath detection and deterioration and recovery simulation technologies , 2014, JOCCH.

[63]  Wen Gao,et al.  Annotating traditional Chinese paintings for immersive virtual exhibition , 2012, JOCCH.

[64]  A. Imran Nordin,et al.  Too Many Questionnaires: Measuring Player Experience Whilst Playing Digital Games , 2014 .

[65]  Tsvi Kuflik,et al.  Listen to What You Look at: Combining an Audio Guide with a Mobile Eye Tracker on the Go , 2016, AI*CH@AI*IA.

[66]  W. M. Reed,et al.  The relationship between the learning strategies and learning styles in a hypermedia environment , 1994 .

[67]  Oleg V. Komogortsev,et al.  Standardization of Automated Analyses of Oculomotor Fixation and Saccadic Behaviors , 2010, IEEE Transactions on Biomedical Engineering.

[68]  Nikolaos M. Avouris,et al.  On Implicit Elicitation of Cognitive Strategies using Gaze Transition Entropies in Pattern Recognition Tasks , 2017, CHI Extended Abstracts.

[69]  Andrew Laghos,et al.  Using Eye Tracking to Understand the Impact of Cognitive Abilities on Search Tasks , 2014, HCI.

[70]  Rasoul Mohammad Hosseinpur,et al.  On the Validity of the Group Embedded Figure Test (GEFT) , 2011 .

[71]  Bonnie MacKay,et al.  Operation Citadel: Exploring the Role of Docents in Mixed Reality , 2015, CHI PLAY.

[72]  Jeffrey C. F. Ho Effect of Real-World Experience on Immersion in Virtual Reality Games: A Preliminary Study , 2016, CCHI.

[73]  R. T. Pithers,et al.  Cognitive learning style: a review of the field dependent-field independent approach , 2002 .

[74]  Steven K. Feiner,et al.  Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[75]  Jean-Michel Hoc,et al.  Cognitive styles as an explanation of experts' individual differences: A case study in computer-assisted troubleshooting diagnosis , 2006, Int. J. Hum. Comput. Stud..

[76]  Panagiotis Germanakos,et al.  Do human cognitive differences in information processing affect preference and performance of CAPTCHA? , 2015, Int. J. Hum. Comput. Stud..

[77]  Randy Pausch,et al.  A user study comparing head-mounted and stationary displays , 1993, Proceedings of 1993 IEEE Research Properties in Virtual Reality Symposium.

[78]  Marco Roccetti,et al.  Diegetic user interfaces for virtual environments with HMDs: a user experience study with oculus rift , 2017, Journal on Multimodal User Interfaces.

[79]  Xavier Granier,et al.  The Revealing Flashlight: Interactive Spatial Augmented Reality for Detail Exploration of Cultural Heritage Artifacts , 2014, JOCCH.

[80]  Jetmir Xhembulla,et al.  Integrating a Location-Based Mobile Game in the Museum Visit , 2015, ACM Journal on Computing and Cultural Heritage.

[81]  Nikolaos M. Avouris,et al.  Do Field Dependence-Independence Differences of Game Players Affect Performance and Behaviour in Cultural Heritage Games? , 2016, CHI PLAY.

[82]  Jennifer Ann Rode,et al.  Using a large projection screen as an alternative to head-mounted displays for virtual environments , 2000, CHI.

[83]  Dietrich Albert,et al.  Tracking the UFO's Paths: Using Eye-Tracking for the Evaluation of Serious Games , 2011, HCI.

[84]  S. Moffat,et al.  Navigation in a “Virtual” Maze: Sex Differences and Correlation With Psychometric Measures of Spatial Ability in Humans , 1998 .

[85]  Jonathan Stevens,et al.  The effect of visual display on performance in mixed reality simulation , 2015 .

[86]  Frank Biocca,et al.  Experimental evaluation of augmented reality in object assembly task , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[87]  Eva Pietroni,et al.  Interacting with Virtual Reconstructions in Museums: The Etruscanning Project , 2014, JOCCH.

[88]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[89]  S. A. Karp,et al.  Stability of cognitive style from childhood to young adulthood. , 1967, Journal of personality and social psychology.

[90]  Claus B. Madsen,et al.  Handheld Visual Representation of a Castle Chapel Ruin , 2016, ACM Journal on Computing and Cultural Heritage.

[91]  Sherry Y. Chen,et al.  Cognitive styles and the use of electronic journals in a mobile context , 2014, J. Documentation.

[92]  Massimo Bergamasco,et al.  Evaluating the Impact of Highly Immersive Technologies and Natural Interaction on Player Engagement and Flow Experience in Games , 2015, ICEC.

[93]  Diane Gromala,et al.  Immersion in Cardboard VR Compared to a Traditional Head-Mounted Display , 2016, HCI.

[94]  Barbara Odenthal,et al.  A Comparative Study of Head‐Mounted and Table‐Mounted Augmented Vision Systems for Assembly Error Detection , 2014 .

[95]  RubinoIrene,et al.  Integrating a Location-Based Mobile Game in the Museum Visit , 2015 .

[96]  Helen Sharp,et al.  The Influence of Cognitive Style, Design Setting and Cultural Background on Sketch-Based Ideation by Novice Interaction Designers , 2017 .

[97]  Josef Froschauer,et al.  Art history concepts at play with ThIATRO , 2013, JOCCH.

[98]  Marios Belk,et al.  Security and Usability in Knowledge-based User Authentication: A Review , 2016, PCI.

[99]  Paul A. Cairns,et al.  Measuring and defining the experience of immersion in games , 2008, Int. J. Hum. Comput. Stud..

[100]  Orit Shaer,et al.  HoloMuse: Enhancing Engagement with Archaeological Artifacts through Gesture-Based Interaction with Holograms , 2017, TEI.

[101]  Panagiotis Germanakos,et al.  The interplay between humans, technology and user authentication: A cognitive processing perspective , 2017, Comput. Hum. Behav..

[102]  Paul A. Cairns,et al.  The Convergence of Player Experience Questionnaires , 2016, CHI PLAY.