Investigating Effects and User Preferences of Extra- and Intradiegetic Virtual Reality Questionnaires

Virtual realities (VR) are becoming an integral part of product development across many industries, for example to assess aesthetics and usability of new features in the automotive industry. The recording of the evaluation is typically conducted by filling out questionnaires after the study participants left the virtual environment. In this paper, we investigate how questionnaires can be best embedded within the virtual environment and compare how VR-questionnaires differ from classical post-test evaluations regarding preference, presence, and questionnaire completion time. In the first study (N = 11), experts rated four design concepts of questionnaires embedded in VR, of which two were designed as extradiegetic and two as intradiegetic user interfaces. We show that intradiegetic UIs have a significantly higher perceived user experience and presence while the usability remains similar. Intradiegetic UIs are preferred by the majority. Based on these findings, we compared intradiegetic VR-questionnaires with paper-based evaluations in a follow up study (N = 24). 67% of the participants preferred the evaluation in VR, even though it takes significantly longer. We found no effect on presence.

[1]  Antonio Krüger,et al.  Enhancing Texture Perception in Virtual Reality Using 3D-Printed Hair Structures , 2019, CHI.

[2]  Marc Erich Latoschik,et al.  Construction of a Validated Virtual Embodiment Questionnaire , 2019, ArXiv.

[3]  Dick de Waard,et al.  A simple procedure for the assessment of acceptance of advanced transport telematics , 1997 .

[4]  Mary C. Whitton,et al.  A Survey of Presence and Related Concepts , 2017, ACM Comput. Surv..

[5]  Florian Alt,et al.  Should I Interrupt or Not?: Understanding Interruptions in Head-Mounted Display Settings , 2019, Conference on Designing Interactive Systems.

[6]  Niels Henze,et al.  "These are not my hands!": Effect of Gender on the Perception of Avatar Hands in Virtual Reality , 2017, CHI.

[7]  Niels Henze,et al.  "Where's Pinky?": The Effects of a Reduced Number of Fingers in Virtual Reality , 2017, CHI PLAY.

[8]  Jon A. Krosnick,et al.  Comparing Oral Interviewing with Self-Administered Computerized QuestionnairesAn Experiment , 2010 .

[9]  Marc Erich Latoschik,et al.  Usability of Gamified Knowledge Learning in VR and Desktop-3D , 2019, CHI.

[10]  Mel Slater,et al.  A Virtual Presence Counter , 2000, Presence: Teleoperators & Virtual Environments.

[11]  Holger Regenbrecht,et al.  The Experience of Presence: Factor Analytic Insights , 2001, Presence: Teleoperators & Virtual Environments.

[12]  Niels Henze,et al.  Using Presence Questionnaires in Virtual Reality , 2019, CHI.

[13]  Julian Frommel,et al.  Integrated Questionnaires: Maintaining Presence in Game Environments for Self-Reported Data Acquisition , 2015, CHI PLAY.

[14]  Christian Timmerer,et al.  An evaluation of Heart Rate and ElectroDermal Activity as an objective QoE evaluation method for immersive virtual reality environments , 2016, 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX).

[15]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[16]  Marc Busch,et al.  Being there again - Presence in real and virtual environments and its relation to usability and user experience using a mobile navigation task , 2017, Int. J. Hum. Comput. Stud..

[17]  Regis Kopper,et al.  Visually-Induced Motion Sickness Reduction via Static and Dynamic Rest Frames , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[18]  Daniel J. Liebling,et al.  Unmet Needs and Opportunities for Mobile Translation AI , 2020, CHI.

[19]  Rainer Malaka,et al.  Breaking The Experience: Effects of Questionnaires in VR User Studies , 2020, CHI.

[20]  Nico Haas.  Evaluation of “In-VR-Questionnaires” , 2017 .

[21]  Florian Hruby,et al.  The Sound of Being There: Audiovisual Cartography with Immersive Virtual Environments , 2019, KN - Journal of Cartography and Geographic Information.

[22]  Ohung Kwon,et al.  Flotation Simulation in a Cable-driven Virtual Environment -- A Study with Parasailing , 2018, CHI.

[23]  Elia Gatti,et al.  Pseudo-Haptic Weight: Changing the Perceived Weight of Virtual Objects By Manipulating Control-Display Ratio , 2019, CHI.

[24]  Carolin Wienrich,et al.  Social Presence and Cooperation in Large-Scale Multi-User Virtual Reality - The Relevance of Social Interdependence for Location-Based Environments , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[25]  Marc Erich Latoschik,et al.  The Effect of Haptic Prediction Accuracy on Presence , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[26]  James Liu,et al.  Evaluation of Handsbusy vs Handsfree Virtual Locomotion , 2018, CHI PLAY.

[27]  Daniel Pietschmann,et al.  Continuous Measurement of Breaks in Presence: Psychophysiology and Orienting Responses , 2017 .

[28]  Judy M. Vance,et al.  Industry use of virtual reality in product design and manufacturing: a survey , 2017, Virtual Reality.

[29]  Albert Dipanda,et al.  Tracking Hands in Interaction with Objects: A Review , 2017, 2017 13th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS).

[30]  Eike Langbehn,et al.  I Can See on My Feet While Walking: Sensitivity to Translation Gains with Visible Feet , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[31]  Steven K. Feiner,et al.  Combating VR sickness through subtle dynamic field-of-view modification , 2016, 2016 IEEE Symposium on 3D User Interfaces (3DUI).

[32]  A. Bowling.  Mode of questionnaire administration can have serious effects on data quality. , 2005, Journal of public health.

[33]  Niels Henze,et al.  Online, VR, AR, Lab, and In-Situ: Comparison of Research Methods to Evaluate Smart Artifacts , 2019, CHI.

[34]  Dirk Wenig,et al.  Examining Design Choices of Questionnaires in VR User Studies , 2020, CHI.

[35]  Carl Machover,et al.  Virtual reality , 1994, IEEE Computer Graphics and Applications.

[36]  Majed Al Zayer,et al.  The Effect of Field-of-View Restriction on Sex Bias in VR Sickness and Spatial Navigation Performance , 2019, CHI.

[37]  Michael J. Singer,et al.  The Factor Structure of the Presence Questionnaire , 2005, Presence: Teleoperators & Virtual Environments.

[38]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.

[39]  Mel Slater,et al.  Measuring Presence: A Response to the Witmer and Singer Presence Questionnaire , 1999, Presence.