VR Sickness Prediction for Navigation in Immersive Virtual Environments using a Deep Long Short Term Memory Model

This paper proposes a new objective metric of visually induced motion sickness (VIMS) in the context of navigation in virtual environments (VEs). Similar to motion sickness in physical environments, VIMS can induce many physiological symptoms such as general discomfort, nausea, disorientation, vomiting, dizziness and fatigue. To improve user satisfaction with VR applications, it is of great significance to develop objective metrics for VIMS that can analyze and estimate the level of VR sickness when a user is exposed to VEs. One of the well-known objective metrics is the postural instability. In this paper, we trained a LSTM model for each participant using a normal-state postural signal captured before the exposure, and if the postural sway signal from post-exposure was sufficiently different from the pre-exposure signal, the model would fail at encoding and decoding the signal properly; the jump in the reconstruction error was called loss and was proposed as the proposed objective measure of simulator sickness. The effectiveness of the proposed metric was analyzed and compared with subjective assessment methods based on the simulator sickness questionnaire (SSQ) in a VR environment, achieving a Pearson correlation coefficient of. 89. Finally, we showed that the proposed method had the potential to be deployed within a closed-loop system and get real-time performance to predict VR sickness, opening new insights to develop user-centered and customized VR applications based on physiological feedback.

[1]  Dong Yu,et al.  Deep Learning: Methods and Applications , 2014, Found. Trends Signal Process..

[2]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[3]  J. L. Higuera-Trujillo,et al.  Psychological and physiological human responses to simulated and real environments: A comparison between Photographs, 360° Panoramas, and Virtual Reality. , 2017, Applied ergonomics.

[4]  Martin Hachet,et al.  Advances in Interaction with 3D Environments , 2015, Comput. Graph. Forum.

[5]  Mike Lambert,et al.  PaperDude: a virtual reality cycling exergame , 2014, CHI Extended Abstracts.

[6]  Takehisa Yairi,et al.  A review on the application of deep learning in system health management , 2018, Mechanical Systems and Signal Processing.

[7]  M. Ali Mirzaei,et al.  Features of the Postural Sway Signal as Indicators to Estimate and Predict Visually Induced Motion Sickness in Virtual Reality , 2017, Int. J. Hum. Comput. Interact..

[8]  Mark S. Dennison,et al.  Use of physiological signals to predict cybersickness , 2016, Displays.

[9]  J. Golding Predicting individual differences in motion sickness susceptibility by questionnaire , 2006 .

[10]  Turgay Aslandere,et al.  Virtual hand-button interaction in a generic virtual reality flight simulator , 2015, 2015 IEEE Aerospace Conference.

[11]  Julien Nelson,et al.  6DoF navigation in virtual worlds: comparison of joystick-based and head-controlled paradigms , 2013, VRST '13.

[12]  D. Harm Physiology of motion sickness symptoms , 1990 .

[13]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[14]  Simon Davis,et al.  A Systematic Review of Cybersickness , 2014, IE.

[15]  Frédéric Mérienne,et al.  Using Cybersickness Indicators to Adapt Navigation in Virtual Reality: A Pre-Study , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[16]  Holger Regenbrecht,et al.  Immersion factors affecting perception and behaviour in a virtual reality power wheelchair simulator. , 2017, Applied ergonomics.

[17]  Michael E. McCauley,et al.  Simulator Sickness: A Reaction to a Transformed Perceptual World. 1. Scope of the Problem , 1983 .

[18]  Chin-Teng Lin,et al.  EEG-Based Learning System for Online Motion Sickness Level Estimation in a Dynamic Vehicle Environment , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[19]  Shelley B Brundage,et al.  Utility of virtual reality environments to examine physiological reactivity and subjective distress in adults who stutter. , 2016, Journal of fluency disorders.

[20]  U. Rajendra Acharya,et al.  Deep learning for healthcare applications based on physiological signals: A review , 2018, Comput. Methods Programs Biomed..

[21]  J T Reason,et al.  Motion Sickness Adaptation: A Neural Mismatch Model 1 , 1978, Journal of the Royal Society of Medicine.

[22]  Jonghyun Choi,et al.  Learning Temporal Regularity in Video Sequences , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Jaehyun Park,et al.  Virtual reality sickness questionnaire (VRSQ): Motion sickness measurement index in a virtual reality environment. , 2018, Applied ergonomics.

[24]  Julie M. Drexler,et al.  Research in visually induced motion sickness. , 2010, Applied ergonomics.

[25]  Shirley Dex,et al.  JR 旅客販売総合システム(マルス)における運用及び管理について , 1991 .

[26]  Kazuhiko Hamamoto,et al.  Investigation of visually induced motion sickness in dynamic 3D contents based on subjective judgment, heart rate variability, and depth gaze behavior , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[27]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[28]  Robert S. Kennedy,et al.  Simulator Sickness Questionnaire: An enhanced method for quantifying simulator sickness. , 1993 .

[29]  Gordon Wetzstein,et al.  Towards a Machine-Learning Approach for Sickness Prediction in 360° Stereoscopic Videos , 2018, IEEE Transactions on Visualization and Computer Graphics.

[30]  Paulina M Baran,et al.  The effects of simulated fog and motion on simulator sickness in a driving simulator and the duration of after-effects. , 2014, Applied ergonomics.

[31]  Hiroki Takada,et al.  [Evaluation of Motion Sickness Induced by 3D Video Clips]. , 2016, Nihon eiseigaku zasshi. Japanese journal of hygiene.

[32]  Steven M. LaValle,et al.  Head tracking for the Oculus Rift , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[33]  T. Stoffregen,et al.  An ecological Theory of Motion Sickness and Postural Instability , 1991 .

[34]  Bernard D. Adelstein,et al.  Demand Characteristics in Assessing Motion Sickness in a Virtual Environment: Or Does Taking a Motion Sickness Questionnaire Make You Sick? , 2007 .