Cybersickness Analysis with EEG Using Deep Learning Algorithms

Cybersickness is a symptom of dizziness that occurs while experiencing Virtual Reality (VR) technology and it is presumed to occur mainly by crosstalk between the sensory and cognitive systems. However, since the sensory and cognitive systems cannot be measured objectively, it is difficult to measure cybersickness. Therefore, methodologies for measuring cybersickness have been studied in various ways. Traditional studies have collected answers to questionnaires or analyzed EEG data using machine learning algorithms. However, the system relying on the questionnaires lacks objectivity, and it is difficult to obtain highly accurate measurements with the machine learning algorithms in previous studies. In this work, we apply and compare Deep Neural Network (DNN) and Convolutional Neural Network (CNN) deep learning algorithms for objective cy-bersickness measurement from EEG data. We also propose a data preprocessing for learning and signal quality weights allowing us to achieve high performance while learning EEG data with the deep learning algorithms. Besides, we analyze video characteristics where cybersickness occurs by examining the 360 video stream segments causing cybersickness in the experiments. Finally, we draw common patterns that cause cybersickness.

[1]  Simon Davis,et al.  Comparing the onset of cybersickness using the Oculus Rift and two virtual roller coasters , 2015 .

[2]  Heiko Hecht,et al.  Pleasant music as a countermeasure against visually induced motion sickness. , 2014, Applied ergonomics.

[3]  Terrence J. Sejnowski,et al.  Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources , 1999, Neural Computation.

[4]  Bao-Liang Lu,et al.  EEG-based emotion recognition during watching movies , 2011, 2011 5th International IEEE/EMBS Conference on Neural Engineering.

[5]  Ali Mansour,et al.  Blind Separation of Sources , 1999 .

[6]  Tzyy-Ping Jung,et al.  EEG-based evaluation system for motion sickness estimation , 2011, 2011 5th International IEEE/EMBS Conference on Neural Engineering.

[7]  Chin-Teng Lin,et al.  An EEG-based classification system of Passenger's motion sickness level by using feature extraction/selection technologies , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[8]  S Makeig,et al.  Blind separation of auditory event-related brain responses into independent components. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[9]  Shiaw-Tsyr Uang,et al.  A study of sickness induced within a 3D virtual store and combated with fuzzy control in the elderly , 2012, 2012 9th International Conference on Fuzzy Systems and Knowledge Discovery.

[10]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[11]  Leontios J. Hadjileontiadis,et al.  Emotion Recognition From EEG Using Higher Order Crossings , 2010, IEEE Transactions on Information Technology in Biomedicine.

[12]  Begoña Garcia-Zapirain,et al.  EEG artifact removal—state-of-the-art and guidelines , 2015, Journal of neural engineering.

[13]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[14]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[15]  Jelte E. Bos,et al.  Internal and external fields of view affect cybersickness , 2011, Displays.

[16]  Karen Blackmore,et al.  Correlating reaction time and nausea measures with traditional measures of cybersickness , 2017, Displays.

[17]  Tonio Ball,et al.  Deep learning with convolutional neural networks for decoding and visualization of EEG pathology , 2017, 2017 IEEE Signal Processing in Medicine and Biology Symposium (SPMB).

[18]  Sunu Wibirama,et al.  Investigation of visual attention in day-night driving simulator during cybersickness occurrence , 2016, 2016 8th International Conference on Information Technology and Electrical Engineering (ICITEE).

[19]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[20]  Bor-Chen Kuo,et al.  Feature Extractions for Small Sample Size Classification Problem , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[21]  Chin-Teng Lin,et al.  EEG-Based Learning System for Online Motion Sickness Level Estimation in a Dynamic Vehicle Environment , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[22]  T. Sejnowski,et al.  Estimating alertness from the EEG power spectrum , 1997, IEEE Transactions on Biomedical Engineering.

[23]  Mark H. Draper,et al.  Effects of Image Scale and System Time Delay on Simulator Sickness within Head-Coupled Virtual Environments , 2001, Hum. Factors.

[24]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[25]  Teemu Leinonen,et al.  Reflection in Learning through a Self-monitoring Device: Design Research on EEG Self-Monitoring during a Study Session , 2017 .

[26]  R. So,et al.  Cybersickness in the presence of scene rotational movements along different axes. , 2001, Applied ergonomics.

[27]  Abdul Wahab,et al.  EEG Emotion Recognition Based on the Dimensional Models of Emotions , 2013 .

[28]  Ronald R. Mourant,et al.  Comparison of Simulator Sickness Using Static and Dynamic Walking Simulators , 2001 .

[29]  Girma Neshir Alemneh EEG based Emotion Recognition of Image Stimuli , 2019 .

[30]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[31]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[32]  Steven K. Feiner,et al.  Combating VR sickness through subtle dynamic field-of-view modification , 2016, 2016 IEEE Symposium on 3D User Interfaces (3DUI).

[33]  Olga Sourina,et al.  Real-Time EEG-Based Human Emotion Recognition and Visualization , 2010, 2010 International Conference on Cyberworlds.

[34]  E. Kochs,et al.  Awareness and the EEG power spectrum: analysis of frequencies. , 2004, British journal of anaesthesia.

[35]  Lisa Rebenitsch,et al.  Managing cybersickness in virtual reality , 2015, XRDS.

[36]  Bao-Liang Lu,et al.  Multimodal emotion recognition using EEG and eye tracking data , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[37]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[38]  Y.C. Chen,et al.  Monitoring Driver's Alertness Based on the Driving Performance Estimation and the EEG Power Spectrum Analysis , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[39]  Joseph J. LaViola,et al.  A discussion of cybersickness in virtual environments , 2000, SGCH.

[40]  Andy T. K. Ho,et al.  A Metric to Quantify Virtual Scene Movement for the Study of Cybersickness: Definition, Implementation, and Verification , 2001, Presence: Teleoperators & Virtual Environments.

[41]  Tonio Ball,et al.  Hierarchical internal representation of spectral features in deep convolutional networks trained for EEG decoding , 2017, 2018 6th International Conference on Brain-Computer Interface (BCI).

[42]  Kwanguk Kim,et al.  Effects of virtual environment platforms on emotional responses , 2014, Comput. Methods Programs Biomed..

[43]  P. Comon Independent Component Analysis , 1992 .

[44]  B. Hjorth EEG analysis based on time domain properties. , 1970, Electroencephalography and clinical neurophysiology.

[45]  Benjamin Weyers,et al.  Examining Rotation Gain in CAVE-like Virtual Environments , 2016, IEEE Transactions on Visualization and Computer Graphics.