Challenge for Affective Brain-Computer Interfaces: Non-stationary Spatio-spectral EEG Oscillations of Emotional Responses

Electroencephalogram (EEG)-based affective brain-computer interfaces (aBCIs) have been attracting ever-growing interest and research resources. Whereas most previous neuroscience studies have focused on single-day/-session recording and sensor-level analysis, less effort has been invested in assessing the fundamental nature of non-stationary EEG oscillations underlying emotional responses across days and individuals. This work thus aimed to use a data-driven blind source separation method, i.e., independent component analysis (ICA), to derive emotion-relevant spatio-spectral EEG source oscillations and assess the extent of non-stationarity. To this end, this work conducted an 8-day music-listening experiment (i.e., roughly interspaced over 2 months) and recorded whole-scalp 30-ch EEG data from 10 subjects. Given the large size of the data (i.e., from 80 sessions), results indicated that EEG non-stationarity was clearly revealed in the numbers and locations of brain sources of interest as well as their spectral modulation to the emotional responses. Less than half of subjects (two to four) showed the same relatively day-stationary (source reproducibility >6 days) spatio-spectral tendency towards one of the binary valence and arousal states. This work substantially advances the previous work by exploiting intra- and inter-individual EEG variability in an ecological multiday scenario. Such EEG non-stationarity may inevitably present a great challenge for the development of an accurate, robust, and generalized emotion-classification model.

[1]  J. Russell A circumplex model of affect. , 1980 .

[2]  Stefan Winkler,et al.  ASCERTAIN: Emotion and Personality Recognition Using Commercial Sensors , 2018, IEEE Transactions on Affective Computing.

[3]  Vincent J. Schmithorst,et al.  Separate cortical networks involved in music perception: preliminary functional MRI evidence for modularity of music processing , 2005, NeuroImage.

[4]  Long Chen,et al.  Incorporation of Multiple-Days Information to Improve the Generalization of EEG-Based Emotion Recognition Over Time , 2018, Front. Hum. Neurosci..

[5]  O. John,et al.  Individual differences in two emotion regulation processes: implications for affect, relationships, and well-being. , 2003, Journal of personality and social psychology.

[6]  Wei Liu,et al.  Intersubject similarity of personality is associated with intersubject similarity of brain connectivity patterns , 2019, NeuroImage.

[7]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[8]  D. Schutter,et al.  Parietal electroencephalogram beta asymmetry and selective attention to angry facial expressions in healthy human subjects , 2001, Neuroscience Letters.

[9]  Alan C. Evans,et al.  Emotional responses to pleasant and unpleasant music correlate with activity in paralimbic brain regions , 1999, Nature Neuroscience.

[10]  Tzyy-Ping Jung,et al.  Revealing spatio-spectral electroencephalographic dynamics of musical mode and tempo perception by independent component analysis , 2014, Journal of NeuroEngineering and Rehabilitation.

[11]  W. David Hairston,et al.  Systems, Subjects, Sessions: To What Extent Do These Factors Influence EEG Data? , 2017, Front. Hum. Neurosci..

[12]  R. Davidson What does the prefrontal cortex “do” in affect: perspectives on frontal EEG asymmetry research , 2004, Biological Psychology.

[13]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[14]  Ricardo Ron-Angevin,et al.  Is Stress State an Important Factor in the BCI-P300 Speller Performance? , 2019, IWANN.

[15]  Tzyy-Ping Jung,et al.  Real-time neuroimaging and cognitive monitoring using wearable dry EEG , 2015, IEEE Transactions on Biomedical Engineering.

[16]  T. Jung,et al.  Translation of EEG Spatial Filters from Resting to Motor Imagery Using Independent Component Analysis , 2012, PloS one.

[17]  Martin Buss,et al.  Feature Extraction and Selection for Emotion Recognition from EEG , 2014, IEEE Transactions on Affective Computing.

[18]  S. Makeig,et al.  Imaging human EEG dynamics using independent component analysis , 2006, Neuroscience & Biobehavioral Reviews.

[19]  Wenming Zheng,et al.  Multichannel EEG-Based Emotion Recognition via Group Sparse Canonical Correlation Analysis , 2017, IEEE Transactions on Cognitive and Developmental Systems.

[20]  Sangtae Ahn,et al.  Exploring Neuro-Physiological Correlates of Drivers' Mental Fatigue Caused by Sleep Deprivation Using Simultaneous EEG, ECG, and fNIRS Data , 2016, Front. Hum. Neurosci..

[21]  Zhenqi Li,et al.  SAE+LSTM: A New Framework for Emotion Recognition From Multi-Channel EEG , 2019, Front. Neurorobot..

[22]  Arnaud Delorme,et al.  Single-Trial Normalization for Event-Related Spectral Decomposition Reduces Sensitivity to Noisy Trials , 2011, Front. Psychology.

[23]  Shinichi Nakajima,et al.  Wasserstein Stationary Subspace Analysis , 2018, IEEE Journal of Selected Topics in Signal Processing.

[24]  C. Allen Dimensions of Personality , 1947, Nature.

[25]  Matthias Rauterberg,et al.  Handbook of Digital Games and Entertainment Technologies , 2016 .

[26]  Scott Makeig,et al.  Information-based modeling of event-related brain dynamics. , 2006, Progress in brain research.

[27]  T. Jung,et al.  Electroencephalographic dynamics of musical emotion perception revealed by independent spectral components , 2010, Neuroreport.

[28]  Yuan-Pin Lin,et al.  Co-modulatory spectral changes in independent brain processes are correlated with task performance , 2012, NeuroImage.

[29]  K. Luan Phan,et al.  Functional Neuroanatomy of Emotion: A Meta-Analysis of Emotion Activation Studies in PET and fMRI , 2002, NeuroImage.

[30]  Ping-Keng Jao,et al.  Improving Cross-Day EEG-Based Emotion Classification Using Robust Principal Component Analysis , 2017, Front. Comput. Neurosci..

[31]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  G. Rees,et al.  The structural basis of inter-individual differences in human behaviour and cognition , 2011, Nature Reviews Neuroscience.

[33]  W. Heller Neuropsychological mechanisms of individual differences in emotion, personality, and arousal. , 1993 .

[34]  M. Grigutsch,et al.  Music and emotion: electrophysiological correlates of the processing of pleasant and unpleasant music. , 2007, Psychophysiology.

[35]  C. Liégeois-Chauvel,et al.  Brain regions involved in the recognition of happiness and sadness in music , 2005, Neuroreport.

[36]  Yuan-Pin Lin,et al.  Exploring Day-to-Day Variability in the Relations Between Emotion and EEG Signals , 2015, HCI.

[37]  Zhen Cui,et al.  From Regional to Global Brain: A Novel Hierarchical Spatial-Temporal Neural Network Model for EEG Emotion Recognition , 2019, IEEE Transactions on Affective Computing.

[38]  Dan Liu,et al.  A Fast, Efficient Domain Adaptation Technique for Cross-Domain Electroencephalography(EEG)-Based Emotion Recognition , 2017, Sensors.

[39]  Klaus Gramann,et al.  Walking through Architectural Spaces: The Impact of Interior Forms on Human Brain Dynamics , 2017, Front. Hum. Neurosci..

[40]  Tzyy-Ping Jung,et al.  Spatial and temporal EEG dynamics of motion sickness , 2010, NeuroImage.

[41]  Yuan-Pin Lin,et al.  Independent Component Ensemble of EEG for Brain–Computer Interface , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[42]  A. Craig,et al.  Driver fatigue: electroencephalography and psychological assessment. , 2002, Psychophysiology.

[43]  R. Oostenveld,et al.  Validating the boundary element method for forward and inverse EEG computations in the presence of a hole in the skull , 2002, Human brain mapping.

[44]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[45]  Chin-Teng Lin,et al.  Theta and Alpha Oscillations in Attentional Interaction during Distracted Driving , 2018, Front. Behav. Neurosci..

[46]  R. Oostenveld,et al.  Independent EEG Sources Are Dipolar , 2012, PloS one.

[47]  Christa Neuper,et al.  Distinct β Band Oscillatory Networks Subserving Motor and Cognitive Control during Gait Adaptation , 2016, The Journal of Neuroscience.

[48]  Bin Hu,et al.  Exploring EEG Features in Cross-Subject Emotion Recognition , 2018, Front. Neurosci..

[49]  James C. Christensen,et al.  The effects of day-to-day variability of physiological data on operator functional state classification , 2012, NeuroImage.

[50]  L. Trainor,et al.  Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions , 2001 .

[51]  T. Jung,et al.  Improving EEG-Based Emotion Classification Using Conditional Transfer Learning , 2017, Front. Hum. Neurosci..

[52]  Alexander Toet,et al.  Neuroticism, Extraversion, Conscientiousness and Stress: Physiological Correlates , 2015, IEEE Transactions on Affective Computing.

[53]  Joydeep Bhattacharya,et al.  Phase synchrony analysis of EEG during music perception reveals changes in functional connectivity due to musical expertise , 2005, Signal Process..

[54]  Tapani Ristaniemi,et al.  Linking Brain Responses to Naturalistic Music Through Analysis of Ongoing EEG and Stimulus Features , 2013, IEEE Transactions on Multimedia.

[55]  T. Sejnowski,et al.  Removing electroencephalographic artifacts by blind source separation. , 2000, Psychophysiology.

[56]  Scott Makeig,et al.  High-frequency Broadband Modulations of Electroencephalographic Spectra , 2009, Front. Hum. Neurosci..

[57]  Ian Daly,et al.  Neural correlates of emotional responses to music: An EEG study , 2014, Neuroscience Letters.

[58]  Silvestro Micera,et al.  Unidirectional brain to muscle connectivity reveals motor cortex control of leg muscles during stereotyped walking , 2017, NeuroImage.

[59]  L. Jäncke,et al.  Independent component processes underlying emotions during natural music listening. , 2016, Social cognitive and affective neuroscience.

[60]  Daniel P. Ferris,et al.  Independent Component Analysis and Source Localization on Mobile EEG Data Can Identify Increased Levels of Acute Stress , 2017, Front. Hum. Neurosci..

[61]  Jon Touryan,et al.  Towards Serious Games for Improved BCI , 2016 .

[62]  Patrik Vuilleumier,et al.  Temporal dynamics of musical emotions examined through intersubject synchrony of brain activity. , 2015, Social cognitive and affective neuroscience.