A Review on the Computational Methods for Emotional State Estimation from the Human EEG

A growing number of affective computing researches recently developed a computer system that can recognize an emotional state of the human user to establish affective human-computer interactions. Various measures have been used to estimate emotional states, including self-report, startle response, behavioral response, autonomic measurement, and neurophysiologic measurement. Among them, inferring emotional states from electroencephalography (EEG) has received considerable attention as EEG could directly reflect emotional states with relatively low costs and simplicity. Yet, EEG-based emotional state estimation requires well-designed computational methods to extract information from complex and noisy multichannel EEG data. In this paper, we review the computational methods that have been developed to deduct EEG indices of emotion, to extract emotion-related features, or to classify EEG signals into one of many emotional states. We also propose using sequential Bayesian inference to estimate the continuous emotional state in real time. We present current challenges for building an EEG-based emotion recognition system and suggest some future directions.  

[1]  M. Balconi,et al.  Brain oscillations and BIS/BAS (behavioral inhibition/activation system) effects on processing masked emotional cues. ERS/ERD and coherence measures of alpha band. , 2009, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[2]  Jim Lagopoulos,et al.  Neural correlates of emotional face processing in bipolar disorder: an event-related potential study. , 2011, Journal of affective disorders.

[3]  Katherine Vytal,et al.  Neuroimaging Support for Discrete Neural Correlates of Basic Emotions: A Voxel-based Meta-analysis , 2010, Journal of Cognitive Neuroscience.

[4]  Jaak Panksepp,et al.  Neuro-Psychoanalysis May Enliven the Mindbrain Sciences , 2007, Cortex.

[5]  F. B. Reguig,et al.  Emotion recognition from physiological signals , 2011, Journal of medical engineering & technology.

[6]  R. Davidson Anterior cerebral asymmetry and the nature of emotion , 1992, Brain and Cognition.

[7]  R Roschmann,et al.  Topographic brain mapping of emotion-related hemisphere asymmetries. , 1992, The International journal of neuroscience.

[8]  Bao-Liang Lu,et al.  Emotion classification based on gamma-band EEG , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[9]  Richard D. Jones,et al.  EEG-Based Lapse Detection With High Temporal Resolution , 2007, IEEE Transactions on Biomedical Engineering.

[10]  Sam J. Maglio,et al.  Emotional category data on images from the international affective picture system , 2005, Behavior research methods.

[11]  Jonas K. Olofsson,et al.  Affective picture processing: An integrative review of ERP findings , 2008, Biological Psychology.

[12]  Joachim M. Buhmann,et al.  Generative Embedding for Model-Based Classification of fMRI Data , 2011, PLoS Comput. Biol..

[13]  J. Panksepp,et al.  Human brain EEG indices of emotions: Delineating responses to affective vocalizations by measuring frontal theta event-related synchronization , 2011, Neuroscience & Biobehavioral Reviews.

[14]  Angelo Gemignani,et al.  The dynamics of EEG gamma responses to unpleasant visual stimuli: From local activity to functional connectivity , 2012, NeuroImage.

[15]  Lorena R. R. Gianotti,et al.  First Valence, Then Arousal: The Temporal Dynamics of Brain Electric Activity Evoked by Emotional Stimuli , 2008, Brain Topography.

[16]  P. Mahalanobis On the generalized distance in statistics , 1936 .

[17]  Lauri Parkkonen,et al.  Dynamical MEG source modeling with multi‐target Bayesian filtering , 2009, Human brain mapping.

[18]  Julien Penders,et al.  Towards wireless emotional valence detection from EEG , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[19]  M. Grigutsch,et al.  Music and emotion: electrophysiological correlates of the processing of pleasant and unpleasant music. , 2007, Psychophysiology.

[20]  José Carlos Príncipe,et al.  A New Spatiotemporal Filtering Method for Single-Trial Estimation of Correlated ERP Subcomponents , 2011, IEEE Transactions on Biomedical Engineering.

[21]  R. Adolphs Neural systems for recognizing emotion , 2002, Current Opinion in Neurobiology.

[22]  J. Fell,et al.  The role of phase synchronization in memory processes , 2011, Nature Reviews Neuroscience.

[23]  C. Joyce,et al.  Automatic removal of eye movement and blink artifacts from EEG data using blind component separation. , 2004, Psychophysiology.

[24]  L. Aftanas,et al.  Analysis of Evoked EEG Synchronization and Desynchronization in Conditions of Emotional Activation in Humans: Temporal and Topographic Characteristics , 2004, Neuroscience and Behavioral Physiology.

[25]  Youxi Wu,et al.  Classification of Mental Task From EEG Signals Using Immune Feature Weighted Support Vector Machines , 2011, IEEE Transactions on Magnetics.

[26]  S. A. Hosseini,et al.  Higher Order Spectra Analysis of EEG Signals in Emotional Stress States , 2010, 2010 Second International Conference on Information Technology and Computer Science.

[27]  Jeffrey B. Henriques,et al.  Resting frontal brain asymmetry predicts affective responses to films. , 1990, Journal of personality and social psychology.

[28]  Thomas Elbert,et al.  Emotion Processing in the Visual Brain: A MEG Analysis , 2008, Brain Topography.

[29]  H. B. Mitchell Sequential Bayesian Inference , 2012 .

[30]  L. Schmidt,et al.  Cross-regional cortical synchronization during affective image viewing , 2010, Brain Research.

[31]  Werner Wittling,et al.  Topographic brain mapping of emotion-related hemisphere asymmetries. , 1992 .

[32]  E. Schaefer,et al.  A circumplex model for maternal behavior. , 1959, Journal of abnormal and social psychology.

[33]  Vera Ferrari,et al.  Repetition and Event-related Potentials: Distinguishing Early and Late Processes in Affective Picture Perception , 2007, Journal of Cognitive Neuroscience.

[34]  Matthias M. Müller,et al.  Effects of emotional arousal in the cerebral hemispheres: a study of oscillatory brain activity and event-related potentials , 2001, Clinical Neurophysiology.

[35]  Tzyy-Ping Jung,et al.  Dry-Contact and Noncontact Biopotential Electrodes: Methodological Review , 2010, IEEE Reviews in Biomedical Engineering.

[36]  Bao-Liang Lu,et al.  EEG-based emotion recognition during watching movies , 2011, 2011 5th International IEEE/EMBS Conference on Neural Engineering.

[37]  A. Mouraux,et al.  Taking into account latency, amplitude, and morphology: improved estimation of single-trial ERPs by wavelet filtering and multiple linear regression. , 2011, Journal of neurophysiology.

[38]  Terence D Sanger,et al.  Bayesian filtering of myoelectric signals. , 2007, Journal of neurophysiology.

[39]  Miguel Angel Mañanas,et al.  A comparative study of automatic techniques for ocular artifact reduction in spontaneous EEG signals based on clinical target variables: A simulation case , 2008, Comput. Biol. Medicine.

[40]  Johan Wagemans,et al.  Single trial ERP reading based on parallel factor analysis. , 2013, Psychophysiology.

[41]  Michael D. Robinson,et al.  Measures of emotion: A review , 2009, Cognition & emotion.

[42]  F. Babiloni,et al.  Mahalanobis distance-based classifiers are able to recognize EEG patterns by using few EEG electrodes , 2001, 2001 Conference Proceedings of the 23rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[43]  P D Bamidis,et al.  Affective Medicine , 2010, Methods of Information in Medicine.

[44]  T. Baumgartner,et al.  From emotion perception to emotion experience: emotions evoked by pictures and classical music. , 2006, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[45]  Isabel Fraga,et al.  Affective ratings of sound stimuli , 2008, Behavior research methods.

[46]  Si Wu,et al.  Sequential Bayesian Decoding with a Population of Neurons , 2003, Neural Computation.

[47]  A. Mehrabian Basic dimensions for a general psychological theory : implications for personality, social, environmental, and developmental studies , 1980 .

[48]  Ian H. Gotlib,et al.  Frontal EEG Alpha Asymmetry, Depression, and Cognitive Functioning , 1998 .

[49]  Alexa B. Roggeveen,et al.  Large-scale gamma-band phase synchronization and selective attention. , 2008, Cerebral cortex.

[50]  O. Pollatos,et al.  On the relationship between interoceptive awareness, emotional experience, and brain processes. , 2005, Brain research. Cognitive brain research.

[51]  Susan M. Bowyer,et al.  Clinical applications of magnetoencephalography in epilepsy , 2010, Annals of Indian Academy of Neurology.

[52]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[53]  Tom Heskes,et al.  Neural Decoding with Hierarchical Generative Models , 2010, Neural Computation.

[54]  S. Rauch,et al.  Neurobiology of emotion perception II: implications for major psychiatric disorders , 2003, Biological Psychiatry.

[55]  Mark D. Holmes,et al.  Intracranial EEG power spectra and phase synchrony during consciousness and unconsciousness , 2009, Consciousness and Cognition.

[56]  R. Ilmoniemi,et al.  Magnetoencephalography-theory, instrumentation, and applications to noninvasive studies of the working human brain , 1993 .

[57]  Christoph Bandt,et al.  A simple classification tool for single-trial analysis of ERP components. , 2009, Psychophysiology.

[58]  Maryam Ahmadi,et al.  Automatic denoising of single-trial evoked potentials , 2013, NeuroImage.

[59]  J. Russell Core affect and the psychological construction of emotion. , 2003, Psychological review.

[60]  C. M. Yee,et al.  Affective valence and information processing. , 1987, Electroencephalography and clinical neurophysiology. Supplement.

[61]  P. Fries Neuronal gamma-band synchronization as a fundamental process in cortical computation. , 2009, Annual review of neuroscience.

[62]  Michael D. Robinson,et al.  Belief and feeling: evidence for an accessibility model of emotional self-report. , 2002, Psychological bulletin.

[63]  R. B. Silberstein,et al.  Steady-State Visually Evoked Potential Topography during Processing of Emotional Valence in Healthy Subjects , 2002, NeuroImage.

[64]  Vacius Jusas,et al.  Using Higher Order Nonlinear Operators for SVM Classification of EEG Data , 2012 .

[65]  M. Bradley,et al.  Brain potentials in affective picture processing: covariation with autonomic arousal and affective report , 2000, Biological Psychology.

[66]  Matthias M. Müller,et al.  Processing of affective pictures modulates right-hemispheric gamma band EEG activity , 1999, Clinical Neurophysiology.

[67]  Erol Basar,et al.  Emotional face expressions are differentiated with brain oscillations. , 2007, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[68]  José Carlos Príncipe,et al.  Sequential Monte Carlo Point-Process Estimation of Kinematics from Neural Spiking Activity for Brain-Machine Interfaces , 2009, Neural Computation.

[69]  I. Christie,et al.  Autonomic specificity of discrete emotion and dimensions of affective space: a multivariate approach. , 2004, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[70]  Stefan Haufe,et al.  Single-trial analysis and classification of ERP components — A tutorial , 2011, NeuroImage.

[71]  Leontios J. Hadjileontiadis,et al.  Emotion Recognition From EEG Using Higher Order Crossings , 2010, IEEE Transactions on Information Technology in Biomedicine.

[72]  Yong Yang,et al.  Automatic Removal of Eye-Movement and Blink Artifacts from EEG Signals , 2010, Brain Topography.

[73]  Stephen J. Roberts,et al.  Adaptive classification for Brain Computer Interface systems using Sequential Monte Carlo sampling , 2009, Neural Networks.

[74]  Bao-Liang Lu,et al.  EEG-Based Emotion Recognition Using Frequency Domain Features and Support Vector Machines , 2011, ICONIP.

[75]  S. Rauch,et al.  Neurobiology of emotion perception I: the neural basis of normal emotion perception , 2003, Biological Psychiatry.

[76]  Edward Cutrell,et al.  BCI for passive input in HCI , 2007 .

[77]  Stephan Hamann,et al.  Mapping discrete and dimensional emotions onto the brain: controversies and consensus , 2012, Trends in Cognitive Sciences.

[78]  Michela Balconi,et al.  Event-Related Oscillations (ERO) and Event-Related Potentials (ERP) in Emotional Face Recognition , 2008, The International journal of neuroscience.

[79]  Jyh-Yeong Chang,et al.  Novel Dry Polymer Foam Electrodes for Long-Term EEG Measurement , 2011, IEEE Transactions on Biomedical Engineering.

[80]  L. F. Barrett Discrete Emotions or Dimensions? The Role of Valence Focus and Arousal Focus , 1998 .

[81]  Leontios J. Hadjileontiadis,et al.  Toward an EEG-Based Recognition of Music Liking Using Time-Frequency Analysis , 2012, IEEE Transactions on Biomedical Engineering.

[82]  T. Jung,et al.  Dry and Noncontact EEG Sensors for Mobile Brain–Computer Interfaces , 2012, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[83]  Chin-Teng Lin,et al.  Design, Fabrication and Experimental Validation of a Novel Dry-Contact Sensor for Measuring Electroencephalography Signals without Skin Preparation , 2011, Sensors.

[84]  Charalampos Bratsas,et al.  Toward Emotion Aware Computing: An Integrated Approach Using Multichannel Neurophysiological Recordings and Affective Visual Stimuli , 2010, IEEE Transactions on Information Technology in Biomedicine.

[85]  Sylvia D. Kreibig,et al.  An affective computing approach to physiological emotion specificity: toward subject-independent and stimulus-independent classification of film-induced emotions. , 2011, Psychophysiology.

[86]  Leontios J. Hadjileontiadis,et al.  A Novel Emotion Elicitation Index Using Frontal Brain Asymmetry for Enhanced EEG-Based Emotion Recognition , 2011, IEEE Transactions on Information Technology in Biomedicine.

[87]  Chrysa D. Lithari,et al.  Are Females More Responsive to Emotional Stimuli? A Neurophysiological Study Across Arousal and Valence Dimensions , 2009, Brain Topography.

[88]  L. Aftanas,et al.  Affective picture processing: event-related synchronization within individually defined human theta band is modulated by valence dimension. , 2001, Neuroscience Letters.

[89]  Elif Derya Übeyli,et al.  Multiclass Support Vector Machines for EEG-Signals Classification , 2007, IEEE Transactions on Information Technology in Biomedicine.

[90]  W. Klimesch,et al.  What does phase information of oscillatory brain activity tell us about cognitive processes? , 2008, Neuroscience & Biobehavioral Reviews.

[91]  Nathalie Fouquet,et al.  Processing emotion from the eyes: A divided visual field and ERP study , 2011, Laterality.

[92]  A. Cichocki,et al.  A novel BCI based on ERP components sensitive to configural processing of human faces , 2012, Journal of neural engineering.

[93]  John Polich,et al.  Affective visual event-related potentials: Arousal, repetition, and time-on-task , 2007, Biological Psychology.

[94]  Don M. Tucker,et al.  A single-trial analytic framework for EEG analysis and its application to target detection and classification , 2008, NeuroImage.

[95]  K. Scherer What are emotions? And how can they be measured? , 2005 .

[96]  Teresa H. Y. Meng,et al.  Model-based neural decoding of reaching movements: a maximum likelihood approach , 2004, IEEE Transactions on Biomedical Engineering.

[97]  Ayman S El-Baz,et al.  INDUCED EEG GAMMA OSCILLATION ALIGNMENT IMPROVES DIFFERENTIATION BETWEEN AUTISM AND ADHD GROUP RESPONSES IN A FACIAL CATEGORIZATION TASK. , 2012, Journal of neurotherapy.

[98]  R. Nagarajan,et al.  Combining Spatial Filtering and Wavelet Transform for Classifying Human Emotions Using EEG Signals , 2011 .

[99]  E. Bernat,et al.  Event-related brain potentials differentiate positive and negative mood adjectives during both supraliminal and subliminal visual processing. , 2001, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[100]  D. Rubin,et al.  A comparison of dimensional models of emotion: Evidence from emotions, prototypical events, autobiographical memories, and words , 2009, Memory.

[101]  R. Barry,et al.  Covariation of EEG Synchronization and Emotional State as Modified by Anxiolytics , 2011, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[102]  Li Yao,et al.  Combining features from ERP components in single-trial EEG for discriminating four-category visual objects , 2012, Journal of neural engineering.

[103]  K. Scherer,et al.  The World of Emotions is not Two-Dimensional , 2007, Psychological science.

[104]  Panagiotis D. Bamidis,et al.  Real time emotion aware applications: A case study employing emotion evocative pictures and neuro-physiological sensing enhanced by Graphic Processor Units , 2012, Comput. Methods Programs Biomed..

[105]  Olga Sourina,et al.  A Fractal-based Algorithm of Emotion Recognition from EEG using Arousal-Valence Model , 2011, BIOSIGNALS.

[106]  M. Balconi,et al.  EEG correlates (event-related desynchronization) of emotional face elaboration: A temporal analysis , 2006, Neuroscience Letters.

[107]  James Bailey,et al.  Improving k-Nearest Neighbour Classification with Distance Functions Based on Receiver Operating Characteristics , 2008, ECML/PKDD.

[108]  Miller Ga,et al.  Affective valence and information processing. , 1987 .

[109]  M. Balconi,et al.  Consciousness and arousal effects on emotional face processing as revealed by brain oscillations. A gamma band analysis. , 2008, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[110]  S M M Martens,et al.  A generative model approach for decoding in the visual event-related potential-based brain–computer interface speller , 2010, Journal of neural engineering.

[111]  W. Singer,et al.  Synchronization of Neural Activity across Cortical Areas Correlates with Conscious Perception , 2007, The Journal of Neuroscience.

[112]  Orlando Francisco Amodeo Bueno,et al.  Comparison of Brazilian and American norms for the International Affective Picture System (IAPS). , 2005, Revista brasileira de psiquiatria.

[113]  E Donchin,et al.  Beyond averaging. II. Single-trial classification of exogenous event-related potentials using stepwise discriminant analysis. , 1980, Electroencephalography and clinical neurophysiology.

[114]  Jae Yun Lee,et al.  Emotion recognition based on the asymmetric left and right activation , 2011 .