Multimodal System for Emotion Recognition Using EEG and Customer Review

Emotion is a basic expression of a human being using which he/she can communicate with the external world. Emotion can be recognized using various media such as movie, image, facial expression, and audio. Emotion recognition can be performed by the brain signal (e.g., Electroencephalogram). A multimodal system of emotion recognition using Electroencephalogram (EEG) and sentiment analysis of customer has been proposed. Four types of emotion, namely: Happy, sad, relaxed and anger have been recognized. The proposed multimodal framework accepts the combination of temporal (EEG signal) and spatial (customer reviews/comments) information as inputs and generates the emotion of user during watching the product on computer screen. The proposed system learns temporal and spatial discriminative features using EEG encoder and text encoder. Both of the encoders transform the features of EEG and text into common feature space. The methodology is being tested on a dataset of 30 users, consisting of EEG and customers’ review data. An accuracy of 98.27% has been recorded.

[1]  Debi Prosad Dogra,et al.  Prediction of advertisement preference by fusing EEG response and sentiment analysis , 2017, Neural Networks.

[2]  Marti A. Hearst.  Direction-based text interpretation as an information access refinement , 1992 .

[3]  L. H. Viet,et al.  Emotion Detection in the Loop from Brain Signals and Facial Images , 2006 .

[4]  Xavier Giró-i-Nieto,et al.  From pixels to sentiment: Fine-tuning CNNs for visual sentiment prediction , 2016, Image Vis. Comput..

[5]  S. Rose,et al.  Salience and choice: Neural correlates of shopping decisions , 2004 .

[6]  K. Scherer,et al.  Emotion recognition from expressions in face, voice, and body: the Multimodal Emotion Recognition Test (MERT). , 2009, Emotion.

[7]  Scott A. Huettel,et al.  Neuromarketing: Ethical Implications of its Use and Potential Misuse , 2016, Journal of Business Ethics.

[8]  Min-Yuh Day,et al.  Deep learning for financial sentiment analysis on finance news providers , 2016, 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).

[9]  Minara P. Anto,et al.  Product rating using sentiment analysis , 2016, 2016 International Conference on Electrical, Electronics, and Optimization Techniques (ICEEOT).

[10]  Bridget K. Behe,et al.  Seeking attention: an eye tracking study of in-store merchandise displays , 2015 .

[11]  Ale Smidts,et al.  Brain Responses to Movie Trailers Predict Individual Preferences for Movies and Their Population-Wide Commercial Success , 2015 .

[12]  Laura Astolfi,et al.  On the Use of EEG or MEG Brain Imaging Tools in Neuromarketing Research , 2011, Comput. Intell. Neurosci..

[13]  Olga Sourina,et al.  EEG-Based Emotion-Adaptive Advertising , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[14]  Loren Terveen,et al.  PHOAKS: a system for sharing recommendations , 1997, CACM.

[15]  Gamini Dissanayake,et al.  Choice modeling and the brain: A study on the Electroencephalogram (EEG) of preferences , 2012, Expert Syst. Appl..

[16]  P. Kenning,et al.  Why Companies Should Make Their Customers Happy: The Neural Correlates of Customer Loyalty , 2007 .

[17]  Sanjay Chakraborty,et al.  EEG based emotion classification using “Correlation Based Subset Selection” , 2018, BICA 2018.

[18]  Javad Frounchi,et al.  Wavelet-based emotion recognition system using EEG signal , 2017, Neural Computing and Applications.

[19]  B. Weber,et al.  Individual Differences in Marketing Placebo Effects: Evidence from Brain Imaging and Behavioral Experiments , 2015 .

[20]  Satoshi Morinaga,et al.  Mining product reputations on the Web , 2002, KDD.

[21]  Bao-Liang Lu,et al.  Multimodal emotion recognition using EEG and eye tracking data , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[22]  Christophe Morin,et al.  Neuromarketing: The New Science of Consumer Behavior , 2011 .

[23]  Swaminathan Natarajan,et al.  Sentiment analysis of Facebook data using Hadoop based open source technologies , 2015, 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA).