Decoding emotional changes of android-gamers using a fused Type-2 fuzzy deep neural network

Abstract With the fastest growing popularity of gaming applications on android phone, analyzing emotion changes of steadfast android-gamers have become a study of utmost interest among most of the psychologists. Recently, some android games are producing negative impacts to the gamers; even in the worst cases the effect is becoming life-threatening too. Most of the existing research works are based on psychological view-point of exploring the impact (positive/negative) of playing android games for the child and adult age-group. However, the online recognition of emotional state changes of the android-gamers while playing video games may be relatively unexplored. To fill this void, the present study proposes a novel method of identifying the emotional state changes of android-gamers by decoding their brain signals and facial images simultaneously during playing video games. Besides above, the second novelty of the paper lies in designing a multimodal fusion method between brain signals and facial images for the said application. To address this challenge, the paper proposes a fused type-2 fuzzy deep neural network (FT2FDNN) which integrates the brain signal processing approach by a general type-2 fuzzy reasoning algorithm with the flavor of the image/video processing approach using a deep convolutional neural network. FT2FDNN uses multiple modalities to extract the similar information (here, emotional changes) simultaneously from the type-2 fuzzy and deep neural representations. The proposed fused type-2 fuzzy deep learning paradigm demonstrates promising results in classifying the emotional changes of gamers with high classification accuracy. Thus the proposed work explores a new era for future researchers.

[1]  Lorenzo Torresani,et al.  Learning Spatiotemporal Features with 3D Convolutional Networks , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  Adam G. Polak,et al.  Analysis of Features Extracted from EEG Epochs by Discrete Wavelet Decomposition and Hilbert Transform for Sleep Apnea Detection , 2018, 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[3]  Discrete Hilbert Transform . Numeric Algorithms , 2009 .

[4]  Jerry M. Mendel,et al.  General Type-2 Fuzzy Logic Systems Made Simple: A Tutorial , 2014, IEEE Transactions on Fuzzy Systems.

[5]  Pratyusha Rakshit,et al.  Working memory modeling using inverse fuzzy relational approach , 2019, Appl. Soft Comput..

[6]  T. Järvilehto,et al.  Frontal midline theta related to learning in a simulated driving task , 1995, Biological Psychology.

[7]  J. Sargent,et al.  Metaanalysis of the relationship between violent video game play and physical aggression over time , 2018, Proceedings of the National Academy of Sciences.

[8]  Sriparna Saha,et al.  Analyzing Adverse Gaming Effects on Emotions Using Neural Networks Based Hybrid Architecture , 2019, 2019 IEEE 16th India Council International Conference (INDICON).

[9]  C. Anderson,et al.  The effect of video game violence on physiological desensitization to real-life violence. , 2007 .

[10]  M. Murugappan,et al.  Time-Frequency Analysis of EEG Signals for Human Emotion Detection , 2008 .

[11]  Su Ruan,et al.  A robust agorithm for eye detection on gray intensity face without spectacles , 2005 .

[12]  S. Kühn,et al.  Does playing violent video games cause aggression? A longitudinal intervention study , 2018, Molecular Psychiatry.

[13]  A. Konar,et al.  A type-2 fuzzy classifier for gesture induced pathological disorder recognition , 2016, Fuzzy Sets Syst..

[14]  Dale Peeples,et al.  Sadness, suicide, and their association with video game and internet overuse among teens: results from the youth risk behavior survey 2007 and 2009. , 2011, Suicide & life-threatening behavior.

[15]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[16]  Mark C. Coulson.  Attributing Emotion to Static Body Postures: Recognition Accuracy, Confusions, and Viewpoint Dependence , 2004 .

[17]  U. Rajendra Acharya,et al.  Deep convolutional neural network for the automated detection and diagnosis of seizure using EEG signals , 2017, Comput. Biol. Medicine.

[18]  Roddy Cowie,et al.  Describing the emotional states that are expressed in speech , 2003, Speech Commun..

[19]  K. Scherer,et al.  The Body Action and Posture Coding System (BAP): Development and Reliability , 2012 .

[20]  Sriparna Saha,et al.  EEG based gesture mimicking by an artificial limb using cascade-correlation learning architecture , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[21]  L. H. Viet,et al.  Emotion Detection in the Loop from Brain Signals and Facial Images , 2006 .

[22]  Yuliyan Velchev,et al.  Human Emotion Estimation from EEG and Face Using Statistical Features and SVM , 2017 .

[23]  C. Anderson,et al.  Violent video game effects on aggression, empathy, and prosocial behavior in eastern and western countries: a meta-analytic review. , 2010, Psychological bulletin.

[24]  Martin Steinert,et al.  Capturing emotion reactivity through physiology measurement as a foundation for affective engineering in engineering design science and engineering practices , 2017, J. Intell. Manuf..

[25]  Veikko Surakka,et al.  Emotions and heart rate while sitting on a chair , 2005, CHI.

[26]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[27]  Beate M Herbert,et al.  Heart rate response after emotional picture presentation is modulated by interoceptive awareness. , 2007, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[28]  Christian E. Waugh,et al.  What good are positive emotions in crises? A prospective study of resilience and emotions following the terrorist attacks on the United States on September 11th, 2001. , 2003, Journal of personality and social psychology.

[29]  Andrew K. Przybylski,et al.  Violent video game engagement is not associated with adolescents' aggressive behaviour: evidence from a registered report , 2019, Royal Society Open Science.

[30]  Andrew K. Przybylski,et al.  The Motivational Pull of Video Games: A Self-Determination Theory Approach , 2006 .

[31]  Pratyusha Rakshit,et al.  Hemodynamic Analysis for Cognitive Load Assessment and Classification in Motor Learning Tasks Using Type-2 Fuzzy Sets , 2019, IEEE Transactions on Emerging Topics in Computational Intelligence.

[32]  D. Watson,et al.  Development and validation of brief measures of positive and negative affect: the PANAS scales. , 1988, Journal of personality and social psychology.

[33]  Leontios J. Hadjileontiadis,et al.  Emotion Recognition from Brain Signals Using Hybrid Adaptive Filtering and Higher Order Crossings Analysis , 2010, IEEE Transactions on Affective Computing.

[34]  Pratyusha Rakshit,et al.  Emotion Recognition from the Lip-Contour of a Subject Using Artificial Bee Colony Optimization Algorithm , 2011, SEMCCO.

[35]  J. Gottman,et al.  Marital interaction and satisfaction: a longitudinal view. , 1989, Journal of consulting and clinical psychology.

[36]  Bao-Liang Lu,et al.  Multimodal emotion recognition using EEG and eye tracking data , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[37]  Myounghoon Jeon,et al.  Estimation of drivers' emotional states based on neuroergonmic equipment: an exploratory study using fNIRS , 2015, AutomotiveUI.

[38]  Christian E. Waugh,et al.  What good are positive emotions in crises? A prospective study of resilience and emotions following the terrorist attacks on the United States on September 11th, 2001. , 2003, Journal of personality and social psychology.

[39]  Stephanie E. Moser,et al.  Feeling good: autonomic nervous system responding in five positive emotions. , 2011, Emotion.

[40]  Samuel D. Bradley,et al.  Death with a story: How story impacts emotional, motivational, and physiological responses to first-person shooter video games , 2004 .

[41]  Charles D. Smith,et al.  Neural substrates of facial emotion processing using fMRI. , 2001, Brain research. Cognitive brain research.

[42]  Jie Huang,et al.  Video-based Sign Language Recognition without Temporal Segmentation , 2018, AAAI.

[43]  Francisco Esteves,et al.  Effects of Playing Violent Computer Games on Emotional Desensitization and Aggressive Behavior , 2011 .

[44]  Monica Fabiani,et al.  Learning to multitask: effects of video game practice on electrophysiological indices of attention and resource allocation. , 2011, Psychophysiology.

[45]  I. Homma,et al.  Breathing rhythms and emotions , 2008, Experimental physiology.

[46]  Ah Chung Tsoi,et al.  Face recognition: a convolutional neural-network approach , 1997, IEEE Trans. Neural Networks.

[47]  Meng Li,et al.  A Lip Contour Extraction Method Using Localized Active Contour Model with Automatic Parameter Selection , 2010, 2010 20th International Conference on Pattern Recognition.

[48]  Michael D. Robinson,et al.  Measures of emotion: A review , 2009, Cognition & emotion.

[49]  Jiahui Pan,et al.  Fusion of Facial Expressions and EEG for Multimodal Emotion Recognition , 2017, Comput. Intell. Neurosci..

[50]  R. Levenson.  The Autonomic Nervous System and Emotion , 2014 .

[51]  Luciano Stegagno,et al.  Blood pressure changes highlight gender differences in emotional reactivity to arousing pictures , 2005, Biological Psychology.

[52]  H. Abdi,et al.  Principal component analysis , 2010 .

[53]  Kari Torkkola,et al.  Feature Extraction by Non-Parametric Mutual Information Maximization , 2003, J. Mach. Learn. Res..

[54]  B. Bushman,et al.  The more you play, the more aggressive you become: A long-term experimental study of cumulative violent video game effects on hostile expectations and aggressive behavior , 2013 .