Emotion Understanding Using Multimodal Information Based on Autobiographical Memories for Alzheimer's Patients

Alzheimer Disease (AD) early detection is considered of high importance for improving the quality of life of patients and their families. Amongst all the different approaches for AD detection, significant work has been focused on emotion analysis through facial expressions, body language or speech. Many studies also use the electroencephalogram in order to capture emotions that patients cannot physically express. Our work introduces an emotion recognition approach using facial expression and EEG signal analysis. A novel dataset was created specifically to remark the autobiographical memory deficits of AD patients. This work uses novel EEG features based on quaternions, facial landmarks and the combination of them. Their performance was evaluated in a comparative study with a state of the art methods that demonstrates the proposed approach.

[1]  M. Mesulam,et al.  Alterations of visual search strategy in Alzheimer's disease and aging. , 2000, Neuropsychology.

[2]  Yan Bao,et al.  Impairment of Vocal Expression of Negative Emotions in Patients with Alzheimer’s Disease , 2014, Front. Aging Neurosci..

[3]  Ulrich Seidl,et al.  Johannes Schröder Symptoms Facial Expression in Alzheimer ' s Disease : Impact of Cognitive Deficits and Neuropsychiatric , 2012 .

[4]  Nicolas Le Bihan,et al.  Quaternion principal component analysis of color images , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[5]  G. Arbanas.  Diagnostic and Statistical Manual of Mental Disorders (DSM-5) , 2015 .

[6]  Johan Hagelbäck,et al.  Evaluating Classifiers for Emotion Recognition Using EEG , 2013, HCI.

[7]  Matti Pietikäinen,et al.  Boosted multi-resolution spatiotemporal descriptors for facial expression recognition , 2009, Pattern Recognit. Lett..

[8]  L. Maffei,et al.  Environmental enrichment strengthens corticocortical interactions and reduces amyloid-β oligomers in aged mice , 2013, Front. Aging Neurosci..

[9]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[10]  Hatice Gunes,et al.  Continuous Prediction of Spontaneous Affect from Multiple Cues and Modalities in Valence-Arousal Space , 2011, IEEE Transactions on Affective Computing.

[11]  Frederik Maes,et al.  Impaired recognition of body expressions in the behavioral variant of frontotemporal dementia , 2015, Neuropsychologia.

[12]  Björn W. Schuller,et al.  Abandoning emotion classes - towards continuous emotion recognition with modelling of long-range dependencies , 2008, INTERSPEECH.

[13]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[14]  William Rowan Hamilton,et al.  ON QUATERNIONS, OR ON A NEW SYSTEM OF IMAGINARIES IN ALGEBRA , 1847 .

[15]  Gwen Littlewort,et al.  Automatic coding of facial expressions displayed during posed and genuine pain , 2009, Image Vis. Comput..

[16]  Björn Schuller,et al.  Emotion Recognition in Naturalistic Speech and Language—A Survey , 2015 .

[17]  P. Rapp,et al.  Time domain measures of inter-channel EEG correlations: a comparison of linear, nonparametric and nonlinear measures , 2013, Cognitive Neurodynamics.

[18]  Aravind E. Vijayan,et al.  EEG-Based Emotion Recognition Using Statistical Measures and Auto-Regressive Modeling , 2015, 2015 IEEE International Conference on Computational Intelligence & Communication Technology.

[19]  Peter Robinson,et al.  OpenFace: An open source facial behavior analysis toolkit , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[20]  Maja Pantic,et al.  Particle filtering with factorized likelihoods for tracking facial features , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[21]  Maja Pantic,et al.  Facial Action Unit Detection using Probabilistic Actively Learned Support Vector Machines on Tracked Facial Point Data , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[22]  Kai-Kuang Ma,et al.  Complex Wavelet-Based Face Recognition Using Independent Component Analysis , 2009, 2009 Fifth International Conference on Intelligent Information Hiding and Multimedia Signal Processing.

[23]  Ivan Aprahamian,et al.  Eye movement analysis and cognitive processing: detecting indicators of conversion to Alzheimer’s disease , 2014, Neuropsychiatric disease and treatment.

[24]  Andrea Cavallaro,et al.  Local Zernike Moment Representation for Facial Affect Recognition , 2013, BMVC.

[25]  Subramanian Ramanathan,et al.  DECAF: MEG-Based Multimodal Database for Decoding Affective Physiological Responses , 2015, IEEE Transactions on Affective Computing.

[26]  Guoying Zhao,et al.  CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation , 2014, PloS one.

[27]  Bofeng Zhang,et al.  Correlation Between Forehead EEG and Sensorimotor Area EEG in Motor Imagery Task , 2009, 2009 Eighth IEEE International Conference on Dependable, Autonomic and Secure Computing.

[28]  Vinay Bettadapura,et al.  Face Expression Recognition and Analysis: The State of the Art , 2012, ArXiv.

[29]  Mohammad Soleymani,et al.  Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection , 2016, IEEE Transactions on Affective Computing.

[30]  Wei Lin Soh.  Emotion recognition using EEG signals , 2015 .

[31]  Mohamed Chetouani,et al.  Robust continuous prediction of human emotions using multiscale dynamic cues , 2012, ICMI '12.

[32]  Maja Pantic,et al.  Web-based database for facial expression analysis , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[33]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[34]  Gernot R. Müller-Putz,et al.  Electroencephalography (EEG) as a Research Tool in the Information Systems Discipline: Foundations, Measurement, and Applications , 2015, Commun. Assoc. Inf. Syst..

[35]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[36]  Andrea Cavallaro,et al.  Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Tülay Adali,et al.  Noncircular Principal Component Analysis and Its Application to Model Selection , 2011, IEEE Transactions on Signal Processing.

[38]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[39]  Maja Pantic,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON AFFECTIVE COMPUTING , 2022 .

[40]  Leontios J. Hadjileontiadis,et al.  Emotion Recognition from Brain Signals Using Hybrid Adaptive Filtering and Higher Order Crossings Analysis , 2010, IEEE Transactions on Affective Computing.

[41]  Tülay Adali,et al.  Complex-Valued Signal Processing: The Proper Way to Deal With Impropriety , 2011, IEEE Transactions on Signal Processing.

[42]  Meng Chen,et al.  Quaternion Fisher Discriminant Analysis for Bimodal Multi-feature Fusion , 2015, ECC.

[43]  M. Hornberger,et al.  Profiles of recent autobiographical memory retrieval in semantic dementia, behavioural-variant frontotemporal dementia, and Alzheimer's disease , 2011, Neuropsychologia.

[44]  Yau-Hwang Kuo,et al.  Emotion recognition based on a novel triangular facial feature extraction method , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[45]  Alexander Hammers,et al.  Neuroanatomical Correlates of Recognizing Face Expressions in Mild Stages of Alzheimer’s Disease , 2015, PloS one.

[46]  J. Turner.  Human Emotions: A Sociological Theory , 2007 .