A Study of the Dynamic Relation between Physiological Changes and Spontaneous Expressions

Recent progress in Affective Computing (AC) has enabled integration of physiological cues and spontaneous expressions to reveal a subject’s emotional state. Due to the lack of an effective technique for evaluating multimodal correlations, experience and intuition play a main role in present AC studies when fusing affective cues or modalities, resulting in unexpected outcomes. This study seeks to demonstrate a dynamic correlation between two such affective cues, physiological changes and spontaneous expressions, which were obtained by a combination of stereo vision based tracking and imaging photoplethysmography (iPPG), with a designed protocol involving 20 healthy subjects. The two cues obtained were sampled into a Statistical Association Space (SAS) to evaluate their dynamic correlation. It is found that the probability densities in the SAS increase as the peaks in two cues are approached. Also the complex form of the high probability density region in the SAS suggests a nonlinear correlation between two cues. Finally the cumulative distribution on the zero time-difference surface is found to be small (<0.047) demonstrating a lack of simultaneity. These results show that the two cues have a close interrelation, that is both asynchronous and nonlinear, in which a peak of one cue heralds a peak in the other.

[1]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[2]  Jeffrey F. Cohn,et al.  The Timing of Facial Motion in posed and Spontaneous Smiles , 2003, Int. J. Wavelets Multiresolution Inf. Process..

[3]  Gwen Littlewort,et al.  Recognizing facial expression: machine learning and application to spontaneous behavior , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[4]  Shiqing Zhang,et al.  Facial Expression Recognition Using Sparse Representation , 2012 .

[5]  Martin D. Levine,et al.  Fully automated recognition of spontaneous facial expressions in videos using random forest classifiers , 2014, IEEE Transactions on Affective Computing.

[6]  Arie Yeredor.  Performance Analysis of GEVD-Based Source Separation With Second-Order Statistics , 2011, IEEE Transactions on Signal Processing.

[7]  Alex Zelinsky,et al.  Learning OpenCV---Computer Vision with the OpenCV Library (Bradski, G.R. et al.; 2008)[On the Shelf] , 2009, IEEE Robotics & Automation Magazine.

[8]  Deva Ramanan,et al.  Face detection, pose estimation, and landmark localization in the wild , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[10]  Sijung Hu,et al.  Opto-physiological modeling applied to photoplethysmographic cardiovascular assessment. , 2013, Journal of healthcare engineering.

[11]  Ioannis Patras,et al.  Fusion of facial expressions and EEG for implicit affective tagging , 2013, Image Vis. Comput..

[12]  Jingsheng Lei,et al.  A clustering ensemble: Two-level-refined co-association matrix with path-based transformation , 2015, Pattern Recognit..

[13]  K Wårdell,et al.  Duplex laser Doppler perfusion imaging. , 1996, Microvascular research.

[14]  Gwen Littlewort,et al.  Fully Automatic Facial Action Recognition in Spontaneous Behavior , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[15]  Johannes Wagner,et al.  From Physiological Signals to Emotions: Implementing and Comparing Selected Methods for Feature Extraction and Classification , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[16]  Robert S. Leiken,et al.  A User’s Guide , 2011 .

[17]  Hatice Gunes,et al.  Bi-modal emotion recognition from expressive face and body gestures , 2007, J. Netw. Comput. Appl..

[18]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[19]  Michael J. Pazzani,et al.  User Modeling for Adaptive News Access , 2000, User Modeling and User-Adapted Interaction.

[20]  Gary R. Bradski,et al.  Learning OpenCV 3: Computer Vision in C++ with the OpenCV Library , 2016 .

[21]  M. V. van Gemert,et al.  Noninvasive imaging of in vivo blood flow velocity using optical Doppler tomography. , 1997, Optics letters.

[22]  Aslak Grinsted,et al.  Nonlinear Processes in Geophysics Application of the Cross Wavelet Transform and Wavelet Coherence to Geophysical Time Series , 2022 .

[23]  Eva Cerezo,et al.  Scalable multimodal fusion for continuous affect sensing , 2011, 2011 IEEE Workshop on Affective Computational Intelligence (WACI).

[24]  Arthur C. Graesser,et al.  Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features , 2010, User Modeling and User-Adapted Interaction.

[25]  Ashish Kapoor,et al.  Multimodal affect recognition in learning environments , 2005, ACM Multimedia.

[26]  W. Cleveland,et al.  Locally Weighted Regression: An Approach to Regression Analysis by Local Fitting , 1988 .

[27]  W. Cleveland.  LOWESS: A Program for Smoothing Scatterplots by Robust Locally Weighted Regression , 1981 .

[28]  G. Lu,et al.  A comparison of photoplethysmography and ECG recording to analyse heart rate variability in healthy subjects , 2009, Journal of medical engineering & technology.

[29]  Olga Sourina,et al.  Real-Time Fractal-Based Valence Level Recognition from EEG , 2013, Trans. Comput. Sci..

[30]  Touradj Ebrahimi,et al.  Affect recognition based on physiological changes during the watching of music videos , 2012, TIIS.

[31]  Jing Xiao,et al.  Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[32]  W. G. Parrott,et al.  Emotions in social psychology : essential readings , 2001 .

[33]  Kasia Muldner,et al.  Emotion Sensors Go To School , 2009, AIED.

[34]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[35]  Fenglei Yang,et al.  A new engineering approach to reveal correlation of physiological change and spontaneous expression from video images , 2015, Photonics West - Biomedical Optics.

[36]  Qiang Ji,et al.  A Unified Probabilistic Framework for Spontaneous Facial Action Modeling and Understanding , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Scotty D. Craig,et al.  Integrating Affect Sensors in an Intelligent Tutoring System , 2004 .

[38]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Raphael C.-W. Phan,et al.  Robust spontaneous facial expression recognition using sparse representation , 2013 .

[40]  F Scheffold,et al.  Dynamic laser speckle imaging of cerebral blood flow. , 2009, Optics express.

[41]  Stephen E Greenwald,et al.  Validation of a device to measure arterial pulse wave velocity by a photoplethysmographic method. , 2002, Physiological measurement.

[42]  L. O. Svaasand,et al.  Remote plethysmographic imaging using ambient light. , 2008, Optics express.

[43]  Rafael A. Calvo,et al.  Detecting Naturalistic Expressions of Nonbasic Affect Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.