A new engineering approach to reveal correlation of physiological change and spontaneous expression from video images

Spontaneous expression is associated with physiological states, i.e., heart rate, respiration, oxygen saturation (SpO2%), and heart rate variability (HRV). There have yet not sufficient efforts to explore correlation of physiological change and spontaneous expression. This study aims to study how spontaneous expression is associated with physiological changes with an approved protocol or through the videos provided from Denver Intensity of Spontaneous Facial Action Database. Not like a posed expression, motion artefact in spontaneous expression is one of evitable challenges to be overcome in the study. To obtain a physiological signs from a region of interest (ROI), a new engineering approach is being developed with an artefact-reduction method consolidated 3D active appearance model (AAM) based track, affine transformation based alignment with opto-physiological mode based imaging photoplethysmography. Also, a statistical association spaces is being used to interpret correlation of spontaneous expressions and physiological states including their probability densities by means of Gaussian Mixture Model. The present work is revealing a new avenue of study associations of spontaneous expressions and physiological states with its prospect of applications on physiological and psychological assessment.

[1]  Maja Pantic,et al.  Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[2]  M. V. van Gemert,et al.  Noninvasive imaging of in vivo blood flow velocity using optical Doppler tomography. , 1997, Optics letters.

[3]  LinLin Shen,et al.  A review on Gabor wavelets for face recognition , 2006, Pattern Analysis and Applications.

[4]  Zahra Khalili,et al.  Emotion recognition system using brain and peripheral signals: Using correlation dimension to improve the results of EEG , 2009, 2009 International Joint Conference on Neural Networks.

[5]  Shang-Hong Lai,et al.  Learning partially-observed hidden conditional random fields for facial expression recognition , 2009, CVPR.

[6]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[7]  Jeffrey F. Cohn,et al.  Observer-based measurement of facial expression with the Facial Action Coding System. , 2007 .

[8]  S. Hsieh,et al.  Classifying Different Emotional States by Means of EEG-Based Functional Connectivity Patterns , 2014, PloS one.

[9]  J. Cohn,et al.  Automated Face Analysis for Affective Computing , 2015 .

[10]  F Scheffold,et al.  Dynamic laser speckle imaging of cerebral blood flow. , 2009, Optics express.

[11]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[12]  B. Girod,et al.  Facial Expression Analysis for Model-Based Coding of Video Sequences , 1997 .

[13]  Stephen E Greenwald,et al.  Validation of a device to measure arterial pulse wave velocity by a photoplethysmographic method. , 2002, Physiological measurement.

[14]  Bao-Liang Lu,et al.  Emotion classification based on gamma-band EEG , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[15]  Nicu Sebe,et al.  Facial expression recognition from video sequences: temporal and static modeling , 2003, Comput. Vis. Image Underst..

[16]  J. Lien,et al.  Automatic recognition of facial expressions using hidden markov models and estimation of expression intensity , 1998 .

[17]  Ioannis Patras,et al.  Fusion of facial expressions and EEG for implicit affective tagging , 2013, Image Vis. Comput..

[18]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[19]  Beat Fasel,et al.  Automati Fa ial Expression Analysis: A Survey , 1999 .

[20]  K Wårdell,et al.  Duplex laser Doppler perfusion imaging. , 1996, Microvascular research.

[21]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Olga Sourina,et al.  Real-Time Fractal-Based Valence Level Recognition from EEG , 2013, Trans. Comput. Sci..

[23]  Nicu Sebe,et al.  Facial expression recognition from video sequences , 2002, Proceedings. IEEE International Conference on Multimedia and Expo.

[24]  Marina L. Gavrilova,et al.  Transactions on computational science XII: special issue on cyberworlds , 2011 .

[25]  Sijung Hu,et al.  Opto-physiological modeling applied to photoplethysmographic cardiovascular assessment. , 2013, Journal of healthcare engineering.

[26]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[27]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[28]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[29]  Qiang Ji,et al.  Data-Free Prior Model for Facial Action Unit Recognition , 2013, IEEE Transactions on Affective Computing.

[30]  Maja Pantic,et al.  Combined Support Vector Machines and Hidden Markov Models for Modeling Facial Action Temporal Dynamics , 2007, ICCV-HCI.

[31]  G. Lu,et al.  A comparison of photoplethysmography and ECG recording to analyse heart rate variability in healthy subjects , 2009, Journal of medical engineering & technology.

[32]  Cordelia Schmid,et al.  A Performance Evaluation of Local Descriptors , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  Takeo Kanade,et al.  Real-time combined 2D+3D active appearance models , 2004, CVPR 2004.

[34]  Mohammad H. Mahoor,et al.  DISFA: A Spontaneous Facial Action Intensity Database , 2013, IEEE Transactions on Affective Computing.

[35]  M. Bartlett,et al.  Face image analysis by unsupervised learning and redundancy reduction , 1998 .

[36]  B. Schmeichel,et al.  Cardiac vagal control predicts spontaneous regulation of negative emotional expression and subsequent cognitive performance , 2010, Biological Psychology.