Exploring the Feasibility of Face Video Based Instantaneous Heart-Rate for Micro-Expression Spotting

Facial micro-expressions (ME) are manifested by human reflexive behavior and thus they are useful to disclose the genuine human emotions. Their analysis plays a pivotal role in many real-world applications encompassing affective computing, biometrics and psychotherapy. The first and foremost step for ME analysis is ME spotting which refers to detection of ME affected frames from a video. ME spotting is a highly challenging research problem and even human experts cannot correctly perform it because MEs are manifested using subtle face deformations and that too for a short duration. It is well established that changes in the human emotions, not only manifest ME but it also introduces changes in instantaneous heart rate. Thus, the manifestation of ME and changes in the instantaneous heart rate are related to the change in human expressions and both of them are estimated using temporal deformations of the face. This provides the motivation of this paper that aims to explore the feasibility of variations in the instantaneous heart rate for performing the correct the ME spotting. Experimental results conducted on a publicly available spontaneous ME spotting dataset, reveal that the variations in instantaneous heart rate can be utilized to improve the ME spotting.

[1]  Peter Robinson,et al.  Constrained Local Neural Fields for Robust Facial Landmark Detection in the Wild , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[2]  Qiang Ji,et al.  Automatic Eye Detection and Its Validation , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[3]  Guoying Zhao,et al.  CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation , 2014, PloS one.

[4]  Puneet Gupta,et al.  Accurate heart-rate estimation from face videos using quality-based fusion , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[5]  Maja Pantic,et al.  Fully Automatic Facial Action Unit Detection and Temporal Analysis , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[6]  Yoshinori Kuno,et al.  Robust Heart Rate Measurement from Video Using Select Random Patches , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[7]  Matti Pietikäinen,et al.  Spatiotemporal Integration of Optical Flow Vectors for Micro-expression Detection , 2015, ACIVS.

[8]  J. Gross,et al.  Hiding feelings: the acute effects of inhibiting negative and positive emotion. , 1997, Journal of abnormal psychology.

[9]  E. A. Haggard,et al.  Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy , 1966 .

[10]  Puneet Gupta,et al.  An efficient slap fingerprint segmentation and hand classification algorithm , 2014, Neurocomputing.

[11]  Hugo D. Critchley,et al.  Activity in the human brain predicting differential heart rate responses to emotional facial expressions , 2005, NeuroImage.

[12]  Daniel McDuff,et al.  Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam , 2011, IEEE Transactions on Biomedical Engineering.

[13]  Frédo Durand,et al.  Detecting Pulse from Head Motions in Video , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Matti Pietikäinen,et al.  Spotting Rapid Facial Movements from Videos Using Appearance-Based Feature Difference Analysis , 2014, 2014 22nd International Conference on Pattern Recognition.

[15]  Mohammad H. Mahoor,et al.  DISFA: A Spontaneous Facial Action Intensity Database , 2013, IEEE Transactions on Affective Computing.

[16]  KokSheik Wong,et al.  Subtle Expression Recognition Using Optical Strain Weighted Features , 2014, ACCV Workshops.

[17]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[18]  Kwang Suk Park,et al.  ROI analysis for remote photoplethysmography on facial video , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[19]  Puneet Gupta,et al.  Serial fusion of Eulerian and Lagrangian approaches for accurate heart-rate estimation using face videos , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[20]  H. Emrah Tasli,et al.  Remote PPG based vital sign measurement using adaptive facial regions , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[21]  Matti Pietikäinen,et al.  A Spontaneous Micro-expression Database: Inducement, collection and baseline , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[22]  John See,et al.  Eulerian emotion magnification for subtle expression recognition , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[23]  KokSheik Wong,et al.  Spontaneous Subtle Expression Detection and Recognition based on Facial Strain , 2016, Signal Process. Image Commun..

[24]  Matti Pietikäinen,et al.  Towards Reading Hidden Emotions: A Comparative Study of Spontaneous Micro-Expression Spotting and Recognition Methods , 2015, IEEE Transactions on Affective Computing.

[25]  Kewei Chen,et al.  Neural correlates of heart rate variability during emotion , 2009, NeuroImage.

[26]  Xiaolan Fu,et al.  CAS(ME)$^2$ : A Database for Spontaneous Macro-Expression and Micro-Expression Spotting and Recognition , 2018, IEEE Transactions on Affective Computing.

[27]  Frédo Durand,et al.  Eulerian video magnification for revealing subtle changes in the world , 2012, ACM Trans. Graph..

[28]  Sujing Wang,et al.  A main directional maximal difference analysis for spotting facial movements from long-term videos , 2017, Neurocomputing.

[29]  Zhilin Zhang,et al.  TROIKA: A General Framework for Heart Rate Monitoring Using Wrist-Type Photoplethysmographic Signals During Intensive Physical Exercise , 2014, IEEE Transactions on Biomedical Engineering.

[30]  Sander Stuijk,et al.  Algorithmic Principles of Remote PPG , 2017, IEEE Transactions on Biomedical Engineering.

[31]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[32]  E. Tan,et al.  Predicting advertising effectiveness by facial expressions in response to amusing persuasive stimuli. , 2014 .

[33]  Paul Ekman,et al.  Lie Catching and Microexpressions , 2009 .

[34]  Majid Sarrafzadeh,et al.  A Self-Calibrating Radar Sensor System for Measuring Vital Signs , 2016, IEEE Transactions on Biomedical Circuits and Systems.

[35]  Mika P. Tarvainen,et al.  An advanced detrending method with application to HRV analysis , 2002, IEEE Transactions on Biomedical Engineering.