Macro- and micro-expression spotting in long videos using spatio-temporal strain

We propose a method for the automatic spotting (temporal segmentation) of facial expressions in long videos comprising of macro- and micro-expressions. The method utilizes the strain impacted on the facial skin due to the non-rigid motion caused during expressions. The strain magnitude is calculated using the central difference method over the robust and dense optical flow field observed in several regions (chin, mouth, cheek, forehead) on each subject's face. This new approach is able to successfully detect and distinguish between large expressions (macro) and rapid and localized expressions (micro). Extensive testing was completed on a dataset containing 181 macro-expressions and 124 micro-expressions. The dataset consists of 56 videos collected at USF, 6 videos from the Canal-9 political debates, and 3 low quality videos found on the internet. A spotting accuracy of 85% was achieved for macro-expressions and 74% of all micro-expressions were spotted.

[1]  Alan F. Smeaton,et al.  Detector adaptation by maximising agreement between independent data sources , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Dmitry B. Goldgof,et al.  Towards macro- and micro-expression spotting in video using strain patterns , 2009, 2009 Workshop on Applications of Computer Vision (WACV).

[3]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[4]  Garrison W. Cottrell,et al.  Representing Face Images for Emotion Classification , 1996, NIPS.

[5]  P. Ekman.  Telling lies: clues to deceit in the marketplace , 1985 .

[6]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[8]  Dmitry B. Goldgof,et al.  Face recognition under camouflage and adverse illumination , 2010, 2010 Fourth IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS).

[9]  Sridhar Godavarthy,et al.  Microexpression spotting in video using optical strain , 2010 .

[10]  Dmitry B. Goldgof,et al.  Facial Strain Pattern as a Soft Forensic Evidence , 2007, 2007 IEEE Workshop on Applications of Computer Vision (WACV '07).

[11]  P. Ekman.  Facial expressions of emotion: an old controversy and new findings. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[12]  Alessandro Vinciarelli,et al.  Canal9: A database of political debates for analysis of social interactions , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[13]  G. Healey,et al.  Face recognition in hyperspectral images - Pattern Analysis and Machine Intelligence, IEEE Transactions on , 2001 .

[14]  Michael J. Black,et al.  Recognizing Facial Expressions in Image Sequences Using Local Parameterized Models of Image Motion , 1997, International Journal of Computer Vision.