Towards macro- and micro-expression spotting in video using strain patterns

This paper presents a novel method for automatic spotting (temporal segmentation) of facial expressions in long videos comprising of continuous and changing expressions. The method utilizes the strain impacted on the facial skin due to the non-rigid motion caused during expressions. The strain magnitude is calculated using the central difference method over the robust and dense optical flow field of each subjects face. Testing has been done on 2 datasets (which includes 100 macro-expressions) and promising results have been obtained. The method is robust to several common drawbacks found in automatic facial expression segmentation including moderate in-plane and out-of-plane motion. Additionally, the method has also been modified to work with videos containing micro-expressions. Micro-expressions are detected utilizing their smaller spatial and temporal extent. A subject's face is divided in to sub-regions (mouth, cheeks, forehead, and eyes) and facial strain is calculated for each of these regions. Strain patterns in individual regions are used to identify subtle changes which facilitate the detection of micro-expressions.

[1]  David McLean,et al.  Silent talker: a new computer-based system for the analysis of facial cues to deception , 2006 .

[2]  Alex Pentland,et al.  Facial expression recognition using a dynamic model and motion energy , 1995, Proceedings of IEEE International Conference on Computer Vision.

[3]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[4]  Larry S. Davis,et al.  A probabilistic framework for rigid and non-rigid appearance based tracking and recognition , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[5]  Dmitry B. Goldgof,et al.  Finite element modeling of facial deformation in videos for computing strain pattern , 2008, 2008 19th International Conference on Pattern Recognition.

[6]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[7]  P. Ekman,et al.  Unmasking the face : a guide to recognizing emotions from facial clues , 1975 .

[8]  Rogério Schmidt Feris,et al.  Manifold Based Analysis of Facial Expression , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[9]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[10]  Fernando De la Torre,et al.  Temporal Segmentation of Facial Behavior , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[11]  Narendra Ahuja,et al.  Facial expression decomposition , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[12]  P. Ekman.  Facial expressions of emotion: an old controversy and new findings. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[13]  Rogério Schmidt Feris,et al.  Manifold Based Analysis of Facial Expression , 2004, CVPR Workshops.

[14]  Faouzi Kallel,et al.  Tissue elasticity reconstruction using linear perturbation method , 1996, IEEE Trans. Medical Imaging.

[15]  Dmitry B. Goldgof,et al.  Facial Strain Pattern as a Soft Forensic Evidence , 2007, 2007 IEEE Workshop on Applications of Computer Vision (WACV '07).

[16]  Ahmed M. Elgammal,et al.  Facial Expression Analysis Using Nonlinear Decomposable Generative Models , 2005, AMFG.