Extraction and Selection of Muscle Based Features for Facial Expression Recognition

In this study we propose a new set of muscle activity based features for facial expression recognition. We extract muscular activities by observing the displacements of facial feature points in an expression video. The facial feature points are initialized on muscular regions of influence in the first frame of the video. These points are tracked through optical flow in sequential frames. Displacements of feature points on the image plane are used to estimate the 3D orientation of a head model and relative displacements of its vertices. We model the human skin as a linear system of equations. The estimated deformation of the wireframe model produces an over-determined system of equations that can be solved under the constraint of the facial anatomy to obtain muscle activation levels. We apply sequential forward feature selection to choose the most descriptive set of muscles for recognition of basic facial expressions.

[1]  Takeo Kanade,et al.  Neural Network-Based Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Kun Lu,et al.  Facial Expression Recognition from Image Sequences Based on Feature Points and Canonical Correlations , 2010, 2010 International Conference on Artificial Intelligence and Computational Intelligence.

[3]  P. Ekman.  Pictures of Facial Affect , 1976 .

[4]  M. Taner Eskil,et al.  Machine Vision and Applications Manuscript No. Nearest Neighbor Weighted Average Customization for Modeling Faces , 2022 .

[5]  Gwen Littlewort,et al.  Recognizing facial expression: machine learning and application to spontaneous behavior , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[6]  Ioannis Pitas,et al.  Texture and shape information fusion for facial expression and facial action unit recognition , 2008, Pattern Recognit..

[7]  Kun Zhang,et al.  Facial expression recognition using geometric and appearance features , 2012, ICIMCS '12.

[8]  Qiang Ji,et al.  Information extraction from image sequences of real-world facial expressions , 2005, Machine Vision and Applications.

[9]  P. Knapp.  Expression of the emotions in man , 1963 .

[10]  WatersKeith.  A muscle model for animation three-dimensional facial expression , 1987 .

[11]  Fadi Dornaika,et al.  Fitting 3D face models for tracking and active appearance model training , 2006, Image Vis. Comput..

[12]  Changbo Hu,et al.  AAM derived face representations for robust facial action recognition , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[13]  A. Mehrabian.  Silent Messages: Implicit Communication of Emotions and Attitudes , 1971 .

[14]  Guodong Guo,et al.  Learning from examples in the small sample case: face expression recognition , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[15]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[16]  Keith Waters,et al.  A muscle model for animation three-dimensional facial expression , 1987, SIGGRAPH.

[17]  Ioannis Pitas,et al.  Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines , 2007, IEEE Transactions on Image Processing.

[18]  P. Ekman,et al.  Handbook of methods in nonverbal behavior research , 1982 .

[19]  Gwen Littlewort,et al.  Discrimination of Moderate and Acute Drowsiness Based on Spontaneous Facial Expressions , 2010, 2010 20th International Conference on Pattern Recognition.

[20]  C. Darwin,et al.  The Expression of the Emotions in Man and Animals , 1956 .

[21]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[22]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[23]  Maja Pantic,et al.  Action unit detection using sparse appearance descriptors in space-time video volumes , 2011, Face and Gesture 2011.

[24]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[25]  Maja Pantic,et al.  Motion history for facial action detection in video , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[26]  Gwen Littlewort,et al.  Fully Automatic Facial Action Recognition in Spontaneous Behavior , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[27]  M. Taner Eskil,et al.  Facial expression recognition based on anatomy , 2014, Comput. Vis. Image Underst..

[28]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[29]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[30]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[31]  Jacob Whitehill,et al.  Haar features for FACS AU recognition , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[32]  Takeo Kanade,et al.  A statistical method for 3D object detection applied to faces and cars , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[33]  J. N. Bassili.  Facial motion in the perception of faces and of emotional expression. , 1978, Journal of experimental psychology. Human perception and performance.

[34]  Daijin Kim,et al.  Natural facial expression recognition using differential-AAM and manifold learning , 2009, Pattern Recognit..

[35]  Hadi Seyedarabi,et al.  Analysis and Synthesis of Facial Expressions by Feature- Points Tracking and Deformable Model , 2007 .

[36]  Christopher G. Goetz.  The Mechanism of Human Facial Expression (Studies in Emotion and Social Interaction) , 1991, Neurology.