Image Ratio Features for Facial Expression Recognition Application

Video-based facial expression recognition is a challenging problem in computer vision and human-computer interaction. To target this problem, texture features have been extracted and widely used, because they can capture image intensity changes raised by skin deformation. However, existing texture features encounter problems with albedo and lighting variations. To solve both problems, we propose a new texture feature called image ratio features. Compared with previously proposed texture features, e.g., high gradient component features, image ratio features are more robust to albedo and lighting variations. In addition, to further improve facial expression recognition accuracy based on image ratio features, we combine image ratio features with facial animation parameters (FAPs), which describe the geometric motions of facial feature points. The performance evaluation is based on the Carnegie Mellon University Cohn-Kanade database, our own database, and the Japanese Female Facial Expression database. Experimental results show that the proposed image ratio feature is more robust to albedo and lighting variations, and the combination of image ratio features and FAPs outperforms each feature alone. In addition, we study asymmetric facial expressions based on our own facial expression database and demonstrate the superior performance of our combined expression recognition system.

[1]  Pong C. Yuen,et al.  Face Recognition by Regularized Discriminant Analysis , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[2]  Yuxiao Hu,et al.  Learning a Spatially Smooth Subspace for Face Recognition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Wei-Yang Lin,et al.  Optimal Linear Combination of Facial Regions for Improving Identification Performance , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[4]  Zhiwei Zhu,et al.  Robust Real-Time Face Pose and Facial Expression Recovery , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[5]  Arun Ross,et al.  A Mosaicing Scheme for Pose-Invariant Face Recognition , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[6]  Qiang Ji,et al.  Facial expression understanding in image sequences using dynamic and active visual information fusion , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[7]  Chun Chen,et al.  Sketch Based Facial Expression Recognition Using Graphics Hardware , 2005, ACII.

[8]  Maja Pantic,et al.  Biologically vs. Logic Inspired Encoding of Facial Actions and Emotions in Video , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[9]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[10]  Sungsoo Park,et al.  Facial expression analysis with facial expression deformation , 2008, 2008 19th International Conference on Pattern Recognition.

[11]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Guodong Guo,et al.  Learning from examples in the small sample case: face expression recognition , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[13]  Timothy F. Cootes,et al.  Comparing Active Shape Models with Active Appearance Models , 1999, BMVC.

[14]  Ashok Samal,et al.  How effective are landmarks and their geometry for face recognition? , 2006, Comput. Vis. Image Underst..

[15]  Nicu Sebe,et al.  Authentic Facial Expression Analysis , 2004, FGR.

[16]  Khashayar Khorasani,et al.  Facial expression recognition using constructive feedforward neural networks , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[17]  Hadi Seyedarabi,et al.  Recognition of six basic facial expressions by feature-points tracking using RBF neural network and fuzzy inference system , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[18]  Qiang Ji,et al.  Active and dynamic information fusion for facial expression understanding from image sequences , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Rogério Schmidt Feris,et al.  Manifold Based Analysis of Facial Expression , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[20]  Zicheng Liu,et al.  Expressive expression mapping with ratio images , 2001, SIGGRAPH.

[21]  Fabio Lavagetto,et al.  The facial animation engine: toward a high-level interface for the design of MPEG-4 compliant animated faces , 1999, IEEE Trans. Circuits Syst. Video Technol..

[22]  Hujun Bao,et al.  Understanding the Power of Clause Learning , 2009, IJCAI.

[23]  Takeo Kanade,et al.  Subtly different facial expression recognition and expression intensity estimation , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[24]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[25]  Pong C. Yuen,et al.  Incremental Linear Discriminant Analysis for Face Recognition , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[26]  Narendra Ahuja,et al.  Facial expression decomposition , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[27]  Christoph Bregler,et al.  Eigen-points [image matching] , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[28]  Takeo Kanade,et al.  Facial Expression Analysis , 2011, AMFG.

[29]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[30]  Thierry Pun,et al.  Valence-arousal evaluation using physiological signals in an emotion recall paradigm , 2007, 2007 IEEE International Conference on Systems, Man and Cybernetics.

[31]  Maja Pantic,et al.  Motion history for facial action detection in video , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[32]  Peter W. McOwan,et al.  A real-time automated system for the recognition of human facial expressions , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[33]  Aleix M. Martínez,et al.  Recognizing Imprecisely Localized, Partially Occluded, and Expression Variant Faces from a Single Sample per Class , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[34]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[35]  Kai-Kuang Ma,et al.  Rotation-invariant and scale-invariant Gabor features for texture image retrieval , 2007, Image Vis. Comput..

[36]  Fernando De la Torre,et al.  Facial Expression Analysis , 2011, Visual Analysis of Humans.

[37]  Nicu Sebe,et al.  Facial expression recognition from video sequences: temporal and static modeling , 2003, Comput. Vis. Image Underst..

[38]  Siu Cheung Hui,et al.  Facial expression recognition from line-based caricatures , 2003, IEEE Trans. Syst. Man Cybern. Part A.

[39]  Guodong Guo,et al.  Simultaneous feature selection and classifier training via linear programming: a case study for face expression recognition , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..