A face emotion tree structure representation with probabilistic recursive neural network modeling

This paper describes a novel structural approach to recognize the human facial features for emotion recognition. Conventionally, features extracted from facial images are represented by relatively poor representations, such as arrays or sequences, with a static data structure. In this study, we propose to extract facial expression features vectors as Localized Gabor Features (LGF) and then transform these feature vectors into FacE Emotion Tree Structures (FEETS) representation. It is an extension of the Human Face Tree Structures (HFTS) representation presented in (Cho and Wong in Lecture notes in computer science, pp 1245–1254, 2005). This facial representation is able to simulate as human perceiving the real human face and both the entities and relationship could contribute to the facial expression features. Moreover, a new structural connectionist architecture based on a probabilistic approach to adaptive processing of data structures is presented. The so-called probabilistic based recursive neural network (PRNN) model extended from Frasconi et al. (IEEE Trans Neural Netw 9:768–785, 1998) is developed to train and recognize human emotions by generalizing the FEETS representation. For empirical studies, we benchmarked our emotion recognition approach against other well known classifiers. Using the public domain databases, such as Japanese Female Facial Expression (JAFFE) (Lyons et al. in IEEE Trans Pattern Anal Mach Intell 21(12):1357–1362, 1999; Lyons et al. in third IEEE international conference on automatic face and gesture recognition, 1998) database and Cohn–Kanade AU-Coded Facial Expression (CMU) Database (Cohn et al. in 7th European conference on facial expression measurement and meaning, 1997), our proposed system might obtain an accuracy of about 85–95% for subject-dependent and subject-independent conditions. Moreover, by testing images having artifacts, the proposed model significantly supports the robust capability to perform facial emotion recognition.

[1]  Khashayar Khorasani,et al.  Facial expression recognition using constructive feedforward neural networks , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[2]  Paul A. Viola,et al.  Robust Real-time Object Detection , 2001 .

[3]  M. Mak,et al.  Estimation of Elliptical Basis Function Parameters by the Em Algorithm with Application to Speaker Veriication (final Version) Paper No.: Tnna069 , 2000 .

[4]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Siu-Yeung Cho,et al.  Probabilistic based recursive model for adaptive processing of data structures , 2008, Expert Syst. Appl..

[6]  Siu-Yeung Cho,et al.  Probabilistic Based Recursive Model for Face Recognition , 2005, FSKD.

[7]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[8]  Franck Davoine,et al.  Appearance factorization based facial expression recognition and synthesis , 2004, ICPR 2004.

[9]  L Sirovich,et al.  Low-dimensional procedure for the characterization of human faces. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[10]  W. Zheng,et al.  Facial expression recognition using kernel canonical correlation analysis (KCCA) , 2006, IEEE Transactions on Neural Networks.

[11]  Tommy W. S. Chow,et al.  Training multilayer neural networks using fast global learning algorithm - least-squares and penalized optimization methods , 1999, Neurocomputing.

[12]  Penio S. Penev,et al.  Local feature analysis: A general statistical theory for object representation , 1996 .

[13]  Timothy F. Cootes,et al.  Automatic Interpretation and Coding of Face Images Using Flexible Models , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Marian Stewart Bartlett,et al.  Classifying Facial Actions , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  A. Wierzbicka.  Emotions Across Languages And Cultures , 1999 .

[16]  Larry S. Davis,et al.  Human expression recognition from motion using a radial basis function network architecture , 1996, IEEE Trans. Neural Networks.

[17]  Leonid Perlovsky,et al.  Neural Networks and Intellect: Using Model-Based Concepts , 2000, IEEE Transactions on Neural Networks.

[18]  Guodong Guo,et al.  Learning from examples in the small sample case: face expression recognition , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[19]  Alessandro Sperduti,et al.  Supervised neural networks for the classification of structures , 1997, IEEE Trans. Neural Networks.

[20]  Sun-Yuan Kung,et al.  Decision-based neural networks with signal/image classification applications , 1995, IEEE Trans. Neural Networks.

[21]  John G. Daugman,et al.  Complete discrete 2-D Gabor transforms by neural networks for image analysis and compression , 1988, IEEE Trans. Acoust. Speech Signal Process..

[22]  Lawrence Sirovich,et al.  Application of the Karhunen-Loeve Procedure for the Characterization of Human Faces , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Ah Chung Tsoi,et al.  An improved algorithm for learning long-term dependency problems in adaptive processing of data structures , 2003, IEEE Trans. Neural Networks.

[24]  Michael David Kelly,et al.  Visual identification of people by computer , 1970 .

[25]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[26]  B. S. Manjunath,et al.  Texture Features for Browsing and Retrieval of Image Data , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  J. P. Jones,et al.  An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. , 1987, Journal of neurophysiology.

[28]  R. Adolphs,et al.  Cortical Systems for the Recognition of Emotion in Facial Expressions , 1996, The Journal of Neuroscience.

[29]  Volker Blanz,et al.  Component-Based Face Recognition with 3D Morphable Models , 2003, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[30]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[31]  Alessandro Sperduti,et al.  A general framework for adaptive processing of data structures , 1998, IEEE Trans. Neural Networks.

[32]  Chengjun Liu,et al.  Independent component analysis of Gabor features for face recognition , 2003, IEEE Trans. Neural Networks.

[33]  J. Thompson.  Development of facial expression of emotion in blind and seeing children. , 1941 .

[34]  Marian Stewart Bartlett,et al.  Viewpoint Invariant Face Recognition using Independent Component Analysis and Attractor Networks , 1996, NIPS.

[35]  Hongbin Zha,et al.  Modeling facial expression space for recognition , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[36]  Alexander H. Waibel,et al.  A real-time face tracker , 1996, Proceedings Third IEEE Workshop on Applications of Computer Vision. WACV'96.

[37]  Ah Chung Tsoi,et al.  Adaptive processing of data structures , 1999, Proceedings Third International Conference on Computational Intelligence and Multimedia Applications. ICCIMA'99 (Cat. No.PR00300).

[38]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[39]  J. Daugman.  Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[40]  S. Baron-Cohen.  Mindblindness: An Essay on Autism and Theory of Mind , 1997 .

[41]  Yoshua Bengio,et al.  Input-output HMMs for sequence processing , 1996, IEEE Trans. Neural Networks.

[42]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[43]  Sun-Yuan Kung,et al.  Face recognition/detection by probabilistic decision-based neural network , 1997, IEEE Trans. Neural Networks.

[44]  John Platt,et al.  Fast training of svms using sequential minimal optimization , 1998 .

[45]  David W. Aha,et al.  Instance-Based Learning Algorithms , 1991, Machine Learning.

[46]  Kenji Mase,et al.  Recognition of Facial Expression from Optical Flow , 1991 .

[47]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[48]  S Marcelja,et al.  Mathematical description of the responses of simple cortical cells. , 1980, Journal of the Optical Society of America.

[49]  Alessio Micheli,et al.  A general framework for unsupervised processing of structured data , 2004, Neurocomputing.

[50]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[51]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[52]  Pat Langley,et al.  Estimating Continuous Distributions in Bayesian Classifiers , 1995, UAI.

[53]  A. Isen,et al.  Positive affect and decision making. , 1993 .

[54]  Witold Pedrycz,et al.  Face recognition: A study in information fusion using fuzzy integral , 2005, Pattern Recognit. Lett..

[55]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[56]  Michael J. Lyons,et al.  Automatic Classification of Single Facial Images , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[57]  Roberto Brunelli,et al.  Face Recognition: Features Versus Templates , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[58]  Roy L. Streit,et al.  Maximum likelihood training of probabilistic neural networks , 1994, IEEE Trans. Neural Networks.

[59]  Stephen J. Roberts,et al.  A Probabilistic Resource Allocating Network for Novelty Detection , 1994, Neural Computation.