Hierarchical Recognition Scheme for Human Facial Expression Recognition Systems

Over the last decade, human facial expressions recognition (FER) has emerged as an important research area. Several factors make FER a challenging research problem. These include varying light conditions in training and test images; need for automatic and accurate face detection before feature extraction; and high similarity among different expressions that makes it difficult to distinguish these expressions with a high accuracy. This work implements a hierarchical linear discriminant analysis-based facial expressions recognition (HL-FER) system to tackle these problems. Unlike the previous systems, the HL-FER uses a pre-processing step to eliminate light effects, incorporates a new automatic face detection scheme, employs methods to extract both global and local features, and utilizes a HL-FER to overcome the problem of high similarity among different expressions. Unlike most of the previous works that were evaluated using a single dataset, the performance of the HL-FER is assessed using three publicly available datasets under three different experimental settings: n-fold cross validation based on subjects for each dataset separately; n-fold cross validation rule based on datasets; and, finally, a last set of experiments to assess the effectiveness of each module of the HL-FER separately. Weighted average recognition accuracy of 98.7% across three different datasets, using three classifiers, indicates the success of employing the HL-FER for human FER.

[1]  Nobuyuki Otsu,et al.  Facial expression recognition using Fisher weight maps , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[2]  P. Shanmugavadivu,et al.  Image Inversion and Bi Level Histogram Equalization for Contrast Enhancement , 2010 .

[3]  Gwen Littlewort,et al.  Recognizing facial expression: machine learning and application to spontaneous behavior , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[4]  S. Gong,et al.  Conditional Mutual Information Based Boosting for Facial Expression Recognition , 2005 .

[5]  Yi-Cheng Zhang,et al.  Using SVM to design facial expression recognition for shape and texture features , 2010, 2010 International Conference on Machine Learning and Cybernetics.

[6]  Fernando De la Torre,et al.  Facial Expression Analysis , 2011, Visual Analysis of Humans.

[7]  Andy Harter,et al.  Parameterisation of a stochastic model for human face identification , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[8]  Azriel Rosenfeld,et al.  Angle Detection on Digital Curves , 1973, IEEE Transactions on Computers.

[9]  David J. Kriegman,et al.  From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Mohammed Yeasin,et al.  Recognition of facial expressions and measurement of levels of interest from video , 2006, IEEE Transactions on Multimedia.

[11]  Fan Chen,et al.  Facial Expression Recognition by Supervised Independent Component Analysis Using MAP Estimation , 2008, IEICE Trans. Inf. Syst..

[12]  Takeo Kanade,et al.  Face Recognition Across Pose and Illumination , 2011, Handbook of Face Recognition.

[13]  Takeo Kanade,et al.  Facial Expression Analysis , 2011, AMFG.

[14]  Xuelong Li,et al.  Iterative Subspace Analysis Based on Feature Line Distance , 2009, IEEE Trans. Image Process..

[15]  S. Sumathi,et al.  Automatic recognition and analysis of human faces and facial expression by LDA using wavelet transform , 2012, 2012 International Conference on Computer Communication and Informatics.

[16]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[17]  Beat Fasel,et al.  Automati Fa ial Expression Analysis: A Survey , 1999 .

[18]  Mariko Nakano-Miyatake,et al.  Eigenface-Gabor Algorithm for Features Extraction in Face Recognition , 2022 .

[19]  Shiqing Zhang,et al.  Facial Expression Recognition Based on Local Binary Patterns and Kernel Discriminant Isomap , 2011, Sensors.

[20]  Haidi Ibrahim,et al.  Image sharpening using sub-regions histogram equalization , 2009, IEEE Transactions on Consumer Electronics.

[21]  N. P. Reddy,et al.  Facial expression (mood) recognition from facial images using committee neural networks , 2009, Biomedical engineering online.

[22]  Michael J. Lyons,et al.  Automatic Classification of Single Facial Images , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Yan Xiaoguang,et al.  Facial Expression Recognition Based on Local Texture Features , 2011, 2011 14th IEEE International Conference on Computational Science and Engineering.

[24]  Karen Das,et al.  Recognition of Facial Expression Using Eigenvector Based Distributed Features and Euclidean Distance Based Decision Making Technique , 2013, ArXiv.

[25]  Mandeep Kaur,et al.  Recognition of Facial Expressions with Principal Component Analysis and Singular Value Decomposition , 2010 .

[26]  Oksam Chae,et al.  Facial expression recognition using Local Directional Pattern (LDP) , 2010, 2010 IEEE International Conference on Image Processing.

[27]  Oksam Chae,et al.  Local Directional Number Pattern for Face Analysis: Face and Expression Recognition , 2013, IEEE Transactions on Image Processing.

[28]  Karen L. Schmidt,et al.  Human facial expressions as adaptations: Evolutionary questions in facial expression research. , 2001, American journal of physical anthropology.

[29]  H. Bülthoff,et al.  The MPI Facial Expression Database — A Validated Database of Emotional and Conversational Facial Expressions , 2012, PloS one.

[30]  Andreas Christmann,et al.  Support vector machines , 2008, Data Mining and Knowledge Discovery Handbook.

[31]  Edward J Ciaccio Book review of applied medical image processing: A basic course , 2011, BioMedical Engineering OnLine.

[32]  Md. Zia Uddin,et al.  An enhanced independent component-based human facial expression recognition from video , 2009, IEEE Transactions on Consumer Electronics.

[33]  Jianhou Gan,et al.  Facial Expression Recognition Based on Local Directional Pattern Using SVM Decision-level Fusion , 2013 .

[34]  Shaogang Gong,et al.  Conditional Mutual Infomation Based Boosting for Facial Expression Recognition , 2005, BMVC.

[35]  H. Yeganeh,et al.  A novel approach for contrast enhancement based on Histogram Equalization , 2008, 2008 International Conference on Computer and Communication Engineering.

[36]  H. Bülthoff,et al.  The contribution of different facial regions to the recognition of conversational expressions. , 2008, Journal of vision.

[37]  Ioannis Pitas,et al.  Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines , 2007, IEEE Transactions on Image Processing.

[38]  Neeta Nain,et al.  A Hybrid Method of Feature Extraction for Facial Expression Recognition , 2011, 2011 Seventh International Conference on Signal Image Technology & Internet-Based Systems.

[39]  Thierry Bouwmans,et al.  Modeling of Dynamic Backgrounds by Type-2 Fuzzy Gaussians Mixture Models , 2010 .

[40]  Hong Yan,et al.  Locating head and face boundaries for head-shoulder images , 1999, Pattern Recognit..

[41]  S. V. Dudul,et al.  Human emotion recognition system using optimally designed SVM with different facial feature extraction techniques , 2008 .

[42]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[43]  Ravindra C. Thool,et al.  Automatic facial feature extraction and expression recognition based on neural network , 2012, ArXiv.

[44]  Ioannis Pitas,et al.  Application of non-negative and local non negative matrix factorization to facial expression recognition , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[45]  D. K. Basu,et al.  Face Recognition Using Appearance Based Approach: A Literature Survey , 2012 .

[46]  Jieyu Zhao,et al.  Curvelet feature extraction for face recognition and facial expression recognition , 2010, 2010 Sixth International Conference on Natural Computation.

[47]  Zhiwei Zhu,et al.  Robust Real-Time Face Pose and Facial Expression Recovery , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[48]  Mark G. Eramian,et al.  Histogram equalization using neighborhood metrics , 2005, The 2nd Canadian Conference on Computer and Robot Vision (CRV'05).

[49]  Lior Shamir,et al.  Evaluation of Face Datasets as Tools for Assessing the Performance of Face Recognition Methods , 2008, International Journal of Computer Vision.

[50]  M. Pietikäinen,et al.  FACIAL EXPRESSION RECOGNITION WITH LOCAL BINARY PATTERNS AND LINEAR PROGRAMMING , 2004 .

[51]  Pong C. Yuen,et al.  Multi-cues eye detection on gray intensity image , 2001, Pattern Recognit..

[52]  J V Tu,et al.  Advantages and disadvantages of using artificial neural networks versus logistic regression for predicting medical outcomes. , 1996, Journal of clinical epidemiology.

[53]  Gwen Littlewort,et al.  A Prototype for Automatic Recognition of Spontaneous Facial Actions , 2002, NIPS.

[54]  Takeo Kanade,et al.  Neural Network-Based Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[55]  Fan Chen,et al.  Facial Expression Recognition by SVM-based Two-stage Classifier on Gabor Features , 2007, MVA.

[56]  Mahesh M. Goyani,et al.  A literature survey on Facial Expression Recognition using Global Features , 2013 .

[57]  Friedhelm Schwenker,et al.  A Multiple Classifier System Approach for Facial Expressions in Image Sequences Utilizing GMM Supervectors , 2010, 2010 20th International Conference on Pattern Recognition.

[58]  J. Crowley,et al.  Estimating Face orientation from Robust Detection of Salient Facial Structures , 2004 .

[59]  Aggelos K. Katsaggelos,et al.  Automatic facial expression recognition using facial animation parameters and multistream HMMs , 2006, IEEE Transactions on Information Forensics and Security.

[60]  T. Sejnowski,et al.  Measuring facial expressions by computer image analysis. , 1999, Psychophysiology.

[61]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[62]  Hyun-Chul Kim,et al.  Constructing support vector machine ensemble , 2003, Pattern Recognit..

[63]  Qiong Liu,et al.  A robust skin color based face detection algorithm , 2010, 2010 2nd International Asia Conference on Informatics in Control, Automation and Robotics (CAR 2010).

[64]  Lianghua He,et al.  Boosted Independent Features for Face Expression Recognition , 2005, ISNN.

[65]  Goran Martinović,et al.  Emotion Recognition System by a Neural Network Based Facial Expression Analysis , 2013 .

[66]  Marti A. Hearst Trends & Controversies: Support Vector Machines , 1998, IEEE Intell. Syst..

[67]  Gwen Littlewort,et al.  Automatic Recognition of Facial Actions in Spontaneous Expressions , 2006, J. Multim..

[68]  Vinay Bettadapura,et al.  Face Expression Recognition and Analysis: The State of the Art , 2012, ArXiv.

[69]  Hua-jun Song,et al.  An Expression Space Model for Facial Expression Analysis , 2008, 2008 Congress on Image and Signal Processing.

[70]  Liakot Ali,et al.  Weighted Local Directional Pattern for Robust Facial Expression Recognition , 2011, 2011 First International Conference on Informatics and Computational Intelligence.

[71]  Oksam Chae,et al.  Local directional pattern variance (ldpv): a robust feature descriptor for facial expression recognition , 2012, Int. Arab J. Inf. Technol..

[72]  Agus Harjoko,et al.  A Neural Network based Facial Expression Recognition using Fisherface , 2012 .

[73]  Ingemar J. Cox,et al.  Feature-based face recognition using mixture-distance , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[74]  Ganesh K. Venayagamoorthy,et al.  Recognition of facial expressions using Gabor wavelets and learning vector quantization , 2008, Eng. Appl. Artif. Intell..

[75]  Xiubao Sui,et al.  Range Limited Bi-Histogram Equalization for image contrast enhancement , 2013 .

[76]  Senthil Ragavan Valayapalayam Kittusamy,et al.  Facial Expressions Recognition Using Eigenspaces , 2012 .

[77]  Daw-Tung Lin,et al.  Facial Expression Classification Using PCA and Hierarchical Radial Basis Function Network , 2006, J. Inf. Sci. Eng..

[78]  B. Prabhakaran,et al.  Facial Expression Recognition Using Dual Layer Hierarchical SVM Ensemble Classification , 2012, 2012 IEEE International Symposium on Multimedia.

[79]  M. Pietikäinen,et al.  Facial Expression Recognition with Local Binary Patterns and Linear Programming 1 , 2005 .

[80]  K. Fujii,et al.  Visualization for the analysis of fluid motion , 2005, J. Vis..

[81]  Bülent Sankur,et al.  Matching of faces in camera images and document photographs , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).