Histogram-based local descriptors for facial expression recognition (FER): A comprehensive study

Abstract This paper aims to present histogram-based local descriptors applied to Facial Expression Recognition (FER) from static images and provide a systematic review and analysis of them. First, we describe the main steps in encoding binary patterns in a local patch, which are required in every histogram-based local descriptor. Then, we list the existing local descriptors, while analysing their strengths and weaknesses. Finally, we present the experimental results of all these descriptors on commonly used facial expression databases, with varying resolution, noise, occlusion, and number of sub-regions, as well as comparing them with the results obtained by the state-of-the-art deep learning methods. This paper aims to bring together different studies of the visual features for FER by evaluating their performances under the same experimental setup, and critically reviewing various classifiers making use of the local descriptors.

[1]  Mohammad Shahidul Islam,et al.  Facial Expression Recognition using Local Arc Pattern , 2014 .

[2]  Maja Pantic,et al.  Automatic Analysis of Facial Actions: A Survey , 2019, IEEE Transactions on Affective Computing.

[3]  Faisal Ahmed,et al.  Gradient directional pattern: A robust feature descriptor for facial expression recognition , 2012 .

[4]  Shutao Li,et al.  Face recognition using Weber local descriptors , 2013, Neurocomputing.

[5]  Simon C. K. Shiu,et al.  Monogenic Binary Coding: An Efficient Local Feature Extraction Approach to Face Recognition , 2012, IEEE Transactions on Information Forensics and Security.

[6]  Hubert Konik,et al.  Framework for reliable, real-time facial expression recognition for low resolution images , 2013, Pattern Recognit. Lett..

[7]  Kostas Karpouzis,et al.  The HUMAINE Database: Addressing the Collection and Annotation of Naturalistic and Induced Emotional Data , 2007, ACII.

[8]  Matti Pietikäinen,et al.  Local binary features for texture classification: Taxonomy and experimental study , 2017, Pattern Recognit..

[9]  Zhiyong Zeng,et al.  A new image retrieval model based on monogenic signal representation , 2015, J. Vis. Commun. Image Represent..

[10]  Oksam Chae,et al.  Local Directional Number Pattern for Face Analysis: Face and Expression Recognition , 2013, IEEE Transactions on Image Processing.

[11]  Roberto Paredes,et al.  Local Deep Neural Networks for gender recognition , 2016, Pattern Recognit. Lett..

[12]  Wen Gao,et al.  Histogram of Gabor Phase Patterns (HGPP): A Novel Object Representation Approach for Face Recognition , 2007, IEEE Transactions on Image Processing.

[13]  Jiwen Lu,et al.  Simultaneous Local Binary Feature Learning and Encoding for Homogeneous and Heterogeneous Face Recognition , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Balasubramanian Raman,et al.  Local tri-directional patterns: A new texture feature descriptor for image retrieval , 2016, Digit. Signal Process..

[15]  Masahide Kaneko,et al.  Facial Expression Recognition Based on Local Fourier Coefficients and Facial Fourier Descriptors , 2017 .

[16]  Uipil Chong,et al.  Facial Expression Recognition Using Local Transitional Pattern on Gabor Filtered Facial Images , 2013 .

[17]  Loris Nanni,et al.  Local binary patterns variants as texture descriptors for medical image analysis , 2010, Artif. Intell. Medicine.

[18]  Jiwen Lu,et al.  Learning Rotation-Invariant Local Binary Descriptor , 2017, IEEE Transactions on Image Processing.

[19]  Kuo-Chin Fan,et al.  A Novel Local Pattern Descriptor—Local Vector Pattern in High-Order Derivative Space for Face Recognition , 2014, IEEE Transactions on Image Processing.

[20]  Michael Felsberg,et al.  The monogenic signal , 2001, IEEE Trans. Signal Process..

[21]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[22]  Matti Pietikäinen,et al.  Face Description with Local Binary Patterns: Application to Face Recognition , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Jiwen Lu,et al.  Cost-Sensitive Local Binary Feature Learning for Facial Age Estimation , 2015, IEEE Transactions on Image Processing.

[24]  Matti Pietikäinen,et al.  Median Robust Extended Local Binary Pattern for Texture Classification , 2016, IEEE Trans. Image Process..

[25]  Baochang Zhang,et al.  Local Derivative Pattern Versus Local Binary Pattern: Face Recognition With High-Order Local Pattern Descriptor , 2010, IEEE Transactions on Image Processing.

[26]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  Loris Nanni,et al.  Survey on LBP based texture descriptors for image classification , 2012, Expert Syst. Appl..

[28]  Matti Pietikäinen,et al.  Spatiotemporal Local Monogenic Binary Patterns for Facial Expression Recognition , 2012, IEEE Signal Processing Letters.

[29]  Steven J. Kirsh,et al.  Violent video game play impacts facial emotion recognition. , 2007, Aggressive behavior.

[30]  Emam Hossain,et al.  Automated Facial Expression Recognition Using Gradient-Based Ternary Texture Patterns , 2013 .

[31]  Kin-Man Lam,et al.  Multi-resolution feature fusion for face recognition , 2014, Pattern Recognit..

[32]  Rassoul Amirfattahi,et al.  Local derivative radial patterns: A new texture descriptor for content-based image retrieval , 2017, Signal Process..

[33]  Changxin Gao,et al.  Face recognition with Riesz binary pattern , 2016, Digit. Signal Process..

[34]  Zafer Aydin,et al.  BAUM-2: a multilingual audio-visual affective face database , 2014, Multimedia Tools and Applications.

[35]  Krystian Mikolajczyk,et al.  Binary Online Learned Descriptors , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Oksam Chae,et al.  Local Directional Texture Pattern image descriptor , 2015, Pattern Recognit. Lett..

[37]  Xin-Ping Guan,et al.  A novel face recognition method based on IWLD and IWBC , 2015, Multimedia Tools and Applications.

[38]  Ville Ojansivu,et al.  Blur Insensitive Texture Classification Using Local Phase Quantization , 2008, ICISP.

[39]  Satish Kumar Singh,et al.  Local Gradient Hexa Pattern: A Descriptor for Face Recognition and Retrieval , 2022, IEEE Transactions on Circuits and Systems for Video Technology.

[40]  Jiwen Lu,et al.  Context-Aware Local Binary Feature Learning for Face Recognition , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[42]  Jiwen Lu,et al.  Learning Compact Binary Face Descriptor for Face Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  M. Islam.  Gender Classification Using Gradient Direction Pattern , 2013, ArXiv.

[44]  Matti Pietikäinen,et al.  Block-Based Methods for Image Retrieval Using Local Binary Patterns , 2005, SCIA.

[45]  Jianfei Cai,et al.  LETRIST: Locally Encoded Transform Feature Histogram for Rotation-Invariant Texture Classification , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[46]  P. Ekman.  Darwin, Deception, and Facial Expression , 2003, Annals of the New York Academy of Sciences.

[47]  Mohammad Shahidol Islam.  Local gradient pattern - A novel feature representation for facial expression recognition , 2014 .

[48]  Illah R. Nourbakhsh,et al.  A survey of socially interactive robots , 2003, Robotics Auton. Syst..

[49]  Songlin Du,et al.  Local spiking pattern and its application to rotation- and illumination-invariant texture classification , 2016 .

[50]  Shuicheng Yan,et al.  Peak-Piloted Deep Network for Facial Expression Recognition , 2016, ECCV.

[51]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[52]  Kin-Man Lam,et al.  Simplified Gabor wavelets for human face recognition , 2008, Pattern Recognit..