Multi-Feature Tensor Neighborhood Preserving Embedding for 3D Facial Expression Recognition

To investigate an effective representation model for 3D facial expression recognition, this paper proposes a multi-feature tensor neighborhood preserving embedding (MFTNPE) method, which seeks various attribute features from raw textured shape scan models of 3D facial expression to construct a multi-feature tensor model. With MFTNPE method, we can project the model into a tensor subspace, explore the interactions and correlations among multimodal data and then extract discriminative low-dimensional tensor features for 3D facial expression recognition. In the study, we first introduce a tensor model based on multi-feature integration for 3D facial expression recognition. Via exploring the textural and geometric features of 3D facial expression data, an original multi-feature tensor model is constructed by stacking various attribute maps orderly. Second, we utilize the tensor completion approach as the scheme of data preprocessing to correct the original multi-feature tensor model. Due to the presence of imperfect data in the projected attribute maps, the tensor completion can predict the missing data in tensor by the trace norm of multi-feature tensor model along each mode. Third, we redefine a weighted matrix for use in the orthogonal-based tensor neighborhood preserving embedding algorithm. In the weighted matrix, the class information is used to guarantee the correct neighborhood relationships for multi-feature tensor samples. This effectively avoids the disturbances and confusions between similar samples while preserving spatial structure and inter-class information. Finally, we conduct extensive experiments on two 3D facial expression recognition datasets. The experimental results show that the proposed method has much better efficiency and performance compared with the state-of-the-art methods.

[1]  Xiaofeng Yuan,et al.  Deep learning with neighborhood preserving embedding regularization and its application for soft sensor in an industrial hydrocracking process , 2021, Inf. Sci..

[2]  Liangchen Hu,et al.  Orthogonal neighborhood preserving discriminant analysis with patch embedding for face recognition , 2020, Pattern Recognit..

[3]  José Marques Soares,et al.  Systematic review of 3D facial expression recognition methods , 2020, Pattern Recognit..

[4]  Gueesang Lee,et al.  A novel 2D and 3D multimodal approach for in-the-wild facial expression recognition , 2019, Image Vis. Comput..

[5]  Francesca Nonis,et al.  3D Approaches and Challenges in Facial Expression Recognition Algorithms—A Literature Review , 2019, Applied Sciences.

[6]  Qiuqi Ruan,et al.  FERLrTc: 2D+3D facial expression recognition via low-rank tensor completion , 2019, Signal Process..

[7]  Jin-Wei Wang,et al.  The new 3D facial expression recognition method based on semantic knowledge of Gaussian mixture model , 2019, 2019 IEEE International Conference of Intelligent Applied Systems on Engineering (ICIASE).

[8]  Vaneet Aggarwal,et al.  Tensor Train Neighborhood Preserving Embedding , 2017, IEEE Transactions on Signal Processing.

[9]  Q. M. Jonathan Wu,et al.  A survey of local feature methods for 3D face recognition , 2017, Pattern Recognit..

[10]  Jian Sun,et al.  Multimodal 2D+3D Facial Expression Recognition With Deep Fusion Convolutional Neural Network , 2017, IEEE Transactions on Multimedia.

[11]  Zhao Xiaoqiang,et al.  Tensor dynamic neighborhood preserving embedding algorithm for fault diagnosis of batch process , 2017 .

[12]  Sergio Escalera,et al.  Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Jian-Jiun Ding,et al.  Facial expression recognition based on improved local binary pattern and class-regularized locality preserving projection , 2015, Signal Process..

[14]  Xi Zhao,et al.  An efficient multimodal 2D + 3D feature-based approach to automatic facial expression recognition , 2015, Comput. Vis. Image Underst..

[15]  Asim Jan,et al.  Automatic 3D facial expression recognition using geometric and textured feature fusion , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[16]  Liming Chen,et al.  Automatic 3D facial expression recognition using geometric scattering representation , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[17]  Zemin Zhang,et al.  Exact Tensor Completion Using t-SVD , 2015, IEEE Transactions on Signal Processing.

[18]  Ibrahim Venkat,et al.  Analysis and evaluation of SURF descriptors for automatic 3D facial expression recognition using different classifiers , 2014, 2014 4th World Congress on Information and Communication Technologies (WICT 2014).

[19]  Andrzej Cichocki,et al.  Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis , 2014, IEEE Signal Processing Magazine.

[20]  Wei Zeng,et al.  An automatic 3D expression recognition framework based on sparse representation of conformal images , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[21]  Liming Chen,et al.  Author manuscript, published in "Workshop 3D Face Biometrics, IEEE Automatic Facial and Gesture Recognition, Shanghai: China (2013)" Fully Automatic 3D Facial Expression Recognition using Differential Mean Curvature Maps and Histograms of Oriented Gradien , 2013 .

[22]  Nilanjan Sarkar,et al.  Understanding How Adolescents with Autism Respond to Facial Expressions in Virtual Reality Environments , 2013, IEEE Transactions on Visualization and Computer Graphics.

[23]  Liming Chen,et al.  Fully automatic 3D facial expression recognition using a region-based approach , 2011, J-HGBU '11.

[24]  Qiuqi Ruan,et al.  Orthogonal Tensor Neighborhood Preserving Embedding for facial expression recognition , 2011, Pattern Recognit..

[25]  Emmanuel Dellandréa,et al.  Automatic 3D Facial Expression Recognition Based on a Bayesian Belief Net and a Statistical Facial Feature Model , 2010, 2010 20th International Conference on Pattern Recognition.

[26]  S. Berretti,et al.  A Set of Selected SIFT Features for 3D Facial Expression Recognition , 2010, 2010 20th International Conference on Pattern Recognition.

[27]  Xiaoou Tang,et al.  Automatic facial expression recognition on a single 3D face by exploring shape deformation , 2009, ACM Multimedia.

[28]  Jieping Ye,et al.  Tensor Completion for Estimating Missing Values in Visual Data , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[30]  Arman Savran,et al.  Bosphorus Database for 3D Face Analysis , 2008, BIOID.

[31]  Patrick J. Flynn,et al.  A Region Ensemble for 3-D Face Recognition , 2008, IEEE Transactions on Information Forensics and Security.

[32]  Dit-Yan Yeung,et al.  Tensor Embedding Methods , 2006, AAAI.

[33]  Jun Wang,et al.  3D Facial Expression Recognition Based on Primitive Surface Feature Distribution , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[34]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[35]  Deng Cai,et al.  Tensor Subspace Analysis , 2005, NIPS.

[36]  Dong Liang,et al.  A facial expression recognition system based on supervised locally linear embedding , 2005, Pattern Recognit. Lett..

[37]  Shaogang Gong,et al.  Appearance Manifold of Facial Expression , 2005, ICCV-HCI.

[38]  Changbo Hu,et al.  Manifold of facial expression , 2003, 2003 IEEE International SOI Conference. Proceedings (Cat. No.03CH37443).

[39]  Anil K. Jain,et al.  Segmentation and Classification of Range Images , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Xuan Li,et al.  A Survey on Tensor Techniques and Applications in Machine Learning , 2019, IEEE Access.

[41]  Michael Spann,et al.  Surface Normals with Modular Approach and Weighted Voting Scheme in 3D Facial Expression Classification , 2014 .

[42]  Victoria Interrante,et al.  A novel cubic-order algorithm for approximating principal direction vectors , 2004, TOGS.