Fast and Light Manifold CNN based 3D Facial Expression Recognition across Pose Variations

This paper proposes a novel approach to 3D Facial Expression Recognition (FER), and it is based on a Fast and Light Manifold CNN model, namely FLM-CNN. Different from current manifold CNNs, FLM-CNN adopts a human vision inspired pooling structure and a multi-scale encoding strategy to enhance geometry representation, which highlights shape characteristics of expressions and runs efficiently. Furthermore, a sampling tree based preprocessing method is presented, and it sharply saves memory when applied to 3D facial surfaces, without much information loss of original data. More importantly, due to the property of manifold CNN features of being rotation-invariant, the proposed method shows a high robustness to pose variations. Extensive experiments are conducted on BU-3DFE, and state-of-the-art results are achieved, indicating its effectiveness.

[1]  Jun Wang,et al.  3D Facial Expression Recognition Based on Primitive Surface Feature Distribution , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[2]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[3]  Patrice Y. Simard,et al.  Best practices for convolutional neural networks applied to visual document analysis , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[4]  Alberto Del Bimbo,et al.  A Set of Selected SIFT Features for 3D Facial Expression Recognition , 2010, 2010 20th International Conference on Pattern Recognition.

[5]  Jonathan Masci,et al.  Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Thomas S. Huang,et al.  3D facial expression recognition based on automatically selected features , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[7]  Gernot Riegler,et al.  OctNet: Learning Deep 3D Representations at High Resolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Subramanian Ramanathan,et al.  Human Facial Expression Recognition using a 3D Morphable Model , 2006, 2006 International Conference on Image Processing.

[9]  Qiang Chen,et al.  Network In Network , 2013, ICLR.

[10]  Abd El Rahman Shabayek,et al.  Facial Expression Recognition via Joint Deep Learning of RGB-Depth Map Latent Representations , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[11]  Liming Chen,et al.  Muscular Movement Model Based Automatic 3D Facial Expression Recognition , 2015, MMM.

[12]  Michael G. Strintzis,et al.  Bilinear Models for 3-D Face and Facial Expression Recognition , 2008, IEEE Transactions on Information Forensics and Security.

[13]  Vincent Lepetit,et al.  DAISY: An Efficient Dense Descriptor Applied to Wide-Baseline Stereo , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Emmanuel Dellandréa,et al.  Automatic 3D Facial Expression Recognition Based on a Bayesian Belief Net and a Statistical Facial Feature Model , 2010, 2010 20th International Conference on Pattern Recognition.

[15]  Iasonas Kokkinos,et al.  Intrinsic shape context descriptors for deformable shapes , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[17]  Liming Chen,et al.  3D facial expression recognition via multiple kernel learning of Multi-Scale Local Normal Patterns , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[18]  Fernando Fernández Martínez,et al.  Towards a robust affect recognition: Automatic facial expression recognition in 3D faces , 2015, Expert Syst. Appl..

[19]  Liming Chen,et al.  Author manuscript, published in "Workshop 3D Face Biometrics, IEEE Automatic Facial and Gesture Recognition, Shanghai: China (2013)" Fully Automatic 3D Facial Expression Recognition using Differential Mean Curvature Maps and Histograms of Oriented Gradien , 2013 .

[20]  Liming Chen,et al.  Automatic 3D facial expression recognition using geometric scattering representation , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[21]  Sepp Hochreiter,et al.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , 2015, ICLR.

[22]  Lijun Yin,et al.  CNN based 3D facial expression recognition using masking and landmark features , 2017, 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII).

[23]  Sebastian Scherer,et al.  VoxNet: A 3D Convolutional Neural Network for real-time object recognition , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[24]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Hasan Demirel,et al.  Facial Expression Recognition Using 3D Facial Feature Distances , 2007, ICIAR.

[26]  Jian-Jun Zhang,et al.  Fast and exact discrete geodesic computation based on triangle-oriented wavefront propagation , 2016, ACM Trans. Graph..

[27]  Subhransu Maji,et al.  Multi-view Convolutional Neural Networks for 3D Shape Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[28]  Federico Sukno,et al.  Local Shape Spectrum Analysis for 3D Facial Expression Recognition , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[29]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[30]  Andrea J. van Doorn,et al.  Surface shape and curvature scales , 1992, Image Vis. Comput..

[31]  Wei Zeng,et al.  An automatic 3D expression recognition framework based on sparse representation of conformal images , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[32]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[33]  Liming Chen,et al.  Fully automatic 3D facial expression recognition using a region-based approach , 2011, J-HGBU '11.

[34]  Jianxiong Xiao,et al.  3D ShapeNets: A deep representation for volumetric shapes , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Xiaoou Tang,et al.  Automatic facial expression recognition on a single 3D face by exploring shape deformation , 2009, ACM Multimedia.

[36]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Pierre Vandergheynst,et al.  Geodesic Convolutional Neural Networks on Riemannian Manifolds , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[38]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[39]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[40]  Xi Zhao,et al.  An efficient multimodal 2D + 3D feature-based approach to automatic facial expression recognition , 2015, Comput. Vis. Image Underst..

[41]  Yang Liu,et al.  O-CNN , 2017, ACM Trans. Graph..

[42]  Jian Sun,et al.  Multimodal 2D+3D Facial Expression Recognition With Deep Fusion Convolutional Neural Network , 2017, IEEE Transactions on Multimedia.

[43]  Liming Chen,et al.  Deep Representation of Facial Geometric and Photometric Attributes for Automatic 3D Facial Expression Recognition , 2015, ArXiv.

[44]  Liming Chen,et al.  HSOG: A Novel Local Image Descriptor Based on Histograms of the Second-Order Gradients , 2014, IEEE Transactions on Image Processing.

[45]  Stefano Berretti,et al.  Shape analysis of local facial patches for 3D facial expression recognition , 2011, Pattern Recognit..

[46]  C. Qi.  Deep Learning on Point Sets for 3 D Classification and Segmentation , 2016 .

[47]  Jonathan Masci,et al.  Learning shape correspondence with anisotropic convolutional neural networks , 2016, NIPS.