Fused Geometry Augmented Images For Analyzing Textured Mesh

In this paper, we propose a multi-modal mesh surface representation by fusing texture and geometric data. Our approach defines an inverse mapping between different geometric descriptors computed on the mesh surface, and the corresponding 2D texture image of the mesh, allowing the construction of fused geometrically augmented images. This new fused modality enables us to learn feature representations from 3D data in a highly efficient manner by employing standard convolutional neural networks in a transfer-learning mode. In contrast to existing methods, the proposed approach is both computationally and memory efficient, preserves intrinsic geometric information and learns highly discriminative feature representations by effectively fusing shape and texture information at the data level. The efficacy is demonstrated on the task of facial expression classification, showing competitive performance with state-of-the-art methods.

[1]  Dimitrios Hatzinakos,et al.  Emotion Recognition from 2D Facial Expressions , 2019, 2019 IEEE Canadian Conference of Electrical and Computer Engineering (CCECE).

[2]  Stefanos Zafeiriou,et al.  Local normal binary patterns for 3D facial action unit detection , 2012, 2012 19th IEEE International Conference on Image Processing.

[3]  Stefano Berretti,et al.  Boosting 3D LBP-based face recognition by fusing shape and texture descriptors on the mesh , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[4]  Martin D. Levine,et al.  Fully automated recognition of spontaneous facial expressions in videos using random forest classifiers , 2014, IEEE Transactions on Affective Computing.

[5]  Xi Zhao,et al.  An efficient multimodal 2D + 3D feature-based approach to automatic facial expression recognition , 2015, Comput. Vis. Image Underst..

[6]  Lijun Yin,et al.  A high-resolution 3D dynamic facial expression database , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[7]  Alberto Del Bimbo,et al.  The Mesh-LBP: A Framework for Extracting Local Binary Patterns From Discrete Manifolds , 2015, IEEE Transactions on Image Processing.

[8]  Robert B. Fisher,et al.  A Color and Texture Based Hierarchical K-NN Approach to the Classification of Non-melanoma Skin Lesions , 2013 .

[9]  Ville Ojansivu,et al.  Blur Insensitive Texture Classification Using Local Phase Quantization , 2008, ICISP.

[10]  Naoufel Werghi,et al.  Segmentation and Modeling of Full Human Body Shape From 3-D Scan Data: A Survey , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[11]  Stefan Carlsson,et al.  CNN Features Off-the-Shelf: An Astounding Baseline for Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[12]  Naoufel Werghi,et al.  Recognition of human body posture from a cloud of 3D data points using wavelet transform coefficients , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[13]  Josef Kittler,et al.  Conformal Mapping of a 3D Face Representation onto a 2D Image for CNN Based Face Recognition , 2018, 2018 International Conference on Biometrics (ICB).

[14]  Jian Sun,et al.  Multimodal 2D+3D Facial Expression Recognition With Deep Fusion Convolutional Neural Network , 2017, IEEE Transactions on Multimedia.

[15]  Tieniu Tan,et al.  Combining Statistics of Geometrical and Correlative Features for 3D Face Recognition , 2006, BMVC.

[16]  J. Paul Siebert,et al.  A functional-based segmentation of human body scans in arbitrary postures , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[17]  Stefanos Zafeiriou,et al.  Binary Pattern Analysis for 3D Facial Action Unit Detection , 2012, BMVC.

[18]  Arman Savran,et al.  Bosphorus Database for 3D Face Analysis , 2008, BIOID.

[19]  Alberto Del Bimbo,et al.  Representing 3D texture on mesh manifolds for retrieval and recognition applications , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  David Zhang,et al.  Monogenic Binary Pattern (MBP): A Novel Feature Extraction and Representation Model for Face Recognition , 2010, 2010 20th International Conference on Pattern Recognition.

[21]  Xiangyu Zhu,et al.  Face Alignment in Full Pose Range: A 3D Total Solution , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Ghassan Hamarneh,et al.  Deep features to classify skin lesions , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[23]  Liming Chen,et al.  3D facial expression recognition via multiple kernel learning of Multi-Scale Local Normal Patterns , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[24]  Arun Ross,et al.  Information fusion in biometrics , 2003, Pattern Recognit. Lett..

[25]  Liming Chen,et al.  Automatic 3D facial expression recognition using geometric scattering representation , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[26]  Karthik Ramani,et al.  Deep Learning 3D Shape Surfaces Using Geometry Images , 2016, ECCV.

[27]  Wen Gao,et al.  Local Gabor binary pattern histogram sequence (LGBPHS): a novel non-statistical model for face representation and recognition , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[28]  Kevin Bailly,et al.  Investigating Deep Neural Forests for Facial Expression Recognition , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).