Exploring Large-scale Unlabeled Faces to Enhance Facial Expression Recognition

Facial Expression Recognition (FER) is an important task in computer vision and has wide applications in human-computer interaction, intelligent security, emotion analysis, and other fields. However, the limited size of FER datasets limits the generalization ability of expression recognition models, resulting in ineffective model performance. To address this problem, we propose a semi-supervised learning framework that utilizes unlabeled face data to train expression recognition models effectively. Our method uses a dynamic threshold module (\textbf{DTM}) that can adaptively adjust the confidence threshold to fully utilize the face recognition (FR) data to generate pseudo-labels, thus improving the model's ability to model facial expressions. In the ABAW5 EXPR task, our method achieved excellent results on the official validation set.

[1]  Alan S. Cowen,et al.  ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[2]  Tao Wang,et al.  Distract Your Attention: Multi-Head Cross Attention Network for Facial Expression Recognition , 2021, Biomimetics.

[3]  Weihong Deng,et al.  Learn From All: Erasing Attention Consistency for Noisy Label Facial Expression Recognition , 2022, ECCV.

[4]  D. Kollias ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges , 2022, ECCV Workshops.

[5]  A. Savchenko Video-based Frame-level Facial Analysis of Affective Behavior on Mobile Devices using EfficientNets , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6]  Jae-Yeop Jeong,et al.  Classification of Facial Expression In-the-Wild based on Ensemble of Multi-head Cross Attention Networks , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[7]  Fei Wang,et al.  Face2Exp: Combating Data Biases for Facial Expression Recognition , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Y. Zhu,et al.  Coarse-to-Fine Cascaded Networks with Smooth Predicting for Video Facial Expression Recognition , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[9]  Xinbo Gao,et al.  Towards Semi-Supervised Deep Facial Expression Recognition with An Adaptive Confidence Margin , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  D. Kollias ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[11]  Fangzhao Wu,et al.  Rethinking InfoNCE: How Many Negative Samples Do You Need? , 2021, IJCAI.

[12]  Gim Hee Lee,et al.  Teaching with Soft Label Smoothing for Mitigating Noisy Labels in Facial Expressions , 2022, European Conference on Computer Vision.

[13]  T. Shinozaki,et al.  FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling , 2021, NeurIPS.

[14]  Guodong Guo,et al.  TransFER: Learning Relation-aware Facial Expression Representations with Transformers , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[15]  Jungseock Joo,et al.  Understanding and Mitigating Annotation Bias in Facial Expression Recognition , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[16]  Yong-Sheng Chen,et al.  Learning Facial Representations from the Cycle-consistency of Face , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[17]  Stefanos Zafeiriou,et al.  Analysing Affective Behavior in the second ABAW2 Competition , 2021, 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[18]  Stefanos Zafeiriou,et al.  Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study , 2021, ArXiv.

[19]  Tao Mei,et al.  Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Stefanos Zafeiriou,et al.  Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework , 2021, ArXiv.

[21]  Debing Zhang,et al.  Partial FC: Training 10 Million Identities on a Single Machine , 2020, 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[22]  S. Zafeiriou,et al.  Analysing Affective Behavior in the First ABAW 2020 Competition , 2020, 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020).

[23]  David Berthelot,et al.  FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence , 2020, NeurIPS.

[24]  Yaoshiang Ho,et al.  The Real-World-Weight Cross-Entropy Loss Function: Modeling the Costs of Mislabeling , 2020, IEEE Access.

[25]  Quoc V. Le,et al.  Self-Training With Noisy Student Improves ImageNet Classification , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Quoc V. Le,et al.  Randaugment: Practical automated data augmentation with a reduced search space , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[27]  Dimitrios Kollias,et al.  Face Behavior à la carte: Expressions, Affect and Action Units in a Single Network , 2019, ArXiv.

[28]  Dimitrios Kollias,et al.  Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace , 2019, BMVC.

[29]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.

[30]  Guoying Zhao,et al.  Deep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond , 2018, International Journal of Computer Vision.

[31]  Stefanos Zafeiriou,et al.  ArcFace: Additive Angular Margin Loss for Deep Face Recognition , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Mohammad H. Mahoor,et al.  AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild , 2017, IEEE Transactions on Affective Computing.

[33]  Stefanos Zafeiriou,et al.  Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition , 2018, ArXiv.

[34]  Shiguang Shan,et al.  Facial Expression Recognition with Inconsistently Annotated Datasets , 2018, ECCV.

[35]  Mert R. Sabuncu,et al.  Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels , 2018, NeurIPS.

[36]  Guoying Zhao,et al.  Aff-Wild: Valence and Arousal ‘In-the-Wild’ Challenge , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[37]  Junping Du,et al.  Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Xiaoou Tang,et al.  From Facial Expression Recognition to Interpersonal Relation Prediction , 2016, International Journal of Computer Vision.

[39]  Yuxiao Hu,et al.  MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition , 2016, ECCV.

[40]  Tolga Tasdizen,et al.  Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning , 2016, NIPS.

[41]  Cha Zhang,et al.  Image based Static Facial Expression Recognition with Multiple Deep Network Learning , 2015, ICMI.

[42]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[43]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[44]  Philip Bachman,et al.  Learning with Pseudo-Ensembles , 2014, NIPS.

[45]  Dong-Hyun Lee,et al.  Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks , 2013 .

[46]  Samarjit Kar,et al.  Cross-entropy measure of uncertain variables , 2012, Inf. Sci..

[47]  Guoyin Wang,et al.  Expression Recognition Methods Based on Feature Fusion , 2010, Brain Informatics.

[48]  Maja Pantic,et al.  Facial Expression Recognition , 2009, Encyclopedia of Biometrics.

[49]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[50]  Allen R. Hanson,et al.  Feature Selection Using Adaboost for Face Expression Recognition , 2005 .

[51]  Bir Bhanu,et al.  Feature Synthesis Using Genetic Programming for Face Expression Recognition , 2004, GECCO.

[52]  J. Russell,et al.  An approach to environmental psychology , 1974 .