Facial Expressions as a Vulnerability in Face Recognition

This work explores facial expression bias as a security vulnerability of face recognition systems. Face recognition technology has experienced great advances during the last decades. However, despite the great performance achieved by state of the art face recognition systems, the algorithms are still sensitive to a large range of covariates. This work presents a comprehensive analysis of how facial expression bias impacts the performance of face recognition technologies. Our study analyzes: i) facial expression biases in the most popular face recognition databases; and ii) the impact of facial expression in face recognition performances. Our experimental framework includes four face detectors, three face recognition models, and four different databases. Our results demonstrate a huge facial expression bias in the most widely used databases, as well as a related impact of face expression in the performance of state-of-the-art algorithms. This work opens the door to new research lines focused on mitigating the observed vulnerability.

[1]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[2]  Patrick J. Flynn,et al.  An evaluation of multimodal 2D+3D face biometrics , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Patrick J. Flynn,et al.  A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition , 2006, Comput. Vis. Image Underst..

[4]  Julian Fierrez,et al.  Vulnerabilities in biometric systems: Attacks and recent advances in liveness detection , 2007 .

[5]  J. Cohn,et al.  Impact of depression on response to comedy: a dynamic facial coding analysis. , 2007, Journal of abnormal psychology.

[6]  Thomas Hadjistavropoulos,et al.  A psychophysical investigation of the facial action coding system as an index of pain variability among older adults with and without Alzheimer's disease. , 2007, Pain medicine.

[7]  Xuan Zou,et al.  Illumination Invariant Face Recognition: A Survey , 2007, 2007 First IEEE International Conference on Biometrics: Theory, Applications, and Systems.

[8]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[9]  Andrew Buchanan,et al.  Facial Expressions for Empathic Communication of Emotion in Animated Characters , 2009 .

[10]  Sergi Villagrasa,et al.  FACe! 3D Facial Animation System based on FACS , 2009 .

[11]  Sébastien Marcel,et al.  On the vulnerability of face verification systems to hill-climbing attacks , 2010, Pattern Recognit..

[12]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[13]  Alexei A. Efros,et al.  Unbiased look at dataset bias , 2011, CVPR 2011.

[14]  Tal Hassner,et al.  Face recognition in unconstrained videos with matched background similarity , 2011, CVPR 2011.

[15]  Anil K. Jain,et al.  Face Recognition Performance: Role of Demographic Information , 2012, IEEE Transactions on Information Forensics and Security.

[16]  Mohammad H. Mahoor,et al.  DISFA: A Spontaneous Facial Action Intensity Database , 2013, IEEE Transactions on Affective Computing.

[17]  Julian Fierrez,et al.  Image Quality Assessment for Fake Biometric Detection: Application to Iris, Fingerprint, and Face Recognition , 2014, IEEE Transactions on Image Processing.

[18]  Yong Tao,et al.  Compound facial expressions of emotion , 2014, Proceedings of the National Academy of Sciences.

[19]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Shengcai Liao,et al.  Learning Face Representation from Scratch , 2014, ArXiv.

[21]  Sébastien Marcel,et al.  Biometric Antispoofing Methods: A Survey in Face Recognition , 2014, IEEE Access.

[22]  Julian Fierrez,et al.  Facial soft biometric features for forensic face recognition. , 2015, Forensic science international.

[23]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[24]  Katherine B. Martin,et al.  Facial Action Coding System , 2015 .

[25]  Abdenour Hadid,et al.  Biometrics Systems Under Spoofing Attack: An evaluation methodology and lessons learned , 2015, IEEE Signal Processing Magazine.

[26]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[27]  Xiangyu Zhu,et al.  High-fidelity Pose and Expression Normalization for face recognition in the wild , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Carlos D. Castillo,et al.  Frontal to profile face verification in the wild , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[29]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Yuxiao Hu,et al.  MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition , 2016, ECCV.

[31]  Dacheng Tao,et al.  A Comprehensive Survey on Pose-Invariant Face Recognition , 2015, ACM Trans. Intell. Syst. Technol..

[32]  Yu Qiao,et al.  Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks , 2016, IEEE Signal Processing Letters.

[33]  Ira Kemelmacher-Shlizerman,et al.  The MegaFace Benchmark: 1 Million Faces for Recognition at Scale , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Aleix M. Martínez,et al.  EmotioNet: An Accurate, Real-Time Algorithm for the Automatic Annotation of a Million Facial Expressions in the Wild , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Louis-Philippe Morency,et al.  Computational study of psychosis symptoms and facial expressions , 2016 .

[36]  Harry Wechsler,et al.  Face Verification Subject to Varying (Age, Ethnicity, and Gender)Demographics Using Deep Learning , 2016 .

[37]  Kiran B. Raja,et al.  On the vulnerability of face recognition systems towards morphed face attacks , 2017, 2017 5th International Workshop on Biometrics and Forensics (IWBF).

[38]  Junmo Kim,et al.  Deep Pyramidal Residual Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Stefanos Zafeiriou,et al.  AgeDB: The First Manually Collected, In-the-Wild Age Database , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[40]  Julian Fierrez,et al.  Facial Soft Biometrics for Recognition in the Wild: Recent Works, Annotation, and COTS Evaluation , 2018, IEEE Transactions on Information Forensics and Security.

[41]  Abdenour Hadid,et al.  Face recognition under spoofing attacks: countermeasures and research directions , 2018, IET Biom..

[42]  Shan Li,et al.  Deep Facial Expression Recognition: A Survey , 2018, IEEE Transactions on Affective Computing.

[43]  Louis-Philippe Morency,et al.  OpenFace 2.0: Facial Behavior Analysis Toolkit , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[44]  Omkar M. Parkhi,et al.  VGGFace2: A Dataset for Recognising Faces across Pose and Age , 2017, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[45]  Yong Dou,et al.  A Community Detection Approach to Cleaning Extremely Large Face Database , 2018, Comput. Intell. Neurosci..

[46]  Anil K. Jain,et al.  IARPA Janus Benchmark - C: Face Dataset and Protocol , 2018, 2018 International Conference on Biometrics (ICB).

[47]  Klemen Grm,et al.  Strengths and weaknesses of deep learning models for face recognition against image degradations , 2017, IET Biom..

[48]  Carlos D. Castillo,et al.  Deep Learning for Understanding Faces: Machines May Be Just as Good, or Better, than Humans , 2018, IEEE Signal Processing Magazine.

[49]  Rama Chellappa,et al.  Recognizing Disguised Faces in the Wild , 2018, IEEE Transactions on Biometrics, Behavior, and Identity Science.

[50]  Mohammad H. Mahoor,et al.  AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild , 2017, IEEE Transactions on Affective Computing.

[51]  P. Ekman,et al.  Facial action coding system , 2019 .

[52]  Daniel McDuff,et al.  A Scalable Approach for Facial Action Unit Classifier Training Using Noisy Data for Pre-Training , 2019, ArXiv.

[53]  Javier Hernandez-Ortega,et al.  Introduction to Face Presentation Attack Detection , 2019, Handbook of Biometric Anti-Spoofing, 2nd Ed..

[54]  Julian Fiérrez,et al.  SensitiveNets: Learning Agnostic Representations with Application to Face Recognition , 2019, ArXiv.

[55]  Carlos D. Castillo,et al.  An Experimental Evaluation of Covariates Effects on Unconstrained Face Verification , 2018, IEEE Transactions on Biometrics, Behavior, and Identity Science.

[56]  Weihong Deng,et al.  Supplementary Material for Unsupervised Face Normalization with Extreme Pose and Expression in the Wild , 2019 .

[57]  Vittorio Cuculo,et al.  OpenFACS: An Open Source FACS-Based 3D Face Animation System , 2019, ICIG.

[58]  Stefanos Zafeiriou,et al.  ArcFace: Additive Angular Margin Loss for Deep Face Recognition , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Richa Singh,et al.  Detecting and Mitigating Adversarial Perturbations for Robust Face Recognition , 2019, International Journal of Computer Vision.

[60]  Tal Hassner,et al.  FSGAN: Subject Agnostic Face Swapping and Reenactment , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[61]  Sébastien Marcel,et al.  Detection of Age-Induced Makeup Attacks on Face Recognition Systems Using Multi-Layer Deep Features , 2020, IEEE Transactions on Biometrics, Behavior, and Identity Science.

[62]  Weihong Deng,et al.  Mitigating Bias in Face Recognition Using Skewness-Aware Reinforcement Learning , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  A. Morales,et al.  DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection , 2020, Inf. Fusion.

[64]  Julian Fierrez,et al.  GANprintR: Improved Fakes and Evaluation of the State of the Art in Face Manipulation Detection , 2019, IEEE Journal of Selected Topics in Signal Processing.

[65]  Kai Zhang,et al.  How Does Gender Balance In Training Data Affect Face Recognition Accuracy? , 2020, 2020 IEEE International Joint Conference on Biometrics (IJCB).

[66]  Ignacio Serna,et al.  Algorithmic Discrimination: Formulation and Exploration in Deep Learning-based Face Biometrics , 2019, SafeAI@AAAI.

[67]  Sixue Gong,et al.  Jointly De-Biasing Face Recognition and Demographic Attribute Estimation , 2019, ECCV.

[68]  Christoph Busch,et al.  Plastic Surgery: An Obstacle for Deep Face Recognition? , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[69]  Irene Kotsia,et al.  RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[70]  Vishal M. Patel,et al.  Quickest Intruder Detection For Multiple User Active Authentication , 2020, 2020 IEEE International Conference on Image Processing (ICIP).

[71]  A. Schacht,et al.  A Comparison of the Affectiva iMotions Facial Expression Analysis Software With EMG for Identifying Facial Expressions of Emotion , 2020, Frontiers in Psychology.

[72]  A. Morales,et al.  A Comprehensive Study on Face Recognition Biases Beyond Demographics , 2021, IEEE Transactions on Technology and Society.

[73]  Ignacio Serna,et al.  SensitiveLoss: Improving Accuracy and Fairness of Face Representations with Discrimination-Aware Deep Learning , 2020, Artif. Intell..