Improving Smiling Detection with Race and Gender Diversity

Recent progress in deep learning has been accompanied by a growing concern for whether models are fair for users, with equally good performance across different demographics. In computer vision research, such questions are relevant to face detection and the related task of face attribute detection, among others. We measure race and gender inclusion in the context of smiling detection, and introduce a method for improving smiling detection across demographic groups. Our method introduces several modifications over existing detection methods, leveraging twofold transfer learning to better model facial diversity. Results show that this technique improves accuracy against strong baselines for most demographic groups as well as overall. Our best-performing model defines a new state-of-the-art for smiling detection, reaching 91% on the Faces of the World dataset. The accompanying multi-head diversity classifier also defines a new state-of-the-art for gender classification, reaching 93.87% on the Faces of the World dataset. This research demonstrates the utility of modeling race and gender to improve a face attribute detection task, using a twofold transfer learning framework that allows for privacy towards individuals in a target dataset.

[1]  Mohamed R. Amer,et al.  Facial Attributes Classification Using Multi-task Representation Learning , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[2]  G. Sapiro,et al.  Learning to Identify While Failing to Discriminate , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[3]  Trevor Darrell,et al.  PANDA: Pose Aligned Networks for Deep Attribute Modeling , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Nathan Srebro,et al.  Equality of Opportunity in Supervised Learning , 2016, NIPS.

[5]  Jian Sun,et al.  Joint Cascade Face Detection and Alignment , 2014, ECCV.

[6]  P. Matthew Bronstad,et al.  Skin and Bones: The Contribution of Skin Tone and Facial Structure to Racial Prototypicality Ratings , 2012, PloS one.

[7]  Gang Hua,et al.  Ordinal Regression with Multiple Output CNN for Age Estimation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Mahadev Satyanarayanan,et al.  OpenFace: A general-purpose face recognition library with mobile applications , 2016 .

[9]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[10]  William T. Freeman,et al.  Face Synthesis from Facial Identity Features , 2017, ArXiv.

[11]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[12]  Ira Kemelmacher-Shlizerman,et al.  The MegaFace Benchmark: 1 Million Faces for Recognition at Scale , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Ashwin Machanavajjhala,et al.  Protecting Visual Secrets Using Adversarial Nets , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[14]  Lijun Yin,et al.  Towards race-related face identification: research on skin color transfer , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[15]  S. Md. Mansoor Roomi,et al.  Race Classification Based on Facial Features , 2011, 2011 Third National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics.

[16]  Olivier Pascalis,et al.  Facial Contrast Is a Cross-Cultural Cue for Perceiving Age , 2017, Front. Psychol..

[17]  Olivier Pascalis,et al.  Minimizing Skin Color Differences Does Not Eliminate the Own-Race Recognition Advantage in Infants. , 2011, Infancy : the official journal of the International Society on Infant Studies.

[18]  V Bruce,et al.  Perceiving the sex and race of faces: the role of shape and colour , 1995, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[19]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[20]  Luc Van Gool,et al.  Structured Output SVM Prediction of Apparent Age, Gender and Smile from Deep Features , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[21]  Elli Angelopoulou,et al.  The Recognition of Ethnic Groups based on Histological Skin Properties , 2011, VMV.

[22]  Haibo He,et al.  Learning Race from Face: A Survey , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Umesh Bindal,et al.  Labial Impressions: A Tool for Identification , 2014 .

[24]  Toniann Pitassi,et al.  Fairness through awareness , 2011, ITCS '12.

[25]  Zhe Zhao,et al.  Data Decisions and Theoretical Implications when Adversarially Learning Fair Representations , 2017, ArXiv.

[26]  Sergio Escalera,et al.  ChaLearn Looking at People and Faces of the World: Face AnalysisWorkshop and Challenge 2016 , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[27]  Sergio Escalera,et al.  ChaLearn looking at people: A review of events and resources , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[28]  Shengcai Liao,et al.  Learning Face Representation from Scratch , 2014, ArXiv.

[29]  Atsuto Maki,et al.  A systematic study of the class imbalance problem in convolutional neural networks , 2017, Neural Networks.

[30]  Hanqing Lu,et al.  DeepBE: Learning Deep Binary Encoding for Multi-label Classification , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[31]  Carlos D. Castillo,et al.  Triplet probabilistic embedding for face verification and clustering , 2016, 2016 IEEE 8th International Conference on Biometrics Theory, Applications and Systems (BTAS).

[32]  Jing Wang,et al.  Walk and Learn: Facial Attribute Representation Learning from Egocentric Video and Contextual Data , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Ira Kemelmacher-Shlizerman,et al.  Level Playing Field for Million Scale Face Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Rama Chellappa,et al.  HyperFace: A Deep Multi-Task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Kevin R Brooks,et al.  No Role for Lightness in the Perception of Black and White? Simultaneous Contrast Affects Perceived Skin Tone, but Not Perceived Race , 2010, Perception.

[36]  Alexei A. Efros,et al.  Unbiased look at dataset bias , 2011, CVPR 2011.

[37]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[38]  Yu Qiao,et al.  Gender and Smile Classification Using Deep Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[39]  Carlos D. Castillo,et al.  An All-In-One Convolutional Neural Network for Face Analysis , 2016, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[40]  Luc Van Gool,et al.  Deep Expectation of Real and Apparent Age from a Single Image Without Facial Landmarks , 2016, International Journal of Computer Vision.

[41]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Marios Savvides,et al.  A robust approach to facial ethnicity classification on large scale face databases , 2012, 2012 IEEE Fifth International Conference on Biometrics: Theory, Applications and Systems (BTAS).

[43]  Jonathan Krause,et al.  Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States , 2017, Proceedings of the National Academy of Sciences.