Facial Soft Biometrics for Recognition in the Wild: Recent Works, Annotation, and COTS Evaluation

The role of soft biometrics to enhance person recognition systems in unconstrained scenarios has not been extensively studied. Here, we explore the utility of the following modalities: gender, ethnicity, age, glasses, beard, and moustache. We consider two assumptions: 1) manual estimation of soft biometrics and 2) automatic estimation from two commercial off-the-shelf systems (COTS). All experiments are reported using the labeled faces in the wild (LFW) database. First, we study the discrimination capabilities of soft biometrics standalone. Then, experiments are carried out fusing soft biometrics with two state-of-the-art face recognition systems based on deep learning. We observe that soft biometrics is a valuable complement to the face modality in unconstrained scenarios, with relative improvements up to 40%/15% in the verification performance when using manual/automatic soft biometrics estimation. Results are reproducible as we make public our manual annotations and COTS outputs of soft biometrics over LFW, as well as the face recognition scores.

[1]  Arun Ross,et al.  Soft biometrics for surveillance: an overview , 2013 .

[2]  Anil K. Jain,et al.  Soft Biometric Traits for Continuous User Authentication , 2010, IEEE Transactions on Information Forensics and Security.

[3]  Mark S. Nixon,et al.  Soft Biometrics; Human Identification Using Comparative Descriptions , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Shiguang Shan,et al.  Heterogeneous Face Attribute Estimation: A Deep Multi-Task Learning Approach , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Johannes Peltola,et al.  Soft biometrics - combining body weight and fat measurements with fingerprint biometrics , 2006, Pattern Recognit. Lett..

[6]  Anil K. Jain,et al.  Feature Selection: Evaluation, Application, and Small Sample Performance , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Xiaogang Wang,et al.  Deep Learning Face Representation from Predicting 10,000 Classes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Rajeswari Mukesh,et al.  A Biometric Approach for Continuous User Authentication by Fusing Hard and Soft Traits , 2014, Int. J. Netw. Secur..

[9]  Najoua Essoukri Ben Amara,et al.  Contribution to the fusion of soft facial and body biometrics for remote people identification , 2016, 2016 2nd International Conference on Advanced Technologies for Signal and Image Processing (ATSIP).

[10]  Shree K. Nayar,et al.  Attribute and simile classifiers for face verification , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[11]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[12]  Salman Yussof,et al.  Integration of multiple soft biometrics for human identification , 2015, Pattern Recognit. Lett..

[13]  Jean-Luc Dugelay,et al.  Soft biometrics systems: Reliability and asymptotic bounds , 2010, 2010 Fourth IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS).

[14]  Julian Fiérrez,et al.  Multiple classifiers in biometrics. Part 2: Trends and challenges , 2018, Inf. Fusion.

[15]  Julian Fiérrez,et al.  Soft Biometrics and Their Application in Person Recognition at a Distance , 2014, IEEE Transactions on Information Forensics and Security.

[16]  Thomas B. Moeslund,et al.  On soft biometrics , 2015, Pattern Recognit. Lett..

[17]  N. Hamdy,et al.  Soft and hard biometrics fusion for improved identity verification , 2004, The 2004 47th Midwest Symposium on Circuits and Systems, 2004. MWSCAS '04..

[18]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[19]  Jonathon S. Hare,et al.  Human face identification via comparative soft biometrics , 2016, 2016 IEEE International Conference on Identity, Security and Behavior Analysis (ISBA).

[20]  Julian Fiérrez,et al.  Quality Measures in Biometric Systems , 2012, IEEE Security & Privacy.

[21]  Jean-Luc Dugelay,et al.  Bag of soft biometrics for person identification , 2010, Multimedia Tools and Applications.

[22]  Anil K. Jain,et al.  Face Matching and Retrieval Using Soft Biometrics , 2010, IEEE Transactions on Information Forensics and Security.

[23]  Anil K. Jain,et al.  Grouper: Optimizing Crowdsourced Face Annotations , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[24]  Bruce A. Draper,et al.  On the effectiveness of soft biometrics for increasing face verification rates , 2015, Comput. Vis. Image Underst..

[25]  Xiaoming Liu,et al.  Demographic Estimation from Face Images: Human vs. Machine Performance , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Shree K. Nayar,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence Describable Visual Attributes for Face Verification and Image Search , 2022 .

[27]  Anil K. Jain,et al.  A Case Study on Unconstrained Facial Recognition Using the Boston Marathon Bombings Suspects , 2013 .

[28]  Gang Hua,et al.  Labeled Faces in the Wild: A Survey , 2016 .

[29]  Julian Fierrez,et al.  Facial soft biometric features for forensic face recognition. , 2015, Forensic science international.

[30]  Arun Ross,et al.  What Else Does Your Biometric Data Reveal? A Survey on Soft Biometrics , 2016, IEEE Transactions on Information Forensics and Security.

[31]  Bruce A. Draper,et al.  A Statistical Assessment of Subject Factors in the PCA Recognition of Human Faces , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[32]  Michael C. Fairhurst,et al.  Enhancing Identity Prediction Using a Novel Approach to Combining Hard- and Soft-Biometric Information , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[33]  Terrance E. Boult,et al.  MOON: A Mixed Objective Optimization Network for the Recognition of Facial Attributes , 2016, ECCV.

[34]  Arun Ross,et al.  50 years of biometric research: Accomplishments, challenges, and opportunities , 2016, Pattern Recognit. Lett..

[35]  Abdenour Hadid,et al.  Biometrics: In Search of Identity and Security (Q & A) , 2018, IEEE MultiMedia.

[36]  Anil K. Jain,et al.  Integrating Faces, Fingerprints, and Soft Biometric Traits for User Recognition , 2004, ECCV Workshop BioAW.

[37]  Sébastien Marcel,et al.  Biometrics: In Search of Identity and Security (Q & A) , 2018 .

[38]  Tieniu Tan,et al.  Demographic Analysis from Biometric Data: Achievements, Challenges, and New Frontiers , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Anil K. Jain,et al.  Suspect identification based on descriptive facial attributes , 2014, IEEE International Joint Conference on Biometrics.

[40]  Rama Chellappa,et al.  Convolutional neural networks for attribute-based active authentication on mobile devices , 2016, 2016 IEEE 8th International Conference on Biometrics Theory, Applications and Systems (BTAS).