A Comprehensive Database for Benchmarking Imaging Systems

Cross-modality face recognition is an emerging topic due to the wide-spread usage of different sensors in day-to-day life applications. The development of face recognition systems relies greatly on existing databases for evaluation and obtaining training examples for data-hungry machine learning algorithms. However, currently, there is no publicly available face database that includes more than two modalities for the same subject. In this work, we introduce the Tufts Face Database that includes images acquired in various modalities: photograph images, thermal images, near infrared images, a recorded video, a computerized facial sketch, and 3D images of each volunteer's face. An Institutional Research Board protocol was obtained and images were collected from students, staff, faculty, and their family members at Tufts University. The database includes over 10,000 images from 113 individuals from more than 15 different countries, various gender identities, ages, and ethnic backgrounds. The contributions of this work are: 1) Detailed description of the content and acquisition procedure for images in the Tufts Face Database; 2) The Tufts Face Database is publicly available to researchers worldwide, which will allow assessment and creation of more robust, consistent, and adaptable recognition algorithms; 3) A comprehensive, up-to-date review on face recognition systems and face datasets.

[1]  Reuben A. Farrugia,et al.  Fusion of Intra- and Inter-modality Algorithms for Face-Sketch Recognition , 2015, CAIP.

[2]  Arman Savran,et al.  Computer Vision and Image Understanding , 2022, SSRN Electronic Journal.

[3]  Jiri Matas,et al.  XM2VTSDB: The Extended M2VTS Database , 1999 .

[4]  Richa Singh,et al.  Composite sketch recognition via deep network - a transfer learning approach , 2015, 2015 International Conference on Biometrics (ICB).

[5]  Anil K. Jain,et al.  Heterogeneous Face Recognition Using Kernel Prototype Similarities , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Jun Guo,et al.  Face Recognition via Collaborative Representation: Its Discriminant Nature and Superposed Representation , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Richa Singh,et al.  Disguise detection and face recognition in visible and thermal spectrums , 2013, 2013 International Conference on Biometrics (ICB).

[8]  Mehryar Emambakhsh,et al.  Nasal Patches and Curves for Expression-Robust 3D Face Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Gang Hua,et al.  Labeled Faces in the Wild: A Survey , 2016 .

[10]  Javier Ruiz-del-Solar,et al.  A comparative study of thermal face recognition methods in unconstrained environments , 2012, Pattern Recognit..

[11]  Jean-Luc Dugelay,et al.  KinectFaceDB: A Kinect Database for Face Recognition , 2014, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[12]  Ioannis A. Kakadiaris,et al.  Face Recognition in the Thermal Infrared Spectrum , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[13]  Xuelong Li,et al.  Transductive Face Sketch-Photo Synthesis , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[14]  Reuben A. Farrugia,et al.  Face photo-sketch recognition using local and global texture descriptors , 2016, 2016 24th European Signal Processing Conference (EUSIPCO).

[15]  Patrick J. Flynn,et al.  A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition , 2006, Comput. Vis. Image Underst..

[16]  Xavier Maldague,et al.  Infrared face recognition: A comprehensive review of methodologies and databases , 2014, Pattern Recognit..

[17]  LinLin Shen,et al.  Directional binary code with application to PolyU near-infrared face database , 2010, Pattern Recognit. Lett..

[18]  Xinbo Gao,et al.  Graphical Representation for Heterogeneous Face Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  D. Jacobs,et al.  Bypassing synthesis: PLS for face recognition with pose, low-resolution and sketch , 2011, CVPR 2011.

[20]  Amit R.Sharma,et al.  Face Photo-Sketch Synthesis and Recognition , 2012 .

[21]  Mohammad Mahdi Dehshibi,et al.  Iranian Face Database with age, pose and expression , 2007, 2007 International Conference on Machine Vision.

[22]  Fei Chen,et al.  A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference , 2010, IEEE Transactions on Multimedia.

[23]  Matti Pietikäinen,et al.  Face Description with Local Binary Patterns: Application to Face Recognition , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Shengcai Liao,et al.  Illumination Invariant Face Recognition Using Near-Infrared Images , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[26]  Jean-Marie Morvan,et al.  Improving Shadow Suppression for Illumination Robust Face Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Meng Joo Er,et al.  Illumination compensation and normalization for robust face recognition using discrete cosine transform in logarithm domain , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[28]  Shihong Lao,et al.  3D template matching for pose invariant face recognition using 3D facial model built with isoluminance line based stereo vision , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[29]  K. Taylor.  Forensic Art and Illustration , 2000 .

[30]  Jiwen Lu,et al.  Context-Aware Local Binary Feature Learning for Face Recognition , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Hyeonjoon Moon,et al.  The FERET evaluation methodology for face-recognition algorithms , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[32]  Sergio Escalera,et al.  Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Pietro Perona,et al.  Pruning training sets for learning of object categories , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[34]  Anil K. Jain,et al.  Heterogeneous face recognition , 2012 .

[35]  Shengcai Liao,et al.  Nighttime Face Recognition at Long Distance: Cross-Distance and Cross-Spectral Matching , 2012, ACCV.

[36]  Shengcai Liao,et al.  Heterogeneous Face Recognition from Local Structures of Normalized Appearance , 2009, ICB.

[37]  Shengcai Liao,et al.  The CASIA NIR-VIS 2.0 Face Database , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[38]  Dr. Ashwini Barbadekar,et al.  A Survey of Face Recognition from Sketches , 2016 .

[39]  Reuben A. Farrugia,et al.  Matching Software-Generated Sketches to Face Photographs With a Very Deep CNN, Morphed Faces, and Transfer Learning , 2018, IEEE Transactions on Information Forensics and Security.

[40]  Roy S. Malpass,et al.  Use of facial composite systems in US law enforcement agencies , 2006 .

[41]  Marcos Faúndez-Zanuy,et al.  A Criterion for Analysis of Different Sensor Combinations with an Application to Face Biometrics , 2010, Cognitive Computation.

[42]  Shahan Nercessian.  Human visual system-based multi-scale tools with biomedical and security applications , 2012 .

[43]  V. Kshirsagar,et al.  Face recognition using Eigenfaces , 2011, 2011 3rd International Conference on Computer Research and Development.

[44]  Michael J. Lyons,et al.  Automatic Classification of Single Facial Images , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  R. Gross.  Face Databases , 2005 .

[46]  P. Jonathon Phillips,et al.  Face recognition vendor test 2002 , 2003, 2003 IEEE International SOI Conference. Proceedings (Cat. No.03CH37443).

[47]  Adam Schmidt,et al.  The put face database , 2008 .

[48]  Harry Wechsler,et al.  The FERET database and evaluation procedure for face-recognition algorithms , 1998, Image Vis. Comput..

[49]  Jirí Mekyska,et al.  A New Face Database Simultaneously Acquired in Visible, Near-Infrared and Thermal Spectrums , 2012, Cognitive Computation.

[50]  Dacheng Tao,et al.  Trunk-Branch Ensemble Convolutional Neural Networks for Video-Based Face Recognition , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  Aly A. Farag,et al.  Face recognition in low resolution thermal images , 2013, Comput. Vis. Image Underst..

[52]  Aboul Ella Hassanien,et al.  Optimized superpixel and AdaBoost classifier for human thermal face recognition , 2018, Signal Image Video Process..

[53]  Tieniu Tan,et al.  Wasserstein CNN: Learning Invariant Features for NIR-VIS Face Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[55]  Reuben A. Farrugia,et al.  Forensic Face Photo-Sketch Recognition Using a Deep Learning-Based Architecture , 2017, IEEE Signal Processing Letters.

[56]  Antonio Fernández-Caballero,et al.  Selection of a Visible-Light vs. Thermal Infrared Sensor in Dynamic Environments Based on Confidence Measures , 2014 .

[57]  Anil K. Jain,et al.  Sketch based face recognition: Forensic vs. composite sketches , 2013, 2013 International Conference on Biometrics (ICB).

[58]  Dahua Lin,et al.  Inter-modality Face Recognition , 2006, ECCV.

[59]  Andrea F. Abate,et al.  2D and 3D face recognition: A survey , 2007, Pattern Recognit. Lett..

[60]  Dong Yi,et al.  Face Matching Between Near Infrared and Visible Light Images , 2007, ICB.

[61]  Reuben A. Farrugia,et al.  A Large-Scale Software-Generated Face Composite Sketch Database , 2016, 2016 International Conference of the Biometrics Special Interest Group (BIOSIG).

[62]  Aleix M. Martinez,et al.  The AR face database , 1998 .

[63]  Wilma A. Bainbridge,et al.  The intrinsic memorability of face photographs. , 2013, Journal of experimental psychology. General.

[64]  Xuelong Li,et al.  Multiple Representations-Based Face Sketch–Photo Synthesis , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[65]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[66]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[67]  Monson H. Hayes,et al.  A hidden markov model-based approach for face detection and recognition , 1999 .

[68]  David J. Kriegman,et al.  Acquiring linear subspaces for face recognition under variable lighting , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[69]  Gabriel Hermosilla Vigneau,et al.  Thermal Face Recognition Under Temporal Variation Conditions , 2017, IEEE Access.

[70]  Shaun J. Canavan,et al.  BP4D-Spontaneous: a high-resolution spontaneous 3D dynamic facial expression database , 2014, Image Vis. Comput..

[71]  Sébastien Marcel,et al.  Spoofing in 2D face recognition with 3D masks and anti-spoofing with Kinect , 2013, 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS).

[72]  Shigeyuki Tomita,et al.  Face identification using thermal image processing , 1997, Proceedings 6th IEEE International Workshop on Robot and Human Communication. RO-MAN'97 SENDAI.

[73]  Jonghyun Choi,et al.  Thermal-to-visible face recognition using partial least squares. , 2015, Journal of the Optical Society of America. A, Optics, image science, and vision.

[74]  Azriel Rosenfeld,et al.  Face recognition: A literature survey , 2003, CSUR.

[75]  David J. Kriegman,et al.  From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[76]  Larry S. Davis,et al.  Thermal to visible face recognition , 2012, Defense + Commercial Sensing.

[77]  Bruce J. Tromberg,et al.  Face Recognition in Hyperspectral Images , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[78]  Fan Chen,et al.  A Thermal Facial Emotion Database and Its Analysis , 2013, PSIVT.

[79]  Simone Bianco,et al.  Large Age-Gap face verification by feature injection in deep networks , 2016, Pattern Recognit. Lett..

[80]  Raimondo Schettini,et al.  UMB-DB: A database of partially occluded 3D faces , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[81]  Terence Sim,et al.  The CMU Pose, Illumination, and Expression (PIE) database , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[82]  Yue Wu,et al.  Learning Pose-Aware Models for Pose-Invariant Face Recognition in the Wild , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[83]  José M. Chaves-González,et al.  Detecting skin in face recognition systems: A colour spaces study , 2010, Digit. Signal Process..

[84]  Dacheng Tao,et al.  Robust Face Recognition via Multimodal Deep Face Representation , 2015, IEEE Transactions on Multimedia.

[85]  Mita Nasipuri,et al.  Thermal Infrared Face Recognition – A Biometric Identification Technique for Robust Security system , 2011 .

[86]  Patrick J. Flynn,et al.  Face Recognition Using 2D and 3D Facial Data , 2003 .

[87]  Alan C. Bovik,et al.  Texas 3D Face Recognition Database , 2010, 2010 IEEE Southwest Symposium on Image Analysis & Interpretation (SSIAI).

[88]  Anil K. Jain,et al.  A longitudinal study of automatic face recognition , 2015, 2015 International Conference on Biometrics (ICB).

[89]  Himanshu S. Bhatt,et al.  Plastic Surgery: A New Dimension to Face Recognition , 2010, IEEE Transactions on Information Forensics and Security.

[90]  Stan Z. Li,et al.  Coupled Spectral Regression for matching heterogeneous faces , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[91]  Richa Singh,et al.  Recognizing composite sketches with digital face images via SSD dictionary , 2014, IEEE International Joint Conference on Biometrics.

[92]  Usman Ali,et al.  Optimized Visual and Thermal Image Fusion for Efficient Face Recognition , 2006, 2006 9th International Conference on Information Fusion.

[93]  Anil K. Jain,et al.  Heterogeneous Face Recognition: Matching NIR to Visible Light Images , 2010, 2010 20th International Conference on Pattern Recognition.

[94]  Timothy Hospedales,et al.  A survey on heterogeneous face recognition: Sketch, infra-red, 3D and low-resolution , 2014, Image Vis. Comput..

[95]  Jiwen Lu,et al.  Coupled Discriminative Feature Learning for Heterogeneous Face Recognition , 2015, IEEE Transactions on Information Forensics and Security.

[96]  Tal Hassner,et al.  Face recognition in unconstrained videos with matched background similarity , 2011, CVPR 2011.

[97]  Hamed Kiani Galoogahi,et al.  Face sketch recognition by Local Radon Binary Pattern: LRBP , 2012, 2012 19th IEEE International Conference on Image Processing.

[98]  Paulo Novais,et al.  Selection of a visible-light vs. Thermal infrared sensor in dynamic environments based on confidence measures , 2014 .

[99]  Jiwen Lu,et al.  Simultaneous Local Binary Feature Learning and Encoding for Homogeneous and Heterogeneous Face Recognition , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[100]  Matti Pietikäinen,et al.  Learning mappings for face synthesis from near infrared to visual light images , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[101]  C. V. Jawahar,et al.  Indian Movie Face Database: A benchmark for face recognition under wide variations , 2013, 2013 Fourth National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics (NCVPRIPG).

[102]  Frédéric Jurie,et al.  CMML: a New Metric Learning Approach for Cross Modal Matching , 2012, ACCV 2012.

[103]  Mohammed Bennamoun,et al.  An efficient 3D face recognition approach using local geometrical signatures , 2014, Pattern Recognit..

[104]  Q. Ruan,et al.  Matching NIR face to VIS face using multi-feature based MSDA , 2014, 2014 12th International Conference on Signal Processing (ICSP).

[105]  Lijun Yin,et al.  A high-resolution 3D dynamic facial expression database , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[106]  Anil K. Jain,et al.  The FaceSketchID System: Matching Facial Composites to Mugshots , 2014, IEEE Transactions on Information Forensics and Security.

[107]  Xiaogang Wang,et al.  Face sketch synthesis and recognition , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[108]  Sami Romdhani,et al.  A 3D Face Model for Pose and Illumination Invariant Face Recognition , 2009, 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.

[109]  Dimitrios Hatzinakos,et al.  Iterative Closest Normal Point for 3D Face Recognition , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[110]  Chunna Tian,et al.  Face Sketch Synthesis using E-HMM and Selective Ensemble , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[111]  Gérard G. Medioni,et al.  Face modeling and recognition in 3-D , 2003, 2003 IEEE International SOI Conference. Proceedings (Cat. No.03CH37443).

[112]  Hanqing Lu,et al.  A nonlinear approach for face sketch synthesis and recognition , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[113]  Anil K. Jain,et al.  Matching Composite Sketches to Face Photos: A Component-Based Approach , 2013, IEEE Transactions on Information Forensics and Security.

[114]  Wen Gao,et al.  The CAS-PEAL Large-Scale Chinese Face Database and Baseline Evaluations , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[115]  Ioannis T. Pavlidis,et al.  Face Detection in the Near-IR Spectrum , 2003, Image Vis. Comput..

[116]  P J. Phillips,et al.  Face Recognition Vendor Test 2000: Evaluation Report , 2001 .

[117]  Asem M. Ali.  A 3D-Based Pose Invariant Face Recognition at a Distance Framework , 2014, IEEE Transactions on Information Forensics and Security.

[118]  Brian C. Lovell,et al.  TV-GAN: Generative Adversarial Network Based Thermal to Visible Face Recognition , 2017, 2018 International Conference on Biometrics (ICB).

[119]  廖文宏,et al.  Facial Feature Detection in Near-Infrared Images , 2003 .

[120]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).