Face Recognition Systems Under Morphing Attacks: A Survey

Recently, researchers found that the intended generalizability of (deep) face recognition systems increases their vulnerability against attacks. In particular, the attacks based on morphed face images pose a severe security risk to face recognition systems. In the last few years, the topic of (face) image morphing and automated morphing attack detection has sparked the interest of several research laboratories working in the field of biometrics and many different approaches have been published. In this paper, a conceptual categorization and metrics for an evaluation of such methods are presented, followed by a comprehensive survey of relevant publications. In addition, technical considerations and tradeoffs of the surveyed methods are discussed along with open issues and challenges in the field.

[1]  D. W. Choi,et al.  Image Morphing Using Mass-Spring System , 2011 .

[2]  Scott Schaefer,et al.  Image deformation using moving least squares , 2006, ACM Trans. Graph..

[3]  Takeo Kanade,et al.  Multi-PIE , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[4]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[5]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Davide Maltoni,et al.  The magic passport , 2014, IEEE International Joint Conference on Biometrics.

[7]  Sanjit K. Mitra,et al.  No-reference video quality metric based on artifact measurements , 2005, IEEE International Conference on Image Processing 2005.

[8]  J. L. Wayman,et al.  Best practices in testing and reporting performance of biometric devices. , 2002 .

[9]  Adam Schmidt,et al.  The put face database , 2008 .

[10]  Kiran B. Raja,et al.  Detecting morphed face images , 2016, 2016 IEEE 8th International Conference on Biometrics Theory, Applications and Systems (BTAS).

[11]  Sébastien Marcel,et al.  Deeply vulnerable: a study of the robustness of face recognition to presentation attacks , 2018, IET Biom..

[12]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[13]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[14]  Kiran B. Raja,et al.  Transferable Deep-CNN Features for Detecting Digital and Print-Scanned Morphed Face Images , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[15]  Mahadev Satyanarayanan,et al.  OpenFace: A general-purpose face recognition library with mobile applications , 2016 .

[16]  Ahmed M. Megreya,et al.  Unfamiliar faces are not faces: Evidence from a matching task , 2006, Memory & cognition.

[17]  Kiran B. Raja,et al.  On the vulnerability of face recognition systems towards morphed face attacks , 2017, 2017 5th International Workshop on Biometrics and Forensics (IWBF).

[18]  Naser Damer,et al.  MorGAN: Recognition Vulnerability and Attack Detectability of Face Morphing Attacks Created by Generative Adversarial Network , 2018, 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS).

[19]  A. Burton,et al.  Passport Officers’ Errors in Face Matching , 2014, PloS one.

[20]  Christoph Busch,et al.  Detecting Morphed Face Images Using Facial Landmarks , 2018, ICISP.

[21]  Manjunath V. Joshi,et al.  Automatic target image detection for morphing , 2015, J. Vis. Commun. Image Represent..

[22]  Heinrich Müller,et al.  Image warping with scattered data interpolation , 1995, IEEE Computer Graphics and Applications.

[23]  Ahmed M Megreya,et al.  Matching faces to photographs: poor performance in eyewitness memory (without the memory). , 2008, Journal of experimental psychology. Applied.

[24]  G. Pike,et al.  When Seeing should not be Believing: Photographs, Credit Cards and Fraud , 1997 .

[25]  Eli Shechtman,et al.  Regenerative morphing , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[26]  Sébastien Marcel,et al.  Biometric Systems under Morphing Attacks: Assessment of Morphing Techniques and Vulnerability Reporting , 2017, 2017 International Conference of the Biometrics Special Interest Group (BIOSIG).

[27]  Jian Sun,et al.  Blessing of Dimensionality: High-Dimensional Feature and Its Efficient Compression for Face Verification , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Davide Maltoni,et al.  On the Effects of Image Alterations on Face Recognition Accuracy , 2016, Face Recognition Across the Imaging Spectrum.

[29]  Patrick J. Flynn,et al.  Overview of the face recognition grand challenge , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[30]  Davide Maltoni,et al.  On the Feasibility of Creating Double-Identity Fingerprints , 2017, IEEE Transactions on Information Forensics and Security.

[31]  Christoph Busch,et al.  On the feasibility of creating morphed iris-codes , 2017, 2017 IEEE International Joint Conference on Biometrics (IJCB).

[32]  Olac Fuentes,et al.  An Approach to Automatic Morphing of Face Images in Frontal View , 2004, MICAI.

[33]  Jana Dittmann,et al.  Modeling Attacks on Photo-ID Documents and Applying Media Forensics for the Detection of Facial Morphing , 2017, IH&MMSec.

[34]  Harry Wechsler,et al.  The FERET database and evaluation procedure for face-recognition algorithms , 1998, Image Vis. Comput..

[35]  Christoph Busch,et al.  Performance variation of morphed face image detection algorithms across different datasets , 2018, 2018 International Workshop on Biometrics and Forensics (IWBF).

[36]  Jonathan Wu.  Face Recognition Jammer using Image Morphing , 2011 .

[37]  Bülent Sankur,et al.  A comparative study of face landmarking techniques , 2013, EURASIP J. Image Video Process..

[38]  Fei Yang,et al.  Face morphing using 3D-aware appearance optimization , 2012, Graphics Interface.

[39]  Weisi Lin,et al.  A no-reference quality metric for measuring image blur , 2003, Seventh International Symposium on Signal Processing and Its Applications, 2003. Proceedings..

[40]  Joshua Correll,et al.  The Chicago face database: A free stimulus set of faces and norming data , 2015, Behavior research methods.

[41]  Christoph Busch,et al.  PRNU-based detection of morphed face images , 2018, 2018 International Workshop on Biometrics and Forensics (IWBF).

[42]  Richa Singh,et al.  SWAPPED! Digital face presentation attack detection via weighted local magnitude pattern , 2017, 2017 IEEE International Joint Conference on Biometrics (IJCB).

[43]  Axel Mecklinger,et al.  Picture database of morphed faces (MoFa) : technical report , 2005 .

[44]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Christoph Busch,et al.  Predicting the vulnerability of biometric systems to attacks based on morphed biometric information , 2018, IET Biom..

[46]  Ian Craw,et al.  Finding Face Features , 1992, ECCV.

[47]  Martin Bichsel.  Automatic interpolation and recognition of face images by morphing , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[48]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[49]  Davis E. King,et al.  Dlib-ml: A Machine Learning Toolkit , 2009, J. Mach. Learn. Res..

[50]  Christoph Busch,et al.  Towards Detection of Morphed Face Images in Electronic Travel Documents , 2018, 2018 13th IAPR International Workshop on Document Analysis Systems (DAS).

[51]  Jana Dittmann,et al.  Automatic Generation and Detection of Visually Faultless Facial Morphs , 2017, VISIGRAPP.

[52]  Ahmed M. Megreya,et al.  Matching Face Images Taken on the Same Day or Months Apart: the Limitations of Photo ID , 2013 .

[53]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[54]  Jing Liao,et al.  Automating Image Morphing Using Structural Similarity on a Halfway Domain , 2014, ACM Trans. Graph..

[55]  Christoph Busch,et al.  Morph Deterction from Single Face Image: a Multi-Algorithm Fusion Approach , 2018, ICBEA '18.

[56]  Robin S S Kramer,et al.  Fraudulent ID using face morphs: Experiments on human and automatic recognition , 2017, PloS one.

[57]  Andreas Uhl,et al.  A survey on biometric cryptosystems and cancelable biometrics , 2011, EURASIP J. Inf. Secur..

[58]  Sung Yong Shin,et al.  Image metamorphosis using snakes and free-form deformations , 1995, SIGGRAPH.

[59]  Kiran B. Raja,et al.  Face morphing versus face averaging: Vulnerability and detection , 2017, 2017 IEEE International Joint Conference on Biometrics (IJCB).

[60]  Raul Vicente-Garcia,et al.  Morphing Detection Using a General- Purpose Face Recognition System , 2018, 2018 26th European Signal Processing Conference (EUSIPCO).

[61]  Enhua Wu,et al.  Robust image metamorphosis immune from ghost and blur , 2012, The Visual Computer.

[62]  Josephine Sullivan,et al.  One millisecond face alignment with an ensemble of regression trees , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[63]  Christoph Busch,et al.  PRNU Variance Analysis for Morphed Face Image Detection , 2018, 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS).

[64]  Azriel Rosenfeld,et al.  Face recognition: A literature survey , 2003, CSUR.

[65]  Lucas Theis,et al.  Fast Face-Swap Using Convolutional Neural Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[66]  Fei Peng,et al.  Face Morphing Detection Using Fourier Spectrum of Sensor Pattern Noise , 2018, 2018 IEEE International Conference on Multimedia and Expo (ICME).

[67]  Matti Pietikäinen,et al.  A comparative study of texture measures with classification based on featured distributions , 1996, Pattern Recognit..

[68]  Anil K. Jain,et al.  Biometric Template Protection: Bridging the performance gap between theory and practice , 2015, IEEE Signal Processing Magazine.

[69]  Lei Zhang,et al.  A Probabilistic Collaborative Representation Based Approach for Pattern Classification , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[70]  Mislav Grgic,et al.  SCface – surveillance cameras face database , 2011, Multimedia Tools and Applications.

[71]  Arun Ross,et al.  Mixing fingerprints for generating virtual identities , 2011, 2011 IEEE International Workshop on Information Forensics and Security.

[72]  Anna Hilsmann,et al.  Accurate and Robust Neural Networks for Security Related Applications Exampled by Face Morphing Attacks , 2018, ArXiv.

[73]  Michael G. Strintzis,et al.  Face Recognition , 2008, Encyclopedia of Multimedia.

[74]  Andreas Wolf,et al.  An Overview of Recent Advances in Assessing and Mitigating the Face Morphing Attack , 2018, 2018 26th European Signal Processing Conference (EUSIPCO).

[75]  Aras Asaad.  Automatic Detection of Image Morphing by Topology-based Analysis , 2018, 2018 26th European Signal Processing Conference (EUSIPCO).

[76]  David J Robertson,et al.  Detecting morphed passport photos: a training and individual differences approach , 2018, Cognitive research: principles and implications.

[77]  Chi Fang,et al.  Histogram of the oriented gradient for face recognition , 2011 .

[78]  Mahmoud Al-Ayyoub,et al.  Impact of digital fingerprint image quality on the fingerprint recognition accuracy , 2017, Multimedia Tools and Applications.

[79]  Zhou Wang,et al.  No-reference perceptual quality assessment of JPEG compressed images , 2002, Proceedings. International Conference on Image Processing.

[80]  Pong C. Yuen,et al.  On the Reconstruction of Face Images from Deep Face Templates , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[81]  Aksh Patel.  Image Morphing Algorithm: A Survey , 2015 .

[82]  Aleix M. Martinez,et al.  The AR face database , 1998 .

[83]  Wen Gao,et al.  The CAS-PEAL Large-Scale Chinese Face Database and Baseline Evaluations , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[84]  Kiran B. Raja,et al.  Detecting Face Morphing Attacks with Collaborative Representation of Steerable Features , 2018, CVIP.

[85]  Davide Maltoni,et al.  Face Demorphing , 2018, IEEE Transactions on Information Forensics and Security.

[86]  Natalia A. Schmid,et al.  Image quality assessment for iris biometric , 2006, SPIE Defense + Commercial Sensing.

[87]  Paul Miller,et al.  Verification of face identities from images captured on video. , 1999 .

[88]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[89]  C. Thomaz,et al.  A new ranking method for principal components analysis and its application to face image analysis , 2010, Image Vis. Comput..

[90]  Thomas W. Sederberg,et al.  Free-form deformation of solid geometric models , 1986, SIGGRAPH.

[91]  Anil K. Jain,et al.  Fingerprint Quality Indices for Predicting Authentication Performance , 2005, AVBPA.

[92]  Shree K. Nayar,et al.  Face swapping: automatically replacing faces in photographs , 2008, SIGGRAPH 2008.

[93]  Yajie Tian,et al.  Handbook of face recognition , 2003 .

[94]  Natalia A. Schmid,et al.  Estimating and Fusing Quality Factors for Iris Biometric Images , 2010, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[95]  Alan C. Bovik,et al.  No-Reference Image Quality Assessment in the Spatial Domain , 2012, IEEE Transactions on Image Processing.

[96]  Thaddeus Beier,et al.  Feature-based image metamorphosis , 1998 .

[97]  Tim Valentine,et al.  CCTV on trial: Matching video images with the defendant in the dock , 2009 .

[98]  Kun Zhou,et al.  Hair Interpolation for Portrait Morphing , 2013, Comput. Graph. Forum.

[99]  Anna Hilsmann,et al.  Detection of Face Morphing Attacks by Deep Learning , 2017, IWDW.

[100]  Sabah Jassim,et al.  Topological Data Analysis for Image Tampering Detection , 2017, IWDW.

[101]  Christoph Busch,et al.  Is your biometric system robust to morphing attaeks? , 2017, 2017 5th International Workshop on Biometrics and Forensics (IWBF).

[102]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[103]  Davide Maltoni,et al.  Face demorphing in the presence of facial appearance variations , 2018, 2018 26th European Signal Processing Conference (EUSIPCO).

[104]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[105]  Anna Hilsmann,et al.  Reflection Analysis for Face Morphing Attack Detection , 2018, 2018 26th European Signal Processing Conference (EUSIPCO).

[106]  Norbert Krüger,et al.  Face Recognition by Elastic Bunch Graph Matching , 1997, CAIP.

[107]  Luuk J. Spreeuwers,et al.  Towards Robust Evaluation of Face Morphing Detection , 2018, 2018 26th European Signal Processing Conference (EUSIPCO).

[108]  George Wolberg,et al.  Image morphing: a survey , 1998, The Visual Computer.

[109]  Timothy F. Cootes,et al.  Active Appearance Models , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[110]  Jana Dittmann,et al.  Generalized Benford's Law for Blind Detection of Morphed Face Images , 2018, IH&MMSec.

[111]  Xi Chen,et al.  CNNs Under Attack: On the Vulnerability of Deep Neural Networks Based Face Recognition to Image Morphing , 2017, IWDW.

[112]  Simon Lucey,et al.  Face alignment through subspace constrained mean-shifts , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[113]  Esa Rahtu,et al.  BSIF: Binarized statistical image features , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[114]  Tom Neubert,et al.  Face Morphing Detection: An Approach Based on Image Degradation Analysis , 2017, IWDW.

[115]  Jana Dittmann,et al.  Benchmarking face morphing forgery detection: Application of stirtrace for impact simulation of different processing steps , 2017, 2017 5th International Workshop on Biometrics and Forensics (IWBF).

[116]  Deva Ramanan,et al.  Face detection, pose estimation, and landmark localization in the wild , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[117]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..