A Study on the Performance of Unconstrained Very Low Resolution Face Recognition: Analyzing Current Trends and New Research Directions

In the past decade, research in the face recognition area has advanced tremendously, particularly in uncontrolled scenarios (face recognition in the wild). This advancement has been achieved partly due to the massive popularity and effectiveness of deep convolutional neural networks and the availability of larger unconstrained datasets. However, several face recognition challenges remain in the context of very low resolution homogeneous (same domain) and heterogeneous (different domain) face recognition. In this survey, we study the seminal and novel methods to tackle the very low resolution face recognition problem and provide an in-depth analysis of their design, effectiveness, and efficiency for a real-time surveillance application. Furthermore, we analyze the advantage of employing deep learning convolutional neural networks, while presenting future research directions for effective deep learning network design in this context.

[1]  Zhenhua Guo,et al.  Large Margin Coupled Mapping for Low Resolution Face Recognition , 2016, PRICAI.

[2]  Stefanos Zafeiriou,et al.  ArcFace: Additive Angular Margin Loss for Deep Face Recognition , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Sergey Alyamkin,et al.  MarginDistillation: distillation for margin-based softmax , 2020, ArXiv.

[4]  V. Kshirsagar,et al.  Face recognition using Eigenfaces , 2011, 2011 3rd International Conference on Computer Research and Development.

[5]  Luuk J. Spreeuwers,et al.  The Effect of Image Resolution on the Performance of a Face Recognition System , 2006, 2006 9th International Conference on Control, Automation, Robotics and Vision.

[6]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[7]  Raúl Monroy,et al.  Fusing pattern discovery and visual analytics approaches in tweet propagation , 2019, Inf. Fusion.

[8]  Qian Zhang,et al.  High Fidelity Face Manipulation with Extreme Pose and Expression , 2019, ArXiv.

[9]  Mark Sandler,et al.  Non-Discriminative Data or Weak Model? On the Relative Importance of Data and Model Resolution , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[10]  Mohamed Abdel-Mottaleb,et al.  Lower Resolution Face Recognition in Surveillance Systems Using Discriminant Correlation Analysis , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[11]  Patrick J. Flynn,et al.  On Low-Resolution Face Recognition in the Wild: Comparisons and New Techniques , 2018, IEEE Transactions on Information Forensics and Security.

[12]  Jian Cheng,et al.  Additive Margin Softmax for Face Verification , 2018, IEEE Signal Processing Letters.

[13]  Qijun Zhao,et al.  Towards resolution invariant face recognition in uncontrolled scenarios , 2016, 2016 International Conference on Biometrics (ICB).

[14]  Shengcai Liao,et al.  Learning Face Representation from Scratch , 2014, ArXiv.

[15]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[16]  Thomas S. Huang,et al.  Image super-resolution as sparse representation of raw image patches , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Heydi Mendez Vazquez,et al.  ShuffleFaceNet: A Lightweight Face Architecture for Efficient and Highly-Accurate Face Recognition , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[18]  Jürgen Beyerer,et al.  Low-resolution Convolutional Neural Networks for video face recognition , 2016, 2016 13th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[19]  Amit R.Sharma,et al.  Face Photo-Sketch Synthesis and Recognition , 2012 .

[20]  Yu Qiao,et al.  A Discriminative Feature Learning Approach for Deep Face Recognition , 2016, ECCV.

[21]  Sridha Sridharan,et al.  Super-resolution for biometrics: A comprehensive survey , 2018, Pattern Recognit..

[22]  Qian Zhang,et al.  VarGFaceNet: An Efficient Variable Group Convolutional Neural Network for Lightweight Face Recognition , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[23]  Narendra Ahuja,et al.  Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Mayank Vatsa,et al.  Dual Directed Capsule Network for Very Low Resolution Image Recognition , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[25]  Weigong Zhang,et al.  Enabling Highly Efficient Capsule Networks Processing Through A PIM-Based Architecture Design , 2019, 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA).

[26]  Hongyang Chao,et al.  TCN: Transferable Coupled Network for Cross-Resolution Face Recognition* , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[27]  Patrick J. Flynn,et al.  Pose-robust recognition of low-resolution face images , 2011, CVPR.

[28]  Quoc V. Le,et al.  Searching for MobileNetV3 , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[29]  Jesús Ariel Carrasco-Ochoa,et al.  A Pattern-Based Approach for Detecting Pneumatic Failures on Temporary Immersion Bioreactors † , 2019, Sensors.

[30]  Brahim Chaib-draa,et al.  Parametric Exponential Linear Unit for Deep Convolutional Neural Networks , 2016, 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA).

[31]  Junping Zhang,et al.  Super-resolution of human face image using canonical correlation analysis , 2010, Pattern Recognit..

[32]  Sivaram Prasad Mudunuri,et al.  Dictionary Alignment With Re-Ranking for Low-Resolution NIR-VIS Face Recognition , 2019, IEEE Transactions on Information Forensics and Security.

[33]  Debing Zhang,et al.  Lightweight Face Recognition Challenge , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[34]  Dong-Chen He,et al.  Texture Unit, Texture Spectrum And Texture Analysis , 1989, 12th Canadian Symposium on Remote Sensing Geoscience and Remote Sensing Symposium,.

[35]  Forrest N. Iandola,et al.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size , 2016, ArXiv.

[36]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Klemen Grm,et al.  Face Hallucination Using Cascaded Super-Resolution and Identity Priors , 2018, IEEE Transactions on Image Processing.

[38]  Sung Wook Baik,et al.  CNN-based anti-spoofing two-tier multi-factor authentication system , 2019, Pattern Recognit. Lett..

[39]  Andrew R. Webb,et al.  Multidimensional scaling by iterative majorization using radial basis functions , 1995, Pattern Recognit..

[40]  Georgios Tzimiropoulos,et al.  How Far are We from Solving the 2D & 3D Face Alignment Problem? (and a Dataset of 230,000 3D Facial Landmarks) , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[41]  Mislav Grgic,et al.  SCface – surveillance cameras face database , 2011, Multimedia Tools and Applications.

[42]  Ali Farhadi,et al.  XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks , 2016, ECCV.

[43]  Carlos D. Castillo,et al.  L2-constrained Softmax Loss for Discriminative Face Verification , 2017, ArXiv.

[44]  Terrance E. Boult,et al.  Large scale unconstrained open set face database , 2013, 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS).

[45]  Xinbo Gao,et al.  Re-Ranking High-Dimensional Deep Local Representation for NIR-VIS Face Recognition , 2019, IEEE Transactions on Image Processing.

[46]  Yiming Zhang,et al.  Coupled marginal discriminant mappings for low-resolution face recognition , 2015 .

[47]  Yuxiao Hu,et al.  MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition , 2016, ECCV.

[48]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[49]  Xudong Jiang,et al.  Deep Coupled ResNet for Low-Resolution Face Recognition , 2018, IEEE Signal Processing Letters.

[50]  Patrick J. Flynn,et al.  Face Recognition in Low Quality Images: A Survey , 2018, ArXiv.

[51]  Jianming Wei,et al.  SADG: Self-Aligned Dual NIR-VIS Generation for Heterogeneous Face Recognition , 2021 .

[52]  Raúl Monroy,et al.  Bagging-RandomMiner: a one-class classifier for file access-based masquerade detection , 2018, Machine Vision and Applications.

[53]  Kishor P. Upla,et al.  ComSupResNet: A Compact Super-Resolution Network for Low-Resolution Face Images , 2020, 2020 8th International Workshop on Biometrics and Forensics (IWBF).

[54]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[55]  Tieniu Tan,et al.  A Light CNN for Deep Face Representation With Noisy Labels , 2015, IEEE Transactions on Information Forensics and Security.

[56]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[57]  Georgios Tzimiropoulos,et al.  Training Binary Neural Networks with Real-to-Binary Convolutions , 2020, ICLR.

[58]  Shiming Ge,et al.  Low-Resolution Face Recognition in the Wild via Selective Knowledge Distillation , 2018, IEEE Transactions on Image Processing.

[59]  Gholamreza Anbarjafari,et al.  Facial image super resolution using sparse representation for improving face recognition in surveillance monitoring , 2016, 2016 24th Signal Processing and Communication Application Conference (SIU).

[60]  Richard Hartley,et al.  Semantic Face Hallucination: Super-Resolving Very Low-Resolution Face Images with Supplementary Attributes , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[61]  Yang Liu,et al.  MobileFaceNets: Efficient CNNs for Accurate Real-time Face Verification on Mobile Devices , 2018, CCBR.

[62]  Chengjun Liu,et al.  Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition , 2002, IEEE Trans. Image Process..

[63]  Chun Qi,et al.  From Local Geometry to Global Structure: Learning Latent Subspace for Low-resolution Face Image Recognition , 2015, IEEE Signal Processing Letters.

[64]  Irene Kotsia,et al.  RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[65]  Octavio Loyola-González,et al.  Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View , 2019, IEEE Access.

[66]  Pablo H. Hennings-Yeomans,et al.  Simultaneous super-resolution and feature extraction for recognition of low-resolution faces , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[67]  Heydi Mendez Vazquez,et al.  Lightweight Low-Resolution Face Recognition for Surveillance Applications , 2021, 2020 25th International Conference on Pattern Recognition (ICPR).

[68]  Richa Singh,et al.  Identity Aware Synthesis for Cross Resolution Face Recognition , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[69]  Patrick J. Flynn,et al.  Multidimensional Scaling for Matching Low-Resolution Face Images , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[70]  Zhenyu Wang,et al.  Low-resolution degradation face recognition over long distance based on CCA , 2015, Neural Computing and Applications.

[71]  Thomas S. Huang,et al.  Studying Very Low Resolution Recognition Using Deep Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[72]  Xiangyu Zhang,et al.  ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design , 2018, ECCV.

[73]  Zhenan Sun,et al.  Adversarial Cross-Spectral Face Completion for NIR-VIS Face Recognition , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[74]  Frederick R. Forst,et al.  On robust estimation of the location parameter , 1980 .

[75]  Wei Liu,et al.  Super-Identity Convolutional Neural Network for Face Hallucination , 2018, ECCV.

[76]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[77]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[78]  Wen Gao,et al.  The CAS-PEAL Large-Scale Chinese Face Database and Baseline Evaluations , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[79]  Stan Z. Li,et al.  Low-resolution face recognition via Simultaneous Discriminant Analysis , 2011, 2011 International Joint Conference on Biometrics (IJCB).

[80]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[81]  Bruce A. Draper,et al.  The challenge of face recognition from digital point-and-shoot cameras , 2013, 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS).

[82]  Shiguang Shan,et al.  Low-Resolution Face Recognition via Coupled Locality Preserving Mappings , 2010, IEEE Signal Processing Letters.

[83]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[84]  Bo Chen,et al.  Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[85]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.

[86]  Hyeonjoon Moon,et al.  The FERET evaluation methodology for face-recognition algorithms , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[87]  Takeo Kanade,et al.  Multi-PIE , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[88]  Kha Gia Quach,et al.  ShrinkTeaNet: Million-scale Lightweight Face Recognition via Shrinking Teacher-Student Networks , 2019, ArXiv.

[89]  Shaogang Gong,et al.  Low-Resolution Face Recognition , 2018, ACCV.

[90]  Yongjie Chu,et al.  Low-resolution face recognition with single sample per person , 2017, Signal Process..

[91]  Zhichao Li,et al.  VarGNet: Variable Group Convolutional Neural Network for Efficient Embedded Computing , 2019, ArXiv.

[92]  Thomas S. Huang,et al.  Deep Networks for Image Super-Resolution with Sparse Prior , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[93]  Xiaoou Tang,et al.  Deep Cascaded Bi-Network for Face Hallucination , 2016, ECCV.

[94]  Bir Bhanu,et al.  Face image super-resolution using 2D CCA , 2014, Signal Process..

[95]  Zhu Ming,et al.  CNN based spatio-temporal feature extraction for face anti-spoofing , 2017, 2017 2nd International Conference on Image, Vision and Computing (ICIVC).

[96]  Jian Yang,et al.  Two-dimensional PCA: a new approach to appearance-based face representation and recognition , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[97]  Bruce A. Draper,et al.  A meta-analysis of face recognition covariates , 2009, 2009 IEEE 3rd International Conference on Biometrics: Theory, Applications, and Systems.

[98]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[99]  Qingmin Liao,et al.  Discriminative Multidimensional Scaling for Low-Resolution Face Recognition , 2018, IEEE Signal Processing Letters.

[100]  Anil K. Jain,et al.  IJB–S: IARPA Janus Surveillance Video Benchmark , 2018, 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS).

[101]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[102]  Shaogang Gong,et al.  Surveillance Face Recognition Challenge , 2018, ArXiv.

[103]  Pooya Davari,et al.  A New Fast and Efficient HMM-Based Face Recognition System Using a 7-State HMM Along With SVD Coefficients , 2008 .

[104]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[105]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[106]  Michael Elad,et al.  On Single Image Scale-Up Using Sparse-Representations , 2010, Curves and Surfaces.

[107]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[108]  Radu Timofte,et al.  AIM 2019 Challenge on Constrained Super-Resolution: Methods and Results , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[109]  Xiaoming Liu,et al.  FAN: Feature Adaptation Network for Surveillance Face Recognition and Normalization , 2019, ACCV.

[110]  Seungjin Choi,et al.  Two-Dimensional Canonical Correlation Analysis , 2007, IEEE Signal Processing Letters.

[111]  Yu Qiao,et al.  Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks , 2016, IEEE Signal Processing Letters.

[112]  Jie Cao,et al.  Softmax Cross Entropy Loss with Unbiased Decision Boundary for Image Classification , 2018, 2018 Chinese Automation Congress (CAC).

[113]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[114]  Geoffrey E. Hinton,et al.  Transforming Autoencoders , 2011 .

[115]  B. V. K. Vijaya Kumar,et al.  Coupled Marginal Fisher Analysis for Low-Resolution Face Recognition , 2012, ECCV Workshops.

[116]  Sivaram Prasad Mudunuri,et al.  GenLR-Net: Deep Framework for Very Low Resolution Face and Object Recognition with Generalization to Unseen Categories , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[117]  Timothy Hospedales,et al.  A survey on heterogeneous face recognition: Sketch, infra-red, 3D and low-resolution , 2014, Image Vis. Comput..

[118]  Tal Hassner,et al.  Face recognition in unconstrained videos with matched background similarity , 2011, CVPR 2011.

[119]  Xiangyu Zhang,et al.  ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.