Cross-spectral cross-resolution face recognition in videos

One of the challenging applications in face recognition is video surveillance, where unconstrained low-resolution video data is captured both in day and night time (visible and near-infrared) with multiple subjects in frames, which are matched with high resolution gallery images. Due to the lack of an existing database for such a cross spectral cross resolution video-to-still face recognition application, this still remains an open research problem. The aim of this research is to come up with robust face recognition algorithms in surveillance scenarios. For this, we first present a video database which can be utilized to benchmark face recognition algorithms addressing crossspectral and cross-resolution matching. The proposed Cross-Spectral Cross-Resolution Video dataset version 2 (CSCRV-v2) contains 460 videos pertaining to 252 subjects with an open-set protocol. We then focus on the first step in the face recognition pipeline i.e., face detection and propose an algorithm FaceFinder. FaceFinder addresses shortcomings (like high false positive rate) of existing face detectors by making use of human body segmentation results of a trained Convolutional Neural Network model specifically designed for semantic segmentation. Experimental results on CSCRV-v2, are compared with that of two off-the-shelf face detection algorithms to show the efficacy of the proposed algorithm. We then focus on face recognition and present results with two commercial matchers for two experimental scenarios on the data. It is our assertion that our research will further facilitate the research community to develop robust face recognition algorithms to handle real world surveillance scenarios.

[1]  Mahadev Satyanarayanan,et al.  OpenFace: A general-purpose face recognition library with mobile applications , 2016 .

[2]  Stefanos Zafeiriou,et al.  A survey on face detection in the wild: Past, present and future , 2015, Comput. Vis. Image Underst..

[3]  Takeo Kanade,et al.  Rotation Invariant Neural Network-Based Face Detection , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[4]  Shengcai Liao,et al.  Illumination Invariant Face Recognition Using Near-Infrared Images , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Bruce A. Draper,et al.  The challenge of face recognition from digital point-and-shoot cameras , 2013, 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS).

[6]  Richa Singh,et al.  On Effectiveness of Histogram of Oriented Gradient Features for Visible to Near Infrared Face Matching , 2014, 2014 22nd International Conference on Pattern Recognition.

[7]  Narendra Ahuja,et al.  Detecting Faces in Images: A Survey , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Vibhav Vineet,et al.  Conditional Random Fields as Recurrent Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[9]  Tal Hassner,et al.  Face recognition in unconstrained videos with matched background similarity , 2011, CVPR 2011.

[10]  Anil K. Jain,et al.  Nighttime face recognition at large standoff: Cross-distance and cross-spectral matching , 2014, Pattern Recognit..

[11]  Patrick J. Flynn,et al.  Active Clustering with Ensembles for Social structure extraction , 2014, IEEE Winter Conference on Applications of Computer Vision.

[12]  Shengcai Liao,et al.  A Fast and Accurate Unconstrained Face Detector , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Richa Singh,et al.  Feature and keypoint selection for visible to near-infrared face matching , 2015, 2015 IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS).

[14]  Doina Precup,et al.  Hierarchical temporal graphical model for head pose estimation and subsequent attribute classification in real-world videos , 2015, Comput. Vis. Image Underst..

[15]  Rama Chellappa,et al.  Partial face detection for continuous authentication , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[16]  Ioannis T. Pavlidis,et al.  Face Detection in the Near-IR Spectrum , 2003, Image Vis. Comput..

[17]  Himanshu S. Bhatt,et al.  Submitted to Ieee Transactions on Image Processing 1 Improving Cross-resolution Face Matching Using Ensemble Based Co-transfer Learning , 2022 .

[18]  Sivaram Prasad Mudunuri,et al.  Low Resolution Face Recognition Across Variations in Pose and Illumination , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Singh Richa,et al.  Cross-spectral cross-resolution video database for face recognition , 2016 .

[20]  Jean-Luc Dugelay,et al.  Learned vs. Hand-Crafted Features for Pedestrian Gender Recognition , 2015, ACM Multimedia.

[21]  Robert T. Collins,et al.  Silhouette-based human identification from body shape and gait , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[22]  Tsuhan Chen,et al.  The CMU Face In Action (FIA) Database , 2005, AMFG.

[23]  Jian-Huang Lai,et al.  Matching NIR Face to VIS Face Using Transduction , 2014, IEEE Transactions on Information Forensics and Security.

[24]  Shengcai Liao,et al.  The CASIA NIR-VIS 2.0 Face Database , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[25]  Richa Singh,et al.  Annotated crowd video face database , 2015, 2015 International Conference on Biometrics (ICB).

[26]  Yongkang Wong,et al.  Patch-based probabilistic image quality assessment for face selection and improved video-based face recognition , 2011, CVPR 2011 WORKSHOPS.

[27]  Takeo Kanade,et al.  Neural Network-Based Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Andrew Zisserman,et al.  "Who are you?" - Learning person specific classifiers from video , 2009, CVPR.

[29]  Erik Learned-Miller,et al.  FDDB: A benchmark for face detection in unconstrained settings , 2010 .

[30]  Marios Savvides,et al.  NIR-VIS heterogeneous face recognition via cross-spectral joint dictionary learning and reconstruction , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[31]  Dong Yi,et al.  Face Matching Between Near Infrared and Visible Light Images , 2007, ICB.

[32]  Gwen Littlewort,et al.  Toward Practical Smile Detection , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Jiri Matas,et al.  Forward-Backward Error: Automatic Detection of Tracking Failures , 2010, 2010 20th International Conference on Pattern Recognition.

[34]  Mislav Grgic,et al.  SCface – surveillance cameras face database , 2011, Multimedia Tools and Applications.

[35]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.