SeLENet: A Semi-Supervised Low Light Face Enhancement Method for Mobile Face Unlock

Facial recognition is becoming a standard feature on new smartphones. However, the face unlocking feature of devices using regular 2D camera sensors exhibits poor performance in low light environments. In this paper, we propose a semi-supervised low light face enhancement method to improve face verification performance on low light face images. The proposed method is a network with two components: decomposition and reconstruction. The decomposition component splits an input low light face image into face normals and face albedo, while the reconstruction component enhances and reconstructs the lighting condition of the input image using the spherical harmonic lighting coefficients of a direct ambient white light. The network is trained in a semi-supervised manner using both labeled synthetic data and unlabeled real data. Qualitative results demonstrate that the proposed method produces more realistic images than the state-of-the-art low light enhancement algorithms. Quantitative experiments confirm the effectiveness of our low light face enhancement method for face verification. By applying the proposed method, the gap of verification accuracy between extreme low light and neutral light face images is reduced from approximately 3% to 0.5%.

[1]  Pat Hanrahan,et al.  An efficient representation for irradiance environment maps , 2001, SIGGRAPH.

[2]  Ran He,et al.  Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[3]  Omkar M. Parkhi,et al.  VGGFace2: A Dataset for Recognising Faces across Pose and Age , 2017, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[4]  Stefanos Zafeiriou,et al.  ArcFace: Additive Angular Margin Loss for Deep Face Recognition , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Chen Wei,et al.  GLADNet: Low-Light Enhancement Network with Global Awareness , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[6]  Ioannis A. Kakadiaris,et al.  UHDB31: A Dataset for Better Understanding Face Recognition Across Pose and Illumination Variation , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[7]  Josef Kittler,et al.  Semi-supervised Adversarial Learning to Generate Photorealistic Face Images of New Identities from 3D Morphable Model , 2018, ECCV.

[8]  Soumik Sarkar,et al.  LLNet: A deep autoencoder approach to natural low-light image enhancement , 2015, Pattern Recognit..

[9]  Ioannis A. Kakadiaris,et al.  Illumination-Invariant Face Recognition With Deep Relit Face Images , 2019, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[10]  Yang Liu,et al.  MobileFaceNets: Efficient CNNs for Accurate Real-time Face Verification on Mobile Devices , 2018, CCBR.

[11]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[12]  Bhiksha Raj,et al.  SphereFace: Deep Hypersphere Embedding for Face Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Carlos D. Castillo,et al.  Frontal to profile face verification in the wild , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[14]  Stefanos Zafeiriou,et al.  Marginal Loss for Deep Face Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[15]  Bernhard Egger,et al.  Morphable Face Models - An Open Framework , 2017, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[16]  Chen Wei,et al.  Deep Retinex Decomposition for Low-Light Enhancement , 2018, BMVC.

[17]  Ioannis A. Kakadiaris,et al.  On the Importance of Feature Aggregation for Face Reconstruction , 2019, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[18]  Fatih Murat Porikli,et al.  LightenNet: A Convolutional Neural Network for weakly illuminated image enhancement , 2018, Pattern Recognit. Lett..

[19]  Xi Zhou,et al.  Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network , 2018, ECCV.

[20]  Yu Qiao,et al.  A Discriminative Feature Learning Approach for Deep Face Recognition , 2016, ECCV.

[21]  Zheng Zhang,et al.  MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems , 2015, ArXiv.

[22]  Ioannis A. Kakadiaris,et al.  Evaluation of a 3D-aided pose invariant 2D face recognition system , 2017, 2017 IEEE International Joint Conference on Biometrics (IJCB).

[23]  Xiaoyan Sun,et al.  Structure-Revealing Low-Light Image Enhancement Via Robust Retinex Model , 2018, IEEE Transactions on Image Processing.

[24]  Joonki Paik,et al.  Dual Autoencoder Network for Retinex-Based Low-Light Image Enhancement , 2018, IEEE Access.

[25]  Shengcai Liao,et al.  Learning Face Representation from Scratch , 2014, ArXiv.

[26]  Yaser Yacoob,et al.  Label Denoising Adversarial Network (LDAN) for Inverse Lighting of Faces , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Bernhard Egger,et al.  Occlusion-Aware 3D Morphable Models and an Illumination Prior for Face Image Analysis , 2018, International Journal of Computer Vision.

[28]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[29]  Davis E. King,et al.  Dlib-ml: A Machine Learning Toolkit , 2009, J. Mach. Learn. Res..

[30]  Qian Chen,et al.  Image denoising by bounded block matching and 3D filtering , 2010, Signal Process..

[31]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).