Asymmetric Joint GANs for Normalizing Face Illumination From a Single Image

Illumination normalization for face recognition is very important when a face is captured under harsh lighting conditions. Instead of designing hand-crafted features, in this paper we formulate face illumination normalization as an image-to-image translation task. A great challenge of face normalization is that human facial structures are particularly sensitive to image structure distortion, which frequently occurs in traditional image-to-image translation tasks. Unfortunately, sometimes even slight facial structure distortions may prohibit human eyes and machine face recognition methods from identifying face identities. To address this issue, a novel GAN-based network architecture called the asymmetric joint generative adversarial network (AJGAN) is developed to normalize face images under arbitrary illumination conditions, without known face geometry and albedo information. In addition, an illumination normalization GAN <inline-formula><tex-math notation="LaTeX">$G_1$</tex-math></inline-formula> and an asymmetric relighting GAN <inline-formula><tex-math notation="LaTeX">$G_2$</tex-math></inline-formula> that maps a frontal-illuminated image to images with various lighting conditions are incorporated in AJGAN to maintain personalized facial structures. To avoid image blurring caused by the under-constrained relighting mapping, we introduce a scheme of one-hot lighting labels into <inline-formula><tex-math notation="LaTeX">$G_2$</tex-math></inline-formula> and enforce label classification loss. Furthermore, the number of training images starting from a very limited number of labels is dynamically extended by the combination of different lighting labels. Qualitative and quantitative experiments on three databases validate that AJGAN significantly outperforms the state-of-the-art methods.