FICGAN: Facial Identity Controllable GAN for De-identification

In this work, we present Facial Identity Controllable GAN (FICGAN) for not only generating high-quality de-identified face images with ensured privacy protection, but also detailed controllability on attribute preservation for enhanced data utility. We tackle the less-explored yet desired functionality in face de-identification based on the two factors. First, we focus on the challenging issue to obtain a high level of privacy protection in the de-identification task while uncompromising the image quality. Second, we analyze the facial attributes related to identity and non-identity and explore the trade-off between the degree of face de-identification and preservation of the source attributes for enhanced data utility. Based on the analysis, we develop Facial Identity Controllable GAN (FICGAN), an autoencoder-based conditional generative model that learns to disentangle the identity attributes from non-identity attributes on a face image. By applying the manifold k-same algorithm to satisfy k-anonymity for strengthened security, our method achieves enhanced privacy protection in de-identified face images. Numerous experiments demonstrate that our model outperforms others in various scenarios of face de-identification. Introduction Face de-identification refers to obscuring the identity of a person’s face by manipulating the identity dominant attributes such as nose, eyes, eyebrows, and mouth, whereas the identity invariant attributes are preserved, such as pose, expression, shadow, and illumination. The face is one of the most sensitive elements compare to other generic objects due to its personal identity information which is directly related to privacy. Therefore, face de-identification is an important research topic in social aspects of security and privacy, and has been utilized to anonymize the faces that appeared in media interviews, the street-views on video surveillance contents, and medical research data (Chi and Hu 2015; Senior 2009). The conventional methods, such as the black-bar masking, blurring, and pixelization, are simple but aggressive to remove the facial information, resulting in reduced data utility and unsuitable image quality (Ribaric and Pavesic 2015; Zhu et al. 2020a). Recently, deep learning methods have been employed for the task of face de-identification. Thanks to the advances in ar X iv :2 11 0. 00 74 0v 1 [ cs .C V ] 2 O ct 2 02 1 Generative Adversarial Networks (GAN) (Goodfellow et al. 2014), simple methods such as face swap (Bitouk et al. 2008) can be used to generate realistic outputs (Li et al. 2019; Zhu et al. 2020b); however, since it employs the target person’s face swapped onto the source image, it can be still identifiable and the risk of privacy leakage still remains. Also, since the facial attributes are simply inherited from the target, the attributes from the source cannot be preserved to be suited for various needs. To tackle this issue, most previous studies have taken either of the following two approaches: simply tweaking parts of the identity attributes or applying k-anonymity1 to face images to ensure privacy at the cost of quality degradation. Although these methods have paved a way for de-identification using deep learning, there is still room to improve. The risk of privacy leakage along with the challenge to control which facial attributes to preserve for data utility still remains for the first approach. Also, the blurry results from the averaged face of k number of images reduce the image quality in both aesthetic and utility aspects for the second approach. To tackle the issues, we have studied the under-explored challenges of securing a high level of privacy protection by achieving k-anonymity based on the k-same algorithm, while achieving the quality and controllability on the preservation of facial attributes. In this work, we propose Facial Identity Controllable GAN (FICGAN), a highly secure and controllable face de-identification GAN model that learns to disentangle identity and non-identity attributes. Our method differentiates itself in two aspects: First, our method achieves k-anonymity for robust security by using the simple manifold k-same algorithm (Yan, Pei, and Nie 2019), which exploits mixing the k number of different face images on the learned latent identity space as a target identity embedding. Second, the identity disentanglement and layer-wised generator enable detailed controllability on the degree of de-identification and also attribute preservation according to users’ preference. Numerous experiments demonstrate that our method can generate privacy-ensured de-identified face images with carefully controlled attributes from the source image to enhance data utility. Furthermore, we provide ample discussions and analysis on the trade-off as well as the relationship between the degree of identity attributes and non-identity attributes of the human face.