Semi-Supervised Learning for Monocular Gaze Redirection

We present a new approach to monocular learning-based gaze redirection problem in images that is able to train on raw sequences of eye images with unknown gaze directions and a small amount of eye images, where the gaze direction is known. The proposed approach is based on a pair of deep networks, where the first encoder-like network maps eye images to a latent space, while the second network maps pairs of latent representations to warping fields implementing the transformation between the pair of the original images. In the proposed system, both networks are trained in an unsupervised manner, while the gaze-annotated images are only used to estimate displacements in the latent space that are characteristic to certain gaze redirections. Quantitative and qualitative evaluation suggests that such characteristic displacement vectors in the learned latent space can be learned from few examples and are transferable across different people and different imaging conditions.