Face alignment using a deep neural network with local feature learning and recurrent regression

Abstract We propose a face alignment method that uses a deep neural network employing both local feature learning and recurrent regression. This method is primarily based on a convolutional neural network(CNN), which automatically learns local feature descriptors from the local facial landmark dataset that we created. Our research is motivated by the belief that investigating a face from its low-level component features would produce more competitive face alignment results, just as a CNN is normally trained to automatically learn a feature hierarchy from the lowest to the highest levels of abstraction. Moreover, by separately training the feature extraction layers and the regression layers, we impose an explicit functional discrimination between the feature extraction and regression tasks. First, we train a feature extraction network that is used to classify the landmark patches in the dataset. Using this pre-trained feature extraction network, we build a face alignment network, which uses an entire face image rather than the local landmark patch as input, thus generating the global facial features. The subsequent local feature extraction layer extracts the local feature set from this global feature, finally generating the local feature descriptors, in which space the network learns a generic descent direction from the currently estimated landmark positions to the ground truth via linear regression applied recurrently. Head pose estimation network also applied to provide a good initial estimate to the local feature extraction layer for accurate convergence. We found that learning of the good local landmark features in pursuit of good landmark classification also leads to a higher face alignment accuracy and achieves state-of-the-art performance on several public benchmark dataset. It signifies the importance of learning not only the global features but the local features for face alignment. We further verify our method's effectiveness when applied to related problems such as head pose estimation, facial landmark tracking, and invisible landmark detection. We believe that good local learning enables a deeper understanding of the face or object resulting in higher performance.