Identity-based Adversarial Training of Deep CNNs for Facial Action Unit Recognition

Automatic detection of facial action units (AU) is a key towards fully understanding spontaneous facial expressions in human emotion analysis. The most recent development shows encouraging results in utilizing deep learning models to recognize facial action units. However, existing AU-labeled spontaneous facial expression datasets are either in a small-scale due to labor-intensive annotations, or lack of sufficient variety in terms of amount, ethical background, age ranges, and facial appearance variations of subjects, thus limiting the learning effectiveness. To mitigate the issue of high redundancy and low level of variants existing among image frames of facial video sequences with respect to both subject identities (ID) and facial action units (AU), we propose a novel learning process with convolutional neural networks (CNNs), named Adversarial Training Framework (ATF). Architecture wise, ATF takes the form of a multi-task learning CNN, where a deep stack of convolutional layers, pooling layers and fully connected layers act as feature learners, and an AU loss together with an ID loss is computed with the shared deep representations. The crucial property of ATF is that during training, the underlying feature layers are trained to minimize the AU loss while maximizing the ID loss. This adversarial training w.r.t. the ID loss essentially makes the learned features effective for AU detection, while invariant to subject identities, thus alleviating the affection of personal identity to a great extent. We have conducted experiments on the public databases BP4D and DISFA, showing significant performance improvements over the peer methods.