Prior Aided Streaming Network for Multi-task Affective Analysis

Automatic affective recognition has been an important research topic in the human-computer interaction (HCI) area. With the recent development of deep learning techniques and large-scale in-the-wild annotated datasets, facial emotion analysis is now aimed at challenges in real world settings. In this paper, we introduce our submission to the 2nd Affective Behavior Analysis in-the-wild (ABAW2) Competition. In dealing with different emotion representations, including Categorical Expression (EXPR), Action Units (AU), and Valence Arousal (VA), we propose a multitask streaming network by a heuristic that the three representations are intrinsically associated with each other. Besides, we leverage an advanced facial expression embedding model as prior knowledge, which is capable of capturing identity-invariant expression features while preserving the expression similarities, to aid the down-streaming recognition tasks. In order to enhance the generalization ability of our model, we generate reliable pseudo labels for unsupervised training and adopt external datasets for fine-tuning. In the official test of ABAW2 Competition, our method ranks first in the EXPR and AU tracks and second in the VA track. The extensive quantitative evaluations, as well as ablation studies on the Aff-Wild2 dataset, prove the effectiveness of our proposed method.