Hierarchical Group-level Emotion Recognition in the Wild

We propose a method for group-level emotion recognition in the wild. The main novelty of our method lies with the recognition of group-level emotions using a hierarchical classification approach. We consider that using the facial expressions of people will only be effective in differentiating images labeled as "Positive" because those labeled as "Neutral" or "Negative" are likely to include similar facial expressions (i.e., less discriminative). Therefore, we first perform binary classification based on facial expression recognition to distinguish "Positive" labels that include discriminative facial expressions (e.g., smile) from the others. We evaluate outcomes that are not classified as "Positive" at the first classification by exploiting scene features that describe what type of events (e.g., demonstration or funeral) are taking place in the image. Classification using scene features will not only be effective in differentiating "Negative" and "Neutral" labels but also in recognizing "Positive" labels, where facial expression features show less discriminative characteristics. The other novelty of the proposed method is to the exploitation of visual attention. Using visual attention allows us to estimate which faces are the main subjects in the target image, thereby suppressing the influences of faces in the background that contribute less to group-level emotion. We demonstrate the effectiveness of our proposed method through experiments using a public dataset.