Facial Expression Recognition Using a Large Out-of-Context Dataset

We develop a method for emotion recognition from facial imagery. This problem is challenging in part because of the subjectivity of ground truth labels and in part because of the relatively small size of existing labeled datasets. We use the FER+ dataset [8], a dataset with multiple emotion labels per image, in order to build an emotion recognition model that encompasses a full range of emotions. Since the amount of data in the FER+ dataset is limited, we explore the use of a much larger face dataset, MS-Celeb-1M [41], in conjunction with the FER+ dataset. Specific layers within an Inception-ResNet-v1 [13, 38] model trained for facial recognition are used for the emotion recognition problem. Thus, we leverage the MS-Celeb-1M dataset in addition to the FER+ dataset and experiment with different architectures to assess the overall performance of neural networks to recognize emotion using facial imagery.

[1]  Bin Zhang,et al.  Study on CNN in the recognition of emotion in audio and images , 2016, 2016 IEEE/ACIS 15th International Conference on Computer and Information Science (ICIS).

[2]  Paolo Favaro,et al.  Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles , 2016, ECCV.

[3]  Christopher Joseph Pal,et al.  EmoNets: Multimodal deep learning approaches for emotion recognition in video , 2015, Journal on Multimodal User Interfaces.

[4]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[5]  Ute Habel,et al.  The impact of facial emotional expressions on behavioral tendencies in women and men. , 2010, Journal of experimental psychology. Human perception and performance.

[6]  Shiguang Shan,et al.  AU-aware Deep Networks for facial expression recognition , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[7]  M. Pantic,et al.  Induced Disgust , Happiness and Surprise : an Addition to the MMI Facial Expression Database , 2010 .

[8]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Yuxiao Hu,et al.  MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition , 2016, ECCV.

[10]  P. Ekman,et al.  Strong evidence for universals in facial expressions: a reply to Russell's mistaken critique. , 1994, Psychological bulletin.

[11]  Yoshua Bengio,et al.  Challenges in representation learning: A report on three machine learning contests , 2013, Neural Networks.

[12]  Philip S. Yu,et al.  Learning Multiple Tasks with Multilinear Relationship Networks , 2015, NIPS.

[13]  Q. Huys,et al.  Individual differences in bodily freezing predict emotional biases in decision making , 2014, Front. Behav. Neurosci..

[14]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[15]  Thomas S. Huang,et al.  How deep neural networks can improve emotion recognition on video data , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[16]  B. Averbeck,et al.  A Selective Emotional Decision-Making Bias Elicited by Facial Expressions , 2012, PloS one.

[17]  A. Ohman,et al.  Emotional conditioning to masked stimuli: expectancies for aversive outcomes following nonrecognized fear-relevant stimuli. , 1998, Journal of experimental psychology. General.

[18]  A. Ohman,et al.  Emotional conditioning to masked stimuli: expectancies for aversive outcomes following nonrecognized fear-relevant stimuli. , 1998, Journal of experimental psychology. General.

[19]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[20]  M. Kosinski,et al.  Deep Neural Networks Are More Accurate Than Humans at Detecting Sexual Orientation From Facial Images , 2018, Journal of personality and social psychology.

[21]  Luca Citi,et al.  Revealing Real-Time Emotional Responses: a Personalized Assessment based on Heartbeat Dynamics , 2014, Scientific Reports.

[22]  M. Guarnera,et al.  Facial Expressions and Ability to Recognize Emotions From Eyes or Mouth in Children , 2015, Europe's journal of psychology.

[23]  Cha Zhang,et al.  Image based Static Facial Expression Recognition with Multiple Deep Network Learning , 2015, ICMI.

[24]  Sung-Nien Yu,et al.  Emotion state identification based on heart rate variability and genetic algorithm. , 2015, Conference proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual Conference.

[25]  P. Beek,et al.  Walk to me when I smile, step back when I’m angry: emotional faces modulate whole-body approach–avoidance behaviors , 2011, Experimental Brain Research.

[26]  B. Knutson.  Facial expressions of emotion influence interpersonal trait inferences , 1996 .

[27]  Paul W. Fieguth,et al.  Efficient Deep Feature Learning and Extraction via StochasticNets , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[28]  Timothée Masquelier,et al.  Deep Networks Can Resemble Human Feed-forward Vision in Invariant Object Recognition , 2015, Scientific Reports.

[29]  Shiguang Shan,et al.  Deeply Learning Deformable Facial Action Parts Model for Dynamic Expression Analysis , 2014, ACCV.

[30]  P. Ekman,et al.  Facial action coding system , 2019 .

[31]  Emad Barsoum,et al.  Training deep networks for facial expression recognition with crowd-sourced label distribution , 2016, ICMI.

[32]  Yurong Chen,et al.  Capturing AU-Aware Facial Features and Their Latent Relations for Emotion Recognition in the Wild , 2015, ICMI.

[33]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[34]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[35]  Karine Sergerie,et al.  The role of the amygdala in emotional processing: A quantitative meta-analysis of functional neuroimaging studies , 2008, Neuroscience & Biobehavioral Reviews.

[36]  S.V. Dudul,et al.  Emotion recognition from facial expression using neural networks , 2008, 2008 Conference on Human System Interactions.

[37]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Abhang Priyanka,et al.  Emotion Recognition using Speech and EEG Signal - A Review , 2011 .

[39]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).