Facial Expression Recognition Under Partial Occlusion from Virtual Reality Headsets based on Transfer Learning

Facial expressions of emotion are a major channel in our daily communications, and it has been subject of intense research in recent years. To automatically infer facial expressions, convolutional neural network based approaches has become widely adopted due to their proven applicability to Facial Expression Recognition (FER) task.On the other hand Virtual Reality (VR) has gained popularity as an immersive multimedia platform, where FER can provide enriched media experiences. However, recognizing facial expression while wearing a head-mounted VR headset is a challenging task due to the upper half of the face being completely occluded. In this paper we attempt to overcome these issues and focus on facial expression recognition in presence of a severe occlusion where the user is wearing a head-mounted display in a VR setting. We propose a geometric model to simulate occlusion resulting from a Samsung Gear VR headset that can be applied to existing FER datasets. Then, we adopt a transfer learning approach, starting from two pretrained networks, namely VGG and ResNet. We further fine-tune the networks on FER+ and RAF-DB datasets. Experimental results show that our approach achieves comparable results to existing methods while training on three modified benchmark datasets that adhere to realistic occlusion resulting from wearing a commodity VR headset. Code for this paper is available at: https://github.com/bita-github/MRP-FER

[1]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[2]  Hélio Pedrini,et al.  Emotion Recognition from Occluded Facial Expressions Using Weber Local Descriptor , 2018, 2018 25th International Conference on Systems, Signals and Image Processing (IWSSIP).

[3]  Emad Barsoum,et al.  Training deep networks for facial expression recognition with crowd-sourced label distribution , 2016, ICMI.

[4]  Wei Liu,et al.  Occlusion Robust Face Recognition Based on Mask Learning With Pairwise Differential Siamese Network , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[5]  Bin Xia,et al.  Occluded Facial Expression Recognition Enhanced through Privileged Information , 2019, ACM Multimedia.

[6]  Rainer Stiefelhagen,et al.  Why Is Facial Occlusion a Challenging Problem? , 2009, ICB.

[7]  Mohammad H. Mahoor,et al.  AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild , 2017, IEEE Transactions on Affective Computing.

[8]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Shan Li,et al.  Real world expression recognition: A highly imbalanced detection problem , 2016, 2016 International Conference on Biometrics (ICB).

[10]  Li Ma,et al.  Facial expression recognition based on transfer learning from deep convolutional networks , 2015, 2015 11th International Conference on Natural Computation (ICNC).

[11]  Vinod Chandran,et al.  Random Gabor based templates for facial expression recognition in images with facial occlusion , 2014, Neurocomputing.

[12]  Roger Zimmermann,et al.  Facefetch: An Efficient and Scalable Face Retrieval System That uses Your Visual Memory , 2019, 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM).

[13]  Shiguang Shan,et al.  Patch-Gated CNN for Occlusion-aware Facial Expression Recognition , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[14]  Verónica Orvalho,et al.  Assessing Facial Expressions in Virtual Reality Environments , 2016, VISIGRAPP.

[15]  Li Shang,et al.  Blended Emotion in-the-Wild: Multi-label Facial Expression Recognition Using Crowdsourced Annotations and Deep Locality Feature Learning , 2018, International Journal of Computer Vision.

[16]  Radu Tudor Ionescu,et al.  Recognizing Facial Expressions of Occluded Faces using Convolutional Neural Networks , 2019, ICONIP.

[17]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Vinod Chandran,et al.  Facial Expression Analysis under Partial Occlusion , 2018, ACM Comput. Surv..

[19]  Irfan A. Essa,et al.  Eyemotion: Classifying Facial Expressions in VR Using Eye-Tracking Cameras , 2017, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[20]  Stefanos Zafeiriou,et al.  300 Faces In-The-Wild Challenge: database and results , 2016, Image Vis. Comput..

[21]  Mel Slater,et al.  Reconstruction and Recognition of Occluded Facial Expressions Using PCA , 2007, ACII.

[22]  Min Hu,et al.  Facial expression recognition under partial occlusion based on fusion of global and local features , 2018, International Conference on Graphic and Image Processing.