Multi-tasking deep convolutional network architecture design for extracting nonverbal communicative information from a face

Abstract Facial expressions convey not only emotions but also communicative information. Therefore, facial expressions should be analysed to understand communication. The objective of this study is to develop an automatic facial expression analysis system for extracting nonverbal communicative information. This study focuses on specific communicative information: emotions expressed through facial movements and the direction of the expressions. We propose a multi-tasking deep convolutional network (DCN) to classify facial expressions, detect the facial regions, and estimate face angles. We reformulate facial region detection and face angle estimation as regression problems and add task-specific output layers in the DCN’s architecture. Experimental results show that the proposed method performs all tasks accurately. In this study, we show the feasibility of the multi-tasking DCN for extracting nonverbal communicative information from a human face.

[1]  Qirong Mao,et al.  Cascaded Multi-level Transformed Dirichlet Process for Multi-pose Facial Expression Recognition , 2018, Comput. J..

[2]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[3]  Tong Zhang,et al.  A Deep Neural Network-Driven Feature Learning Method for Multi-view Facial Expression Recognition , 2016, IEEE Transactions on Multimedia.

[4]  K. Scherer,et al.  Interaction effects of perceived gaze direction and dynamic facial expression: Evidence for appraisal theories of emotion , 2007 .

[5]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  B. Parkinson,et al.  Personality and Social Psychology Review Do Facial Movements Express Emotions or Communicate Motives? Personality and Social Psychology Review Additional Services and Information for Do Facial Movements Express Emotions or Communicate Motives? Emotion Expression vs. Motive Communication Emotion Expr , 2022 .

[7]  Luc Van Gool,et al.  Random Forests for Real Time 3D Face Analysis , 2012, International Journal of Computer Vision.

[8]  Yongzhao Zhan,et al.  Multi-pose facial expression recognition based on SURF boosting , 2015, 2015 International Conference on Affective Computing and Intelligent Interaction (ACII).

[9]  P. Ekman,et al.  Facial Expressions of Emotion , 1979 .

[10]  Shiguang Shan,et al.  AU-inspired Deep Networks for Facial Expression Feature Learning , 2015, Neurocomputing.

[11]  C. Frith.  Role of facial expressions in social interactions , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[12]  Kostas Karpouzis,et al.  Emotion recognition through facial expression analysis based on a neurofuzzy network , 2005, Neural Networks.

[13]  Yu Qiao,et al.  Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks , 2016, IEEE Signal Processing Letters.

[14]  Haitao Zhao,et al.  A visual attention based ROI detection method for facial expression recognition , 2018, Neurocomputing.

[15]  Skyler T. Hawk,et al.  Presentation and validation of the Radboud Faces Database , 2010 .

[16]  R. K. Kulkarni,et al.  Face detection and facial expression recognition system , 2014, 2014 International Conference on Electronics and Communication Systems (ICECS).

[17]  R. Frank.  Passions Within Reason: The Strategic Role of the Emotions , 1990 .

[18]  Rachael E. Jack.  Understanding the face as a dynamic communication tool , 2015 .

[19]  Carlos D. Castillo,et al.  An All-In-One Convolutional Neural Network for Face Analysis , 2016, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[20]  A. Tapus,et al.  Children with Autism Social Engagement in Interaction with Nao, an Imitative Robot - A Series of Single Case Experiments , 2012 .

[21]  Takeo Kanade,et al.  Multi-PIE , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[22]  Aurobinda Routray,et al.  Automatic facial expression recognition using features of salient facial patches , 2015, IEEE Transactions on Affective Computing.

[23]  Karim Faez,et al.  Pose-Invariant Facial Expression Recognition Based on 3D Face Reconstruction and Synthesis from a Single 2D Image , 2014, 2014 22nd International Conference on Pattern Recognition.

[24]  Beat Fasel,et al.  Automati Fa ial Expression Analysis: A Survey , 1999 .

[25]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[26]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[27]  Jun Wang,et al.  3D Facial Expression Recognition Based on Primitive Surface Feature Distribution , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[28]  Shikha Tripathi,et al.  Pose invariant method for emotion recognition from 3D images , 2015, 2015 Annual IEEE India Conference (INDICON).

[29]  Mohammad H. Mahoor,et al.  Going deeper in facial expression recognition using deep neural networks , 2015, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[30]  Claude E. Shannon,et al.  The mathematical theory of communication , 1950 .

[31]  C. Darwin.  The Expression of the Emotions in Man and Animals , .

[32]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[33]  Aude Billard,et al.  Robotic assistants in therapy and education of children with autism: can a small humanoid robot help encourage social interaction skills? , 2005, Universal Access in the Information Society.

[34]  Nilanjan Sarkar,et al.  Emotion-sensitive robots - a new paradigm for human-robot interaction , 2004, 4th IEEE/RAS International Conference on Humanoid Robots, 2004..

[35]  Nicu Sebe,et al.  Facial expression recognition from video sequences: temporal and static modeling , 2003, Comput. Vis. Image Underst..

[36]  P. Ekman.  Universals and cultural differences in facial expressions of emotion. , 1972 .

[37]  T. Scott-Phillips.  Defining biological communication , 2008, Journal of evolutionary biology.

[38]  Payman Moallem,et al.  Facial emotion recognition method based on Pyramid Histogram of Oriented Gradient over three direction of head , 2014, 2014 4th International Conference on Computer and Knowledge Engineering (ICCKE).