PersEmoN: A Deep Network for Joint Analysis of Apparent Personality, Emotion and Their Relationship

Apparent personality and emotion analysis are both central to affective computing. Existing works solve them individually. In this paper we investigate if such high-level affect traits and their relationship can be jointly learned from face images in the wild. To this end, we introduce PersEmoN, an end-to-end trainable and deep Siamese-like network. It consists of two convolutional network branches, one for emotion and the other for apparent personality. Both networks share their bottom feature extraction module and are optimized within a multi-task learning framework. Emotion and personality networks are dedicated to their own annotated dataset. Furthermore, an adversarial-like loss function is employed to promote representation coherence among heterogeneous dataset sources. Based on this, we also explore the emotion-to-apparent-personality relationship. Extensive experiments demonstrate the effectiveness of PersEmoN.

[1]  Marianne Winslett,et al.  Give Me One Portrait Image, I Will Tell You Your Emotion and Personality , 2018, ACM Multimedia.

[2]  Xiaogang Wang,et al.  DeepID3: Face Recognition with Very Deep Neural Networks , 2015, ArXiv.

[3]  David Masip,et al.  Interpreting CNN Models for Apparent Personality Trait Regression , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[4]  Angelo Cangelosi,et al.  Emotion recognition in the wild using deep neural networks and Bayesian classifiers , 2017, ICMI.

[5]  Sergio Escalera,et al.  First Impressions: A Survey on Vision-Based Apparent Personality Trait Analysis , 2018, IEEE Transactions on Affective Computing.

[6]  Mohammad H. Mahoor,et al.  AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild , 2017, IEEE Transactions on Affective Computing.

[7]  Hatice Gunes,et al.  Automatic Prediction of Impressions in Time and across Varying Context: Personality, Attractiveness and Likeability , 2017, IEEE Transactions on Affective Computing.

[8]  Xiu-Shen Wei,et al.  Deep Bimodal Regression for Apparent Personality Analysis , 2016, ECCV Workshops.

[9]  Sergio Escalera,et al.  First Impressions: A Survey on Computer Vision-Based Apparent Personality Trait Analysis , 2018, ArXiv.

[10]  Xiaogang Wang,et al.  Deep Learning Face Representation from Predicting 10,000 Classes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Janine Willis,et al.  First Impressions , 2006, Psychological science.

[12]  Guoying Zhao,et al.  Aff-Wild: Valence and Arousal ‘In-the-Wild’ Challenge , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[13]  J. Russell.  A circumplex model of affect. , 1980 .

[14]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[15]  Le Zhang,et al.  A Deep Network for Arousal-Valence Emotion Prediction with Acoustic-Visual Cues , 2018, ArXiv.

[16]  S. Gosling,et al.  Personality and Social Psychology Bulletin Personality Judgments Based on Physical Appearance Personality Judgments Based on Physical Appearance , 2022 .

[17]  Trevor Darrell,et al.  Simultaneous Deep Transfer Across Domains and Tasks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[18]  Weisi Lin,et al.  Do Others Perceive You As You Want Them To?: Modeling Personality based on Selfies , 2015, ASM@ACM Multimedia.

[19]  Shuicheng Yan,et al.  Estimation of Affective Level in the Wild with Multiple Memory Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[20]  H. Eysenck.  Dimensions of Personality , 1947 .

[21]  Honglak Lee,et al.  Deep learning for robust feature generation in audiovisual emotion recognition , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[22]  Alessandro Vinciarelli,et al.  A Survey of Personality Computing , 2014, IEEE Transactions on Affective Computing.

[23]  Nicu Sebe,et al.  AMIGOS: A dataset for Mood, personality and affect research on Individuals and GrOupS , 2017, ArXiv.

[24]  Yu Qiao,et al.  Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks , 2016, IEEE Signal Processing Letters.

[25]  Yann LeCun,et al.  Signature Verification Using A "Siamese" Time Delay Neural Network , 1993, Int. J. Pattern Recognit. Artif. Intell..

[26]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[27]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Marcel van Gerven,et al.  Deep Impression: Audiovisual Deep Residual Networks for Multimodal Apparent Personality Trait Recognition , 2016, ECCV Workshops.

[29]  Sergio Escalera,et al.  ChaLearn Joint Contest on Multimedia Challenges Beyond Visual Analysis: An overview , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[30]  Trevor Darrell,et al.  Simultaneous Deep Transfer Across Domains and Tasks , 2015, ICCV.

[31]  Mihai Gavrilescu,et al.  Predicting the Sixteen Personality Factors (16PF) of an individual by analyzing facial features , 2017, EURASIP J. Image Video Process..

[32]  Luc Van Gool,et al.  Temporal Segment Networks: Towards Good Practices for Deep Action Recognition , 2016, ECCV.

[33]  R. Depue,et al.  Neurobiology of the structure of personality: Dopamine, facilitation of incentive motivation, and extraversion , 1999, Behavioral and Brain Sciences.

[34]  Mohammad H. Mahoor,et al.  Facial Affect Estimation in the Wild Using Deep Residual and Convolutional Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[35]  K. Scherer,et al.  Personality and emotion , 2009 .

[36]  Sergio Escalera,et al.  Multimodal First Impression Analysis with Deep Residual Networks , 2018, IEEE Transactions on Affective Computing.

[37]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[38]  Albert Ali Salah,et al.  Combining Deep Facial and Ambient Features for First Impression Estimation , 2016, ECCV Workshops.

[39]  James A. Russell,et al.  Predicting the Big Two of Affect from the Big Five of Personality , 2001 .

[40]  Stefan Winkler,et al.  ASCERTAIN: Emotion and Personality Recognition Using Commercial Sensors , 2018, IEEE Transactions on Affective Computing.

[41]  Sethuraman Panchanathan,et al.  Multimodal emotion recognition using deep learning architectures , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[42]  Stéphane Ayache,et al.  Explaining First Impressions: Modeling, Recognizing, and Explaining Apparent Personality from Videos , 2018, ArXiv.

[43]  Anurag Mittal,et al.  Bi-modal First Impressions Recognition Using Temporally Ordered Deep Audio and Stochastic Visual Features , 2016, ECCV Workshops.

[44]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[45]  Sergio Escalera,et al.  ChaLearn LAP 2016: First Round Challenge on First Impressions - Dataset and Results , 2016, ECCV Workshops.

[46]  Bhiksha Raj,et al.  SphereFace: Deep Hypersphere Embedding for Face Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  J. Pennebaker,et al.  LEXICAL PREDICTORS OFPERSONALITY TYPE , 2005 .

[48]  Subramanian Ramanathan,et al.  SALSA: A Novel Dataset for Multimodal Group Behavior Analysis , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Xiaogang Wang,et al.  Deep Learning Face Representation by Joint Identification-Verification , 2014, NIPS.

[50]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[51]  Wei-Yi Chang,et al.  FATAUVA-Net: An Integrated Deep Learning Framework for Facial Attribute Recognition, Action Unit Detection, and Valence-Arousal Estimation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[52]  Aleix M. Martínez,et al.  EmotioNet: An Accurate, Real-Time Algorithm for the Automatic Annotation of a Million Facial Expressions in the Wild , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[54]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).