Your Face Mirrors Your Deepest Beliefs-Predicting Personality and Morals through Facial Emotion Recognition

Can we really “read the mind in the eyes”? Moreover, can AI assist us in this task? This paper answers these two questions by introducing a machine learning system that predicts personality characteristics of individuals on the basis of their face. It does so by tracking the emotional response of the individual’s face through facial emotion recognition (FER) while watching a series of 15 short videos of different genres. To calibrate the system, we invited 85 people to watch the videos, while their emotional responses were analyzed through their facial expression. At the same time, these individuals also took four well-validated surveys of personality characteristics and moral values: the revised NEO FFI personality inventory, the Haidt moral foundations test, the Schwartz personal value system, and the domain-specific risk-taking scale (DOSPERT). We found that personality characteristics and moral values of an individual can be predicted through their emotional response to the videos as shown in their face, with an accuracy of up to 86% using gradient-boosted trees. We also found that different personality characteristics are better predicted by different videos, in other words, there is no single video that will provide accurate predictions for all personality characteristics, but it is the response to the mix of different videos that allows for accurate prediction.

[1]  Nitesh V. Chawla,et al.  SMOTE: Synthetic Minority Over-sampling Technique , 2002, J. Artif. Intell. Res..

[2]  Scott M. Lundberg,et al.  Consistent feature attribution for tree ensembles , 2017, ArXiv.

[3]  L. F. Barrett.  How Emotions Are Made: The Secret Life of the Brain , 2017 .

[4]  S. Schwartz,et al.  Toward A Universal Psychological Structure of Human Values , 1987 .

[5]  M. Kosinski,et al.  Computer-based personality judgments are more accurate than those made by humans , 2015, Proceedings of the National Academy of Sciences.

[6]  J. Tangney,et al.  Moral affect: the good, the bad, and the ugly. , 1991, Journal of personality and social psychology.

[7]  Michael P. Haselhuhn,et al.  Men’s Facial Width-to-Height Ratio Predicts Aggression: A Meta-Analysis , 2015, PloS one.

[8]  P. Ekman.  Facial expression and emotion. , 1993, The American psychologist.

[9]  Ain Simpson.  Moral Foundations Theory , 2020, Encyclopedia of Personality and Individual Differences.

[10]  Peter A. Gloor,et al.  Reducing Videoconferencing Fatigue through Facial Emotion Recognition , 2021, Future Internet.

[11]  Cesare Lombroso,et al.  Criminal Man, According to the Classification of Cesare Lombroso , 1972 .

[12]  Wanmin Wu,et al.  Quality of experience evaluation of voice communication: an affect-based approach , 2011, Human-centric Computing and Information Sciences.

[13]  T. Denson,et al.  Group Facial Width-to-Height Ratio Predicts Intergroup Negotiation Outcomes , 2018, Front. Psychol..

[14]  Karen L. Blair,et al.  What do two men kissing and a bucket of maggots have in common? Heterosexual men’s indistinguishable salivary α-amylase responses to photos of two men kissing and disgusting images , 2017 .

[15]  ByoungChul Ko,et al.  A Brief Review of Facial Emotion Recognition Based on Visual Information , 2018, Sensors.

[16]  J. Prinz,et al.  The emotional basis of moral judgments , 2006 .

[17]  J. Tangney,et al.  Moral emotions and moral behavior. , 2007, Annual review of psychology.

[18]  Gregory J. Boyle,et al.  The SAGE handbook of personality theory and assessment: Volume 2 — Personality measurement and testing , 2008 .

[19]  Carlos Guestrin,et al.  XGBoost : Reliable Large-scale Tree Boosting System , 2015 .

[20]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  R. O’Hara,et al.  Facial morphology predicts male fitness and rank but not survival in Second World War Finnish soldiers , 2013, Biology Letters.

[22]  S. Baron-Cohen,et al.  The "Reading the Mind in the Eyes" Test revised version: a study with normal adults, and adults with Asperger syndrome or high-functioning autism. , 2001, Journal of child psychology and psychiatry, and allied disciplines.

[23]  S. Schwartz,et al.  Bringing Values Back In The Adequacy of the European Social Survey to Measure Values in 20 Countries , 2008 .

[24]  E. Weber,et al.  A Domain-Specific Risk-Taking (DOSPERT) Scale for Adult Populations , 2006, Judgment and Decision Making.

[25]  Haibo He,et al.  ADASYN: Adaptive synthetic sampling approach for imbalanced learning , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[26]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[27]  Xiu-Shen Wei,et al.  Deep Bimodal Regression of Apparent Personality Traits from Short Video Sequences , 2018, IEEE Transactions on Affective Computing.

[28]  J. Ward,et al.  Increased facial width-to-height ratio and perceived dominance in the faces of the UK's leading business leaders. , 2014, British journal of psychology.

[29]  Sergio Escalera,et al.  ChaLearn LAP 2016: First Round Challenge on First Impressions - Dataset and Results , 2016, ECCV Workshops.

[30]  P. Costa,et al.  The revised NEO personality inventory (NEO-PI-R) , 2008 .

[31]  Scott Lundberg,et al.  A Unified Approach to Interpreting Model Predictions , 2017, NIPS.

[32]  F. Galton.  Composite Portraits, Made by Combining Those of Many Different Persons Into a Single Resultant Figure. , 1879 .

[33]  Sebastian Möller,et al.  Estimation of the Quality of Experience During Video Streaming From Facial Expression and Gaze Direction , 2020, IEEE Transactions on Network and Service Management.

[34]  S. Schwartz.  Universals in the Content and Structure of Values: Theoretical Advances and Empirical Tests in 20 Countries , 1992 .

[35]  Kathleen Taylor.  Disgust is a factor in extreme prejudice. , 2007, The British journal of social psychology.

[36]  Sergio Escalera,et al.  ChaLearn Looking at People Challenge 2014: Dataset and Results , 2014, ECCV Workshops.

[37]  C. Hjortsjö.  Man's face and mimic language , 1969 .

[38]  Lamine Amour,et al.  An improved QoE estimation method based on QoS and affective computing , 2018, 2018 International Symposium on Programming and Systems (ISPS).