Say CHEESE: Common Human Emotional Expression Set Encoder and Its Application to Analyze Deceptive Communication

In this paper we introduce the Common Human Emotional Expression Set Encoder (CHEESE) framework for objectively determining which, if any, subsets of the facial action units associated with smiling are well represented by a small finite set of clusters according to an information theoretic metric. Smile-related AUs (6,7,10,12,14) in over 1.3M frames of facial expressions from 151 pairs of individuals playing a communication game involving deception were analyzed with CHEESE. The combination of AU6 (cheek raiser) and AU12 (lip corner puller) are shown to cluster well into five different types of expression. Liars showed high intensity AU6 and AU12 more often compared to honest speakers. Additionally, interrogators were found to express a higher frequency of low intensity AU6 with high intensity AU12 (i.e. polite smiles) when they were being lied to, suggesting that deception analysis should be done in consideration of both the message sender's and the receiver's facial expressions.

[1]  Daniel McDuff,et al.  Exploring Temporal Patterns in Classifying Frustrated and Delighted Smiles , 2012, IEEE Trans. Affect. Comput..

[2]  Stephen Porter,et al.  Cry me a river: identifying the behavioral consequences of extremely high-stakes interpersonal deception. , 2012, Law and human behavior.

[3]  R. Buck,et al.  Communication of affect through facial expressions in humans. , 1972, Journal of personality and social psychology.

[4]  S. P. Lloyd,et al.  Least squares quantization in PCM , 1982, IEEE Trans. Inf. Theory.

[5]  Paul L. Rosin,et al.  Facial dynamics as indicators of trustworthiness and cooperative behavior. , 2007, Emotion.

[6]  Catherine C. Eckel,et al.  The Value of a Smile: Game Theory with a Human Face , 2001 .

[7]  Stephen Porter,et al.  Crocodile tears: facial, verbal and body language behaviours associated with genuine and fabricated remorse. , 2011, Law and human behavior.

[8]  Daniel McDuff,et al.  AFFDEX SDK: A Cross-Platform Real-Time Multi-Face Expression Recognition Toolkit , 2016, CHI Extended Abstracts.

[9]  Lisa Feldman Barrett,et al.  Perceptions of emotion from facial expressions are not culturally universal: evidence from a remote culture. , 2014, Emotion.

[10]  Mohamed Abouelenien,et al.  Deception Detection using Real-life Trial Data , 2015, ICMI.

[11]  Thomas Hugh Feeley.  Exploring sanctioned and unsanctioned lies in interpersonal deception , 1996 .

[12]  D. Keltner.  Signs of appeasement: evidence for the distinct displays of embarrassment, amusement, and shame , 1995 .

[13]  T. Caliński,et al.  A dendrite method for cluster analysis , 1974 .

[14]  Peter Robinson,et al.  OpenFace: An open source facial behavior analysis toolkit , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[15]  P. Ekman,et al.  The Duchenne smile: emotional expression and brain physiology. II. , 1990, Journal of personality and social psychology.

[16]  Rachael E. Jack,et al.  Cultural Confusions Show that Facial Expressions Are Not Universal , 2009, Current Biology.

[17]  L. Fleischer.  Telling Lies Clues To Deceit In The Marketplace Politics And Marriage , 2016 .

[18]  G. W. Milligan,et al.  An examination of procedures for determining the number of clusters in a data set , 1985 .

[19]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[20]  Michael Lewis,et al.  Deception in 3-Year-Olds , 1989 .

[21]  Louis-Philippe Morency,et al.  Are You Friendly or Just Polite? - Analysis of Smiles in Spontaneous Face-to-Face Interactions , 2011, ACII.

[22]  J. Burgoon,et al.  Interpersonal Deception Theory , 2015 .

[23]  Vipin Kumar,et al.  The Challenges of Clustering High Dimensional Data , 2004 .

[24]  J. Cohn,et al.  Movement Differences between Deliberate and Spontaneous Facial Expressions: Zygomaticus Major Action in Smiling , 2006, Journal of nonverbal behavior.

[25]  P. Ekman,et al.  Nonverbal leakage and clues to deception. , 1969, Psychiatry.

[26]  Jonathan Gratch,et al.  Trust me: multimodal signals of trustworthiness , 2016, ICMI.

[27]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[28]  Md. Kamrul Hasan,et al.  Automated Dyadic Data Recorder (ADDR) Framework and Analysis of Facial Cues in Deceptive Communication , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[29]  Rosalind W. Picard,et al.  Acted vs. natural frustration and delight: Many people smile in natural frustration , 2011, Face and Gesture 2011.

[30]  Albert Ali Salah,et al.  Recognition of Genuine Smiles , 2015, IEEE Transactions on Multimedia.

[31]  Rui Li,et al.  Understanding Social Interpersonal Interaction via Synchronization Templates of Facial Events , 2018, AAAI.

[32]  C. Harris,et al.  Facial expressions, smile types, and self-report during humour, tickle, and pain , 2005 .

[33]  Gwen Littlewort,et al.  The computer expression recognition toolbox (CERT) , 2011, Face and Gesture 2011.

[34]  Mark A. deTurck,et al.  The Behavioral Correlates of Sanctioned and Unsanctioned Deceptive Communication , 1998 .

[35]  Olatz Arbelaitz,et al.  An extensive comparative study of cluster validity indices , 2013, Pattern Recognit..

[36]  A. Todorov,et al.  EvaluaTiNg faCES ON TruSTwOrThiNESS afTEr miNimal TimE ExpOSurE , 2009 .

[37]  J. Cohn,et al.  All Smiles are Not Created Equal: Morphology and Timing of Smiles Perceived as Amused, Polite, and Embarrassed/Nervous , 2009, Journal of nonverbal behavior.

[38]  P. Ekman,et al.  Felt, false, and miserable smiles , 1982 .

[39]  P. Ekman,et al.  Behavioral Markers and Recognizability of the Smile of Enjoyment , 1993 .