Recognition of Genuine Smiles

Automatic distinction between genuine (spontaneous) and posed expressions is important for visual analysis of social signals. In this paper, we describe an informative set of features for the analysis of face dynamics, and propose a completely automatic system to distinguish between genuine and posed enjoyment smiles. Our system incorporates facial landmarking and tracking, through which features are extracted to describe the dynamics of eyelid, cheek, and lip corner movements. By fusing features over different regions, as well as over different temporal phases of a smile, we obtain a very accurate smile classifier. We systematically investigate age and gender effects, and establish that age-specific classification significantly improves the results, even when the age is automatically estimated. We evaluate our system on the 400-subject UvA-NEMO database we have recently collected, as well as on three other smile databases from the literature . Through an extensive experimental evaluation, we show that our system improves the state of the art in smile classification and provides useful insights in smile psychophysics.

[1]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[2]  P. Ekman,et al.  Deliberate facial movement. , 1980 .

[3]  P. Ekman,et al.  The symmetry of emotional and deliberate facial actions. , 1981, Psychophysiology.

[4]  P. Ekman,et al.  Felt, false, and miserable smiles , 1982 .

[5]  E. Koff,et al.  Facial asymmetry in posed and spontaneous expressions of emotion , 1983, Brain and Cognition.

[6]  P. Ekman.  Telling lies: clues to deceit in the marketplace , 1985 .

[7]  Thomas W. Sederberg,et al.  Free-form deformation of solid geometric models , 1986, SIGGRAPH.

[8]  P. Ekman,et al.  Smiles when lying. , 1988, Journal of personality and social psychology.

[9]  P. Ekman,et al.  The Duchenne smile: emotional expression and brain physiology. II. , 1990, Journal of personality and social psychology.

[10]  G. Boulogne,et al.  The Mechanism of Human Facial Expression , 1990 .

[11]  L. Rubin.  The Mechanism of Human Facial Expression , 1992 .

[12]  C. Cacou.  Anthropometry of the head and face , 1995 .

[13]  A. Martínez,et al.  The AR face databasae , 1998 .

[14]  Thomas S. Huang,et al.  Explanation-based facial motion tracking using a piecewise Bezier volume deformation model , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[15]  Jiri Matas,et al.  XM2VTSDB: The Extended M2VTS Database , 1999 .

[16]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  J. Movellan,et al.  Are Your Eyes Smiling? Detecting genuine smiles with support vector machines and Gabor wavelets , 2001 .

[18]  Karen L. Schmidt,et al.  Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles , 2003, Biological Psychology.

[19]  P. Ekman.  Darwin, Deception, and Facial Expression , 2003, Annals of the New York Academy of Sciences.

[20]  Jing Xiao,et al.  Multimodal coordination of facial action, head rotation, and eye motion during spontaneous smiles , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[21]  Jeffrey F. Cohn,et al.  The Timing of Facial Motion in posed and Spontaneous Smiles , 2003, Int. J. Wavelets Multiresolution Inf. Process..

[22]  L. Kuncheva.  Fusion of Continuous‐Valued Outputs , 2004 .

[23]  Maja Pantic,et al.  Detecting facial actions and their temporal segments in nearly frontal-view face image sequences , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[24]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  J. Cohn,et al.  Movement Differences between Deliberate and Spontaneous Facial Expressions: Zygomaticus Major Action in Smiling , 2006, Journal of nonverbal behavior.

[26]  Maja Pantic,et al.  Spontaneous vs. posed facial behavior: automatic analysis of brow actions , 2006, ICMI '06.

[27]  Yunus Saatci,et al.  Cascaded classification of gender and facial expression using active appearance models , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[28]  Hatice Gunes,et al.  How to distinguish posed from spontaneous smiles using geometric features , 2007, ICMI '07.

[29]  M. Del Giudice,et al.  Differences between children and adults in the recognition of enjoyment smiles. , 2007, Developmental psychology.

[30]  Maja Pantic,et al.  Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[31]  Arman Savran,et al.  Bosphorus Database for 3D Face Analysis , 2008, BIOID.

[32]  J. Cohn,et al.  All Smiles are Not Created Equal: Morphology and Timing of Smiles Perceived as Amused, Polite, and Embarrassed/Nervous , 2009, Journal of nonverbal behavior.

[33]  A. Manstead,et al.  Can Duchenne smiles be feigned? New evidence on felt and false smiles. , 2009, Emotion.

[34]  Karen L. Schmidt,et al.  Comparison of Deliberate and Spontaneous Facial Movement in Smiles and Eyebrow Raises , 2009, Journal of nonverbal behavior.

[35]  Rama Chellappa,et al.  Computational methods for modeling facial aging: A survey , 2009, J. Vis. Lang. Comput..

[36]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  Yun Fu,et al.  Age Synthesis and Estimation via Faces: A Survey , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  P. Gosselin,et al.  The voluntary control of facial action units in adults. , 2010, Emotion.

[39]  Fei Chen,et al.  A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference , 2010, IEEE Transactions on Multimedia.

[40]  Albert Ali Salah,et al.  Eyes do not lie: spontaneous versus posed smiles , 2010, ACM Multimedia.

[41]  M. Pantic,et al.  Induced Disgust , Happiness and Surprise : an Addition to the MMI Facial Expression Database , 2010 .

[42]  Qingshan Liu,et al.  Segment and recognize expression phase by fusion of motion area and neutral divergence features , 2011, Face and Gesture 2011.

[43]  M. Del Giudice,et al.  Individual Differences in the Recognition of Enjoyment Smiles: No Role for Perceptual–Attentional Factors and Autistic-Like Traits , 2011, Front. Psychology.

[44]  Vinod Chandran,et al.  Geometry vs. Appearance for Discriminating between Posed and Spontaneous Emotions , 2011, ICONIP.

[45]  Danielle M Shore,et al.  The value of genuine and polite smiles. , 2011, Emotion.

[46]  Matti Pietikäinen,et al.  Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[47]  Albert Ali Salah,et al.  Are You Really Smiling at Me? Spontaneous versus Posed Enjoyment Smiles , 2012, ECCV.

[48]  Albert Ali Salah,et al.  A smile can reveal your age: enabling facial dynamics in age estimation , 2012, ACM Multimedia.

[49]  Maja Pantic,et al.  Fully Automatic Recognition of the Temporal Phases of Facial Actions , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[50]  Albert Ali Salah,et al.  A Statistical Method for 2-D Facial Landmarking , 2012, IEEE Transactions on Image Processing.

[51]  Xiaolong Wang,et al.  A study on human age estimation under facial expression changes , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[52]  Maja Pantic,et al.  The MAHNOB Laughter database , 2013, Image Vis. Comput..

[53]  Judith A. Hall,et al.  The Deliberate Duchenne Smile: Individual Differences in Expressive Control , 2013 .

[54]  Xiaoping Chen,et al.  Analyses of the Differences between Posed and Spontaneous Facial Expressions , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[55]  Xin Li,et al.  Facial Expression Recognition Influenced by Human Aging , 2013, IEEE Transactions on Affective Computing.

[56]  Chao Zhang,et al.  Age estimation with expression changes using multiple aging subspaces , 2013, 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS).

[57]  L. Fleischer.  Telling Lies Clues To Deceit In The Marketplace Politics And Marriage , 2016 .