A Survey on Databases of Facial Macro-expression and Micro-expression

A crucial step for developing and testing a system of facial expression analysis is to choose the database which suits best the targeted context application. We propose in this paper a survey based on the review of 69 databases, taking into account both macro- and micro-expressions. To the best of our knowledge, there are no other surveys with so many databases. We review the existing facial expression databases according to 18 characteristics grouped in 6 categories (population, modalities, data acquisition hardware, experimental conditions, experimental protocol and annotations). These characteristics are meant to be helpful for researchers when they are choosing a database which suits their context application. We bring to light the trends between posed, spontaneous and in-the-wild databases, as well as micro-expression databases. We finish with future directions, including crowd sourcing and databases with groups of people.

[1]  Matti Pietikäinen,et al.  A Spontaneous Micro-expression Database: Inducement, collection and baseline , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[2]  P. Ekman,et al.  Nonverbal Leakage and Clues to Deception †. , 1969, Psychiatry.

[3]  Jeffrey F. Cohn,et al.  Sayette Group Formation Task (GFT) Spontaneous Facial Expression Database , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[4]  Shrikanth S. Narayanan,et al.  The Vera am Mittag German audio-visual emotional speech database , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[5]  Takeo Kanade,et al.  Multi-PIE , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[6]  Maja Pantic,et al.  Web-based database for facial expression analysis , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[7]  P. Ekman,et al.  Facial action coding system , 2019 .

[8]  Renaud Séguier,et al.  A Survey on Databases for Facial Expression Analysis , 2018, VISIGRAPP.

[9]  Paul Ekman,et al.  Lie Catching and Microexpressions , 2009 .

[10]  Maja Pantic,et al.  AFEW-VA database for valence and arousal estimation in-the-wild , 2017, Image Vis. Comput..

[11]  Michel Valstar,et al.  Advances, Challenges, and Opportunities in Automatic Facial Expression Recognition , 2016 .

[12]  Mohammad H. Mahoor,et al.  DISFA: A Spontaneous Facial Action Intensity Database , 2013, IEEE Transactions on Affective Computing.

[13]  Yuichi Ohta,et al.  Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor , 2009, ICDP.

[14]  Tamás D. Gedeon,et al.  Automatic Group Happiness Intensity Analysis , 2015, IEEE Transactions on Affective Computing.

[15]  Peter Bull,et al.  Detecting Deception from Emotional and Unemotional Cues , 2009 .

[16]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[17]  Alessandro Vinciarelli,et al.  Canal9: A database of political debates for analysis of social interactions , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[18]  Luc Van Gool,et al.  A 3-D Audio-Visual Corpus of Affective Communication , 2010, IEEE Transactions on Multimedia.

[19]  Daniel McDuff,et al.  Affectiva-MIT Facial Expression Dataset (AM-FED): Naturalistic and Spontaneous Facial Expressions Collected "In-the-Wild" , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[20]  Jean-Claude Martin,et al.  Collection and Annotation of a Corpus of Human-Human Multimodal Interactions: Emotion and Others Anthropomorphic Characteristics , 2007, ACII.

[21]  Rui Zheng,et al.  A Parametric Survey for Facial Expression Database , 2012, BICS.

[22]  D. Lundqvist,et al.  Karolinska Directed Emotional Faces , 2015 .

[23]  Hatice Gunes,et al.  How to distinguish posed from spontaneous smiles using geometric features , 2007, ICMI '07.

[24]  Moi Hoon Yap,et al.  Objective Classes for Micro-Facial Expression Recognition , 2017, J. Imaging.

[25]  Hatice Gunes,et al.  A Bimodal Face and Body Gesture Database for Automatic Analysis of Human Nonverbal Affective Behavior , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[26]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[27]  L. H. Viet,et al.  Emotion Detection in the Loop from Brain Signals and Facial Images , 2006 .

[28]  Guoying Zhao,et al.  Facial Affect “In-the-Wild”: A Survey and a New Database , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[29]  Shaun J. Canavan,et al.  BP4D-Spontaneous: a high-resolution spontaneous 3D dynamic facial expression database , 2014, Image Vis. Comput..

[30]  John See,et al.  Facial Micro-Expressions Grand Challenge 2018 Summary , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[31]  Adrian Hilton,et al.  A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling , 2011, 2011 International Conference on Computer Vision.

[32]  Lijun Yin,et al.  A high-resolution 3D dynamic facial expression database , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[33]  Natalie C. Ebner,et al.  FACES—A database of facial expressions in young, middle-aged, and older women and men: Development and validation , 2010, Behavior research methods.

[34]  Mohammad H. Mahoor,et al.  Extended DISFA Dataset: Investigating Posed and Spontaneous Facial Expressions , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[35]  Tamás D. Gedeon,et al.  Collecting Large, Richly Annotated Facial-Expression Databases from Movies , 2012, IEEE MultiMedia.

[36]  Fei Chen,et al.  A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference , 2010, IEEE Transactions on Multimedia.

[37]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[38]  J. Cohn,et al.  Movement Differences between Deliberate and Spontaneous Facial Expressions: Zygomaticus Major Action in Smiling , 2006, Journal of nonverbal behavior.

[39]  Dirk Heylen,et al.  The Sensitive Artificial Listner: an induction technique for generating emotionally coloured conversation , 2008 .

[40]  Cigdem Eroglu Erdem,et al.  BAUM-1: A Spontaneous Audio-Visual Face Database of Affective and Mental States , 2017, IEEE Transactions on Affective Computing.

[41]  Skyler T. Hawk,et al.  Moving faces, looking places: validation of the Amsterdam Dynamic Facial Expression Set (ADFES). , 2011, Emotion.

[42]  Jeffrey F. Cohn,et al.  Dynamics of facial expression: normative characteristics and individual differences , 2001, IEEE International Conference on Multimedia and Expo, 2001. ICME 2001..

[43]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[44]  Jeffrey F. Cohn,et al.  Painful data: The UNBC-McMaster shoulder pain expression archive database , 2011, Face and Gesture 2011.

[45]  Jeffrey F. Cohn,et al.  The Timing of Facial Motion in posed and Spontaneous Smiles , 2003, Int. J. Wavelets Multiresolution Inf. Process..

[46]  Margaret McRorie,et al.  The Belfast Induced Natural Emotion Database , 2012, IEEE Transactions on Affective Computing.

[47]  Alice J. O'Toole,et al.  A video database of moving faces and people , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Qi Wu,et al.  CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[49]  Peter Robinson,et al.  3D Corpus of Spontaneous Complex Mental States , 2011, ACII.

[50]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[51]  Roddy Cowie,et al.  Beyond emotion archetypes: Databases for emotion modelling using neural networks , 2005, Neural Networks.

[52]  Skyler T. Hawk,et al.  Presentation and validation of the Radboud Faces Database , 2010 .

[53]  Ayoub Al-Hamadi,et al.  “BioVid Emo DB”: A multimodal database for emotion analyses validated by subjective ratings , 2016, 2016 IEEE Symposium Series on Computational Intelligence (SSCI).

[54]  Daniel McDuff,et al.  AM-FED+: An Extended Dataset of Naturalistic Facial Expressions Collected in Everyday Settings , 2019, IEEE Transactions on Affective Computing.

[55]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[56]  Fabien Ringeval,et al.  Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[57]  Maja Pantic,et al.  The SEMAINE corpus of emotionally coloured character interactions , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[58]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[59]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[60]  Edward Kim,et al.  Vinereactor: Crowdsourced Spontaneous Facial Expression Data , 2016, ICMR.

[61]  J. Russell,et al.  A Description of the Affective Quality Attributed to Environments , 1980 .

[62]  R. Cowie,et al.  A new emotion database: considerations, sources and scope , 2000 .

[63]  Dmitry B. Goldgof,et al.  Macro- and micro-expression spotting in long videos using spatio-temporal strain , 2011, Face and Gesture 2011.

[64]  Arman Savran,et al.  Bosphorus Database for 3D Face Analysis , 2008, BIOID.

[65]  Tamás D. Gedeon,et al.  Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[66]  H. Bülthoff,et al.  The MPI Facial Expression Database — A Validated Database of Emotional and Conversational Facial Expressions , 2012, PloS one.

[67]  Guoying Zhao,et al.  CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation , 2014, PloS one.

[68]  Xiaolan Fu,et al.  CAS(ME)$^2$ : A Database for Spontaneous Macro-Expression and Micro-Expression Spotting and Recognition , 2018, IEEE Transactions on Affective Computing.

[69]  Michael J. Black,et al.  Recognizing Facial Expressions in Image Sequences Using Local Parameterized Models of Image Motion , 1997, International Journal of Computer Vision.

[70]  Carlos Busso,et al.  IEMOCAP: interactive emotional dyadic motion capture database , 2008, Lang. Resour. Evaluation.

[71]  Nadine Mandran,et al.  DynEmo: A video database of natural facial expressions of emotions. , 2013 .

[72]  Nicholas Costen,et al.  SAMM: A Spontaneous Micro-Facial Movement Dataset , 2018, IEEE Transactions on Affective Computing.

[73]  Ya Li,et al.  CHEAVD: a Chinese natural emotional audio–visual database , 2016, Journal of Ambient Intelligence and Humanized Computing.

[74]  Björn W. Schuller,et al.  AVEC 2013: the continuous audio/visual emotion and depression recognition challenge , 2013, AVEC@ACM Multimedia.

[75]  Lina I Skora,et al.  A Review of Dynamic Datasets for Facial Expression Research , 2017 .

[76]  Anastasios Delopoulos,et al.  The MUG facial expression database , 2010, 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10.

[77]  Paul E. Debevec,et al.  Effect of illumination on automatic expression recognition: A novel 3D relightable facial database , 2011, Face and Gesture 2011.