Synthesis of Facial Expressions in Photographs

The synthesis of facial expressions has applicationsin areas such as interactive games, biometrics systems, and training of people with disorders, among others. Although this is an area relatively well explored in the literature, there are no recent studies proposing to systematize an overview of research in the area. This systematic review analyzes the approaches to the synthesis of facial expressions in photographs, as well as important aspects of the synthesis process, such as preprocessing techniques, databases, and evaluation metrics. Forty-eight studies from three different scientific databases were analyzed. From these studies, we established an overview of the process, including all the stages used to synthesize expressions in facial images. We also analyze important aspects involved in these stages such as methods and techniques of each stage, databases, and evaluation metrics. We observed that machine learning approaches are the most widely used to synthesize expressions. Landmark identification, deformation, mapping, fusion, and training are common tasks considered in the approaches. We also found that few studies used metrics to evaluate the results, and most studies used public databases. Although the studies analyzed generated consistent and realistic results while preserving the identity of the subject, there are still research themes to be exploited.

[1]  Baining Guo,et al.  Geometry-driven photorealistic facial expression synthesis , 2003, IEEE Transactions on Visualization and Computer Graphics.

[2]  Jiri Matas,et al.  XM2VTSDB: The Extended M2VTS Database , 1999 .

[3]  P. Ekman,et al.  The role of context in interpreting facial expression: comment on Russell and Fehr (1987). , 1988, Journal of experimental psychology. General.

[4]  Beny Lafer,et al.  Facial emotion recognition in bipolar disorder: a critical review. , 2009, Revista brasileira de psiquiatria.

[5]  B. Abboud,et al.  Bilinear factorisation for facial expression analysis and synthesis , 2005 .

[6]  Peter Robinson,et al.  A facial affect mapping engine , 2014, IUI Companion '14.

[7]  Hassan Aghaeinia,et al.  Diverse videos synthesis using manifold-based parametric motion model for facial understanding , 2016, IET Image Process..

[8]  Josephine Sullivan,et al.  One millisecond face alignment with an ensemble of regression trees , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Nanning Zheng,et al.  Eye Synthesis Using the Eye Curve Model , 2007, 19th IEEE International Conference on Tools with Artificial Intelligence(ICTAI 2007).

[10]  Jun-yong Noh,et al.  Expression cloning , 2001, SIGGRAPH.

[11]  Yoshua Bengio,et al.  Challenges in representation learning: A report on three machine learning contests , 2013, Neural Networks.

[12]  Zhang Ting Facial Expression Transformation,Aging and Invisible View Reconstruction (2) , 2003 .

[13]  Anastasios Delopoulos,et al.  The MUG facial expression database , 2010, 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10.

[14]  LinLin Shen,et al.  A Novel Transient Wrinkle Detection Algorithm and Its Application for Expression Synthesis , 2017, IEEE Transactions on Multimedia.

[15]  Seyyed Ali Seyyedsalehi,et al.  Nonlinear analysis and synthesis of video images using deep dynamic bottleneck neural networks for face recognition , 2018, Neural Networks.

[16]  Jeffrey R. Spies,et al.  Effects of damping head movement and facial expression in dyadic conversation using real–time facial expression tracking and synthesized avatars , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[17]  Vinay Bettadapura,et al.  Face Expression Recognition and Analysis: The State of the Art , 2012, ArXiv.

[18]  Thomas Vetter,et al.  Expression invariant 3D face recognition with a Morphable Model , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[19]  E. Courchesne,et al.  The brain response to personally familiar faces in autism: findings of fusiform activity and beyond. , 2004, Brain : a journal of neurology.

[20]  John McDonald,et al.  Photo-realistic facial expression synthesis , 2005, Image Vis. Comput..

[21]  A.Z. Kouzani Facial expression synthesis , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[22]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[23]  Claes Wohlin,et al.  An Experimental Comparison of Usage-Based and Checklist-Based Reading , 2003, IEEE Trans. Software Eng..

[24]  Lianhong Cai,et al.  Facial expression synthesis based on motion patterns learned from face database , 2010, 2010 IEEE International Conference on Image Processing.

[25]  Fátima de Lourdes dos Santos Nunes,et al.  Generating Facial Emotions for Diagnosis and Training , 2015, 2015 IEEE 28th International Symposium on Computer-Based Medical Systems.

[26]  Heng Yang,et al.  Facial feature point detection: A comprehensive survey , 2014, Neurocomputing.

[27]  Justus Thies,et al.  Face2Face: real-time face capture and reenactment of RGB videos , 2019, Commun. ACM.

[28]  Bing Zhou,et al.  Facial expression cloning with elastic and muscle models , 2014, J. Vis. Commun. Image Represent..

[29]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[30]  Takeo Kanade,et al.  Multi-PIE , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[31]  LinLin Shen,et al.  Facial expression synthesis with direction field preservation based mesh deformation and lighting fitting based wrinkle mapping , 2018, Multimedia Tools and Applications.

[32]  Zicheng Liu,et al.  Expressive expression mapping with ratio images , 2001, SIGGRAPH.

[33]  Jong Min Lee,et al.  Automatic Synthesis of Realistic Facial Expressions , 2007, 2007 IEEE International Symposium on Signal Processing and Information Technology.

[34]  Yang Song,et al.  Age Progression/Regression by Conditional Adversarial Autoencoder , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Tomaso A. Poggio,et al.  Reanimating Faces in Images and Video , 2003, Comput. Graph. Forum.

[36]  Zhuo Feng Facial Expression Transformation,Aging and Invisible View Reconstruction (1) , 2003 .

[37]  Igor S. Pandzic,et al.  Facial animation framework for the web and mobile platforms , 2002, Web3D '02.

[38]  Yufang Cheng,et al.  3D Animated Facial Expression and Autism in Taiwan , 2008, 2008 Eighth IEEE International Conference on Advanced Learning Technologies.

[39]  Yiqiang Chen,et al.  SVR-based facial texture driving for realistic expression synthesis , 2004, Third International Conference on Image and Graphics (ICIG'04).

[40]  Jeong Ho Lee,et al.  Realistic expression mapping robust to various lighting conditions , 2009, 2009 Digest of Technical Papers International Conference on Consumer Electronics.

[41]  Xu Chao,et al.  Facial Expression Recognition and Synthesis on Affective Emotions Composition , 2008, 2008 International Seminar on Future BioMedical Information Engineering.

[42]  R Kikinis,et al.  Deformable modeling of facial tissue for craniofacial surgery simulation. , 1998, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[43]  Oliver G. B. Garrod,et al.  Four not six: Revealing culturally common facial expressions of emotion. , 2016, Journal of experimental psychology. General.

[44]  Dipti Prasad Mukherjee,et al.  Synthesis of emotional expressions specific to facial structure , 2012, ICVGIP '12.

[45]  Xiaofei He,et al.  Coupled Dictionary Learning for the Detail-Enhanced Synthesis of 3-D Facial Expressions , 2016, IEEE Transactions on Cybernetics.

[46]  Lei Huang,et al.  3D Facial Expression Synthesis from a Single Image Using a Model Set , 2012, ACCV Workshops.

[47]  Stephan F Taylor,et al.  Brain mapping biomarkers of socio-emotional processing in schizophrenia. , 2012, Schizophrenia bulletin.

[48]  P. Ekman,et al.  Measuring facial movement , 1976 .

[49]  Zhou Wang,et al.  No-reference perceptual quality assessment of JPEG compressed images , 2002, Proceedings. International Conference on Image Processing.

[50]  A. J. Fridlund Sociality of Solitary Smiling: Potentiation by an Implicit Audience , 1991 .

[51]  Wei Wei,et al.  Facial expression transfer method based on frequency analysis , 2016, Pattern Recognit..

[52]  Hans-Peter Seidel,et al.  A Generic Framework for Efficient 2-D and 3-D Facial Expression Analogy , 2007, IEEE Transactions on Multimedia.

[53]  Yen-Wei Chen,et al.  Two-step subspace learning for texture synthesis of facial images , 2012, 2012 6th International Conference on New Trends in Information Science, Service Science and Data Mining (ISSDM2012).

[54]  Xueyin Lin,et al.  Mapping Emotional Status to Facial Expressions , 2002, ICPR.

[55]  Ahmed M. Elgammal,et al.  Nonlinear Shape and Appearance Models for Facial Expression Analysis and Synthesis , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[56]  MengChu Zhou,et al.  Image Ratio Features for Facial Expression Recognition Application , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[57]  Björn W. Schuller,et al.  Efficient Recognition of Authentic Dynamic Facial Expressions on the Feedtum Database , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[58]  Jean-Claude Martin,et al.  Multimedia interfaces for users with high functioning autism: An empirical investigation , 2008, Int. J. Hum. Comput. Stud..

[59]  Feng Dong,et al.  Building highly realistic facial modeling and animation: a survey , 2007, The Visual Computer.

[60]  Saeed Rahati Quchani,et al.  A fast and accurate facial expression synthesis system for color face images using face graph and deep belief network , 2010, 2010 International Conference on Electronics and Information Engineering.

[61]  Irwin King,et al.  Facial expression synthesis by radial basis function network and image warping , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[62]  Jan Kautz,et al.  Visio-lization: generating novel facial images , 2009, ACM Trans. Graph..

[63]  M. Bartlett,et al.  Automatic Decoding of Facial Movements Reveals Deceptive Pain Expressions , 2014, Current Biology.

[64]  J. Russell Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. , 1994, Psychological bulletin.

[65]  Yihao Zhang,et al.  Facial expression mapping based on elastic and muscle-distribution-based models , 2012, 2012 IEEE International Symposium on Circuits and Systems.

[66]  Skyler T. Hawk,et al.  Presentation and validation of the Radboud Faces Database , 2010 .

[67]  A. J. Fridlund,et al.  Facial Displays Are Tools for Social Influence , 2018, Trends in Cognitive Sciences.

[68]  Hongxun Yao,et al.  Hierarchical semantic image matching using CNN feature pyramid , 2018, Comput. Vis. Image Underst..

[69]  Shan Li,et al.  Deep Facial Expression Recognition: A Survey , 2018, IEEE Transactions on Affective Computing.

[70]  Nanning Zheng,et al.  Extended facial expression synthesis using statistical appearance model , 2009, 2009 4th IEEE Conference on Industrial Electronics and Applications.

[71]  Matti Pietikäinen,et al.  Facial expression recognition from near-infrared videos , 2011, Image Vis. Comput..

[72]  Guoyin Wang,et al.  A Multi-step Backward Cloud Generator Algorithm , 2012, RSCTC.

[73]  P. Ekman,et al.  Emotion in the Human Face: Guidelines for Research and an Integration of Findings , 1972 .

[74]  Margaret McRorie,et al.  The Belfast Induced Natural Emotion Database , 2012, IEEE Transactions on Affective Computing.

[75]  Simon Lucey,et al.  Real-time avatar animation from a single image , 2011, Face and Gesture 2011.

[76]  Dongjian He,et al.  Facial Expression Synthesis Based on Expressional Feature Space and Expression Decomposition , 2008, 2008 International Conference on Computer Science and Software Engineering.

[77]  Maja Pantic,et al.  Web-based database for facial expression analysis , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[78]  Mikkel B. Stegmann,et al.  The IMM Face Database, An Annotated Dataset of 240 Face Images , 2004 .

[79]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[80]  Hao Chen,et al.  Incremental tensor by face synthesis estimating for face recognition , 2009, 2009 International Conference on Machine Learning and Cybernetics.

[81]  Qionghai Dai,et al.  A data-driven approach for facial expression synthesis in video , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[82]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[83]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[84]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[85]  Shinya Nakano,et al.  A natural smile synthesis from an artificial smile , 2009, SIGGRAPH '09.

[86]  William T. Freeman,et al.  Synthesizing Normalized Faces from Facial Identity Features , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[87]  Terence Sim,et al.  The CMU Pose, Illumination, and Expression Database , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[88]  Chun Chen,et al.  Subtle Facial Expression Modeling with Vector Field Decomposition , 2006, 2006 International Conference on Image Processing.

[89]  Amela Karahasanovic,et al.  A survey of controlled experiments in software engineering , 2005, IEEE Transactions on Software Engineering.

[90]  Peter Robinson,et al.  Constrained Local Neural Fields for Robust Facial Landmark Detection in the Wild , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[91]  Xueyin Lin,et al.  Facial expressional image synthesis controlled by emotional parameters , 2005, Pattern Recognit. Lett..

[92]  M. Petticrew,et al.  Systematic Reviews in the Social Sciences: A Practical Guide , 2005 .

[93]  Narendra Ahuja,et al.  Facial expression decomposition , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[94]  Shuliang Wang,et al.  Facial Expression Synthesis Based on Cloud Model , 2010, 2010 2nd International Workshop on Intelligent Systems and Applications.

[95]  Simon Lucey,et al.  Deformable Model Fitting by Regularized Landmark Mean-Shift , 2010, International Journal of Computer Vision.

[96]  Ahmad Y. Javaid,et al.  Facial Emotion Recognition: A Survey and Real-World User Experiences in Mixed Reality , 2018, Sensors.

[97]  S. Baron-Cohen,et al.  Systemizing empathy: Teaching adults with Asperger syndrome or high-functioning autism to recognize complex emotions using interactive multimedia , 2006, Development and Psychopathology.

[98]  Lianhong Cai,et al.  Facial Expression Synthesis Based on Emotion Dimensions for Affective Talking Avatar , 2010, Modeling Machine Emotions for Realizing Intelligence.

[99]  Franck Davoine,et al.  Appearance factorization based facial expression recognition and synthesis , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[100]  Jung-Woo Ha,et al.  StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[101]  Wijnand A. IJsselsteijn,et al.  How affective technologies can influence intimate interactions and improve social connectedness , 2014, Int. J. Hum. Comput. Stud..

[102]  Shigeo Morishima,et al.  Automatic generation of facial wrinkles according to expression changes , 2011, SA '11.

[103]  Zhou Wang,et al.  Modern Image Quality Assessment , 2006, Modern Image Quality Assessment.

[104]  Javier R. Movellan,et al.  Learning to Make Facial Expressions , 2009, 2009 IEEE 8th International Conference on Development and Learning.

[105]  Xueyin Lin,et al.  Emotional facial expression model building , 2003, Pattern Recognition Letters.

[106]  Ross Buck,et al.  The communication of emotion , 1984 .

[107]  Alex Martin,et al.  Facial Emotion Recognition in Autism Spectrum Disorders: A Review of Behavioral and Neuroimaging Studies , 2010, Neuropsychology Review.

[108]  N. Sarkar,et al.  Design of a Virtual Reality Based Adaptive Response Technology for Children With Autism , 2013, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[109]  Nanning Zheng,et al.  Facial Expression Sequence Synthesis Based on Shape and Texture Fusion Model , 2007, 2007 IEEE International Conference on Image Processing.

[110]  R. Mukundan,et al.  Facial expressions using a Quadratic Deformation Model: Analysis and synthesis , 2009, 2009 24th International Conference Image and Vision Computing New Zealand.

[111]  C. Darwin The Expression of the Emotions in Man and Animals , .

[112]  P. Ekman Pictures of Facial Affect , 1976 .

[113]  Te-Feng Su,et al.  Parallelized Face Based RMS System on a Multi-core Embedded Computing Platform , 2011, 2011 40th International Conference on Parallel Processing Workshops.

[114]  Mohammad H. Mahoor,et al.  DISFA: A Spontaneous Facial Action Intensity Database , 2013, IEEE Transactions on Affective Computing.

[115]  Weicheng Xie,et al.  Lighting difference based wrinkle mapping for expression synthesis , 2015, 2015 8th International Congress on Image and Signal Processing (CISP).

[116]  Stefanos Zafeiriou,et al.  Menpo: A Comprehensive Platform for Parametric Image Alignment and Visual Deformable Models , 2014, ACM Multimedia.

[117]  Luiz Velho,et al.  Expression Transfer between Photographs through Multilinear AAM's , 2006, 2006 19th Brazilian Symposium on Computer Graphics and Image Processing.

[118]  James A. Russell,et al.  "The role of context in interpreting facial expression:" Reply to Ekman and O'Sullivan , 1988 .

[119]  Cory J. Hayes,et al.  Facial Expression Synthesis on Robots: An ROS Module , 2015, HRI.

[120]  Bertram E. Shi,et al.  Photorealistic facial expression synthesis by the conditional difference adversarial autoencoder , 2017, 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII).

[121]  Wei Wei,et al.  A realistic dynamic facial expression transfer method , 2012, Neurocomputing.

[122]  Hao Wang,et al.  Affective interaction based on person-independent facial expression space , 2008, Neurocomputing.

[123]  Xin Li,et al.  Face alive icon , 2007, J. Vis. Lang. Comput..

[124]  Nanning Zheng,et al.  Interactive facial sketch expression generation using local constraints , 2009, 2009 IEEE International Conference on Intelligent Computing and Intelligent Systems.

[125]  Franck Davoine,et al.  Facial expression recognition and synthesis based on an appearance model , 2004, Signal Process. Image Commun..

[126]  Qionghai Dai,et al.  A Data-Driven Approach for Facial Expression Retargeting in Video , 2014, IEEE Transactions on Multimedia.