Facial component-based blended facial expressions generation from static neutral face images

Facial expression synthesis is getting a wide-spread attention since past several years due to its multimedia applications. In most of the earlier research works, example images of target expressions are required to produce synthesized facial expressions. The paper aims to generate six basic and twelve blended facial expressions from a static and RGB neutral face image without any exemplar of expressive face images. The proposed automatic expression generation system consists of several sub-systems, namely, a knowledge-based system, a module for symbolic formulations of basic and blended facial expressions, an expressive facial components generator and an expressive face generator. The knowledge-based system stores the normalized facial feature parameter values. Symbolic formulations of facial expressions are used to reconstruct facial expressions from a static neutral face image using the parameters stored in the knowledge base. Expressive facial components generator performs automatic expressive facial feature generation as well as automatic facial feature extraction and landmark annotation. Finally, expressive facial components are combined to produce an expressive face in expressive face generator. The system generated expressive face images are validated in four different ways: inter-rater reliability measure, similarity measurement in the frequency domain, similarity measurement using SSIM, FSIM, HOG features and accuracy measurement using both appearance-based and geometry-based feature extraction methods. The geometry based feature extraction method generates 90% recognition accuracy for system generated face images.

[1]  Nanning Zheng,et al.  Extended facial expression synthesis using statistical appearance model , 2009, 2009 4th IEEE Conference on Industrial Electronics and Applications.

[2]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[3]  LinLin Shen,et al.  Facial expression synthesis with direction field preservation based mesh deformation and lighting fitting based wrinkle mapping , 2018, Multimedia Tools and Applications.

[4]  Calvin R. Maurer,et al.  A Linear Time Algorithm for Computing Exact Euclidean Distance Transforms of Binary Images in Arbitrary Dimensions , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Zicheng Liu,et al.  Expressive expression mapping with ratio images , 2001, SIGGRAPH.

[6]  Hans-Peter Seidel,et al.  A Generic Framework for Efficient 2-D and 3-D Facial Expression Analogy , 2007, IEEE Transactions on Multimedia.

[7]  D Marr,et al.  Theory of edge detection , 1979, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[8]  Paul A. Viola,et al.  Robust Real-time Object Detection , 2001 .

[9]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Justus Thies,et al.  Real-time expression transfer for facial reenactment , 2015, ACM Trans. Graph..

[11]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[12]  Xiaoou Tang,et al.  Optical and Sonar Image Classification: Wavelet Packet Transform vs Fourier Transform , 2000, Comput. Vis. Image Underst..

[13]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[14]  Mita Nasipuri,et al.  Mathematical Representations of Blended Facial Expressions towards Facial Expression Modeling , 2015, IHCI.

[15]  Xue Li,et al.  A 2D Personalized Facial Expression Generation Approach , 2013, ICCC.

[16]  Fernando De la Torre,et al.  Bilinear Kernel Reduced Rank Regression for Facial Expression Synthesis , 2010, ECCV.

[17]  Mahantapas Kundu,et al.  Construction of human faces from textual descriptions , 2011, Soft Comput..

[18]  Dipti Prasad Mukherjee,et al.  Synthesis of emotional expressions specific to facial structure , 2012, ICVGIP '12.

[19]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[20]  Linda G. Shapiro,et al.  Computer and Robot Vision , 1991 .

[21]  Mrinal Kanti Bhowmik,et al.  DeitY-TU face database: its design, multiple camera capturing, characteristics, and evaluation , 2014 .

[22]  Carlos E. Thomaz,et al.  A Photo-Realistic Generator of Most Expressive and Discriminant Changes in 2D Face Images , 2015, 2015 Sixth International Conference on Emerging Security Technologies (EST).

[23]  Rolf Adams,et al.  Radial Decomposition of Discs and Spheres , 1993, CVGIP Graph. Model. Image Process..

[24]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[25]  Hao Li,et al.  Example-based facial rigging , 2010, ACM Transactions on Graphics.

[26]  N. L. Johnson,et al.  Continuous Univariate Distributions. , 1995 .

[27]  P. Ekman,et al.  Facial action coding system , 2019 .

[28]  Pierre Soille,et al.  Morphological Image Analysis: Principles and Applications , 2003 .

[29]  Yihao Zhang,et al.  Facial expression mapping based on elastic and muscle-distribution-based models , 2012, 2012 IEEE International Symposium on Circuits and Systems.

[30]  Ning Xu,et al.  Intra-and-Inter-Constraint-Based Video Enhancement Based on Piecewise Tone Mapping , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[31]  A. Z. Kouzani,et al.  Constructing artificial images of facial expressions , 2001, The Seventh Australian and New Zealand Intelligent Information Systems Conference, 2001.

[32]  I. King,et al.  Radial Basis Network for Facial Expression Synthesis , 1996 .

[33]  Aseem Agarwala,et al.  Efficient gradient-domain compositing using quadtrees , 2007, ACM Trans. Graph..

[34]  Qionghai Dai,et al.  A Data-Driven Approach for Facial Expression Retargeting in Video , 2014, IEEE Transactions on Multimedia.

[35]  Anthony C. Boucouvalas,et al.  Expressive Image Generator for an Emotion Extraction Engine , 2004 .

[36]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[37]  B. Abboud,et al.  Bilinear factorisation for facial expression analysis and synthesis , 2005 .

[38]  Wilhelm Burger,et al.  Digital Image Processing - An Algorithmic Introduction using Java , 2008, Texts in Computer Science.

[39]  John Daugman,et al.  How iris recognition works , 2002, IEEE Transactions on Circuits and Systems for Video Technology.

[40]  Jia-Shing Sheu,et al.  Automatic Generation of Facial Expression Using Triangular Geometric Deformation , 2014 .

[41]  Geoffrey E. Hinton,et al.  Generating Facial Expressions with Deep Belief Nets , 2008 .

[42]  Bing Zhou,et al.  Facial expression cloning with elastic and muscle models , 2014, J. Vis. Commun. Image Represent..

[43]  Chuan-Kai Yang,et al.  An interactive facial expression generation system , 2008, Multimedia Tools and Applications.

[44]  Yong Tao,et al.  Compound facial expressions of emotion , 2014, Proceedings of the National Academy of Sciences.

[45]  Baining Guo,et al.  Geometry-driven photorealistic facial expression synthesis , 2003, IEEE Transactions on Visualization and Computer Graphics.

[46]  Xiaogang Jin,et al.  Data-driven facial expression synthesis via Laplacian deformation , 2010, Multimedia Tools and Applications.

[47]  Franck Davoine,et al.  Bilinear factorization for facial expression analysis and synthesis , 2004 .

[48]  Juebo Wu,et al.  A Cloud Model-based Approach for Facial Expression Synthesis , 2011, J. Multim..

[49]  Anthony C. Boucouvalas,et al.  Emotion Extraction Engine : Expressive Image generator , 2003 .

[50]  Ming C. Lin,et al.  Free-flowing granular materials with two-way solid coupling , 2010, SIGGRAPH 2010.

[51]  Qiang Ji,et al.  Active and dynamic information fusion for facial expression understanding from image sequences , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Jim R. Parker,et al.  Algorithms for image processing and computer vision , 1996 .

[53]  Yihjia Tsai,et al.  Facial Expression Synthesis Based on Imitation , 2012 .