Mirror mirror

We describe a method for providing feedback on portrait expressions, and for selecting the most attractive expressions from large video/photo collections. We capture a video of a subject's face while they are engaged in a task designed to elicit a range of positive emotions. We then use crowdsourcing to score the captured expressions for their attractiveness. We use these scores to train a model that can automatically predict attractiveness of different expressions of a given person. We also train a cross-subject model that evaluates portrait attractiveness of novel subjects and show how it can be used to automatically mine attractive photos from personal photo collections. Furthermore, we show how, with a little bit ($5-worth) of extra crowdsourcing, we can substantially improve the cross-subject model by "fine-tuning" it to a new individual using active learning. Finally, we demonstrate a training app that helps people learn how to mimic their best expressions.

[1]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Fei Yang,et al.  Expression flow for 3D-aware face component transfer , 2011, ACM Trans. Graph..

[3]  Fernando De la Torre,et al.  Supervised Descent Method and Its Applications to Face Alignment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Dani Lischinski,et al.  Data-driven enhancement of facial attractiveness , 2008, ACM Trans. Graph..

[5]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[6]  Kristen Grauman,et al.  Beyond Comparing Image Pairs: Setwise Active Learning for Relative Attributes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Li Zhang,et al.  Spacetime faces: high resolution capture for modeling and animation , 2004, SIGGRAPH 2004.

[8]  Maja Pantic,et al.  Action unit detection using sparse appearance descriptors in space-time video volumes , 2011, Face and Gesture 2011.

[9]  J. Cohn,et al.  All Smiles are Not Created Equal: Morphology and Timing of Smiles Perceived as Amused, Polite, and Embarrassed/Nervous , 2009, Journal of nonverbal behavior.

[10]  Daniel McDuff,et al.  Crowdsourcing Facial Responses to Online Videos , 2012, IEEE Transactions on Affective Computing.

[11]  Yong Tao,et al.  Compound facial expressions of emotion , 2014, Proceedings of the National Academy of Sciences.

[12]  Michael F. Cohen,et al.  Very low frame-rate video streaming for face-to-face teleconference , 2005, Data Compression Conference.

[13]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[14]  Aaron Hertzmann,et al.  Exploratory font selection using crowdsourced attributes , 2014, ACM Trans. Graph..

[15]  E. Ruppin,et al.  A machine learning predictor of facial attractiveness revealing human-like psychophysical biases , 2008, Vision Research.

[16]  H. A. David,et al.  The method of paired comparisons , 1966 .

[17]  R. A. Bradley,et al.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS THE METHOD OF PAIRED COMPARISONS , 1952 .

[18]  Serge J. Belongie,et al.  Relative ranking of facial attractiveness , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[19]  A. Manstead,et al.  Can Duchenne smiles be feigned? New evidence on felt and false smiles. , 2009, Emotion.

[20]  Antonio Torralba,et al.  Modifying the Memorability of Face Photographs , 2013, 2013 IEEE International Conference on Computer Vision.

[21]  Brian Curless,et al.  Candid portrait selection from video , 2011, ACM Trans. Graph..

[22]  R. A. Bradley,et al.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS , 1952 .

[23]  Alexander C. Berg,et al.  Hipster Wars: Discovering Elements of Fashion Styles , 2014, ECCV.

[24]  Kristen Grauman,et al.  Relative attributes , 2011, 2011 International Conference on Computer Vision.

[25]  I. N. Springer,et al.  Mirror, mirror on the wall…: self-perception of facial beauty versus judgement by others. , 2012, Journal of cranio-maxillo-facial surgery : official publication of the European Association for Cranio-Maxillo-Facial Surgery.

[26]  Vivek Kwatra,et al.  All smiles: automatic photo enhancement by facial expression analysis , 2012, CVMP.

[27]  Yihong Gong,et al.  Predicting Facial Beauty without Landmarks , 2010, ECCV.

[28]  Maya R. Gupta,et al.  How to Analyze Paired Comparison Data , 2011 .

[29]  Tsuhan Chen,et al.  Clothing cosegmentation for recognizing people , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Anita Sellent,et al.  The Good, the Bad and the Ugly: Attractive Portraits from Video Sequences , 2008 .

[31]  John Hart,et al.  ACM Transactions on Graphics , 2004, SIGGRAPH 2004.

[32]  R. A. Bradley,et al.  Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons , 1952 .

[33]  A. Todorov,et al.  The functional basis of face evaluation , 2008, Proceedings of the National Academy of Sciences.

[34]  Judith A. Hall,et al.  The Deliberate Duchenne Smile: Individual Differences in Expressive Control , 2013 .

[35]  Edward H. Adelson,et al.  Personal photo enhancement using example images , 2010, TOGS.

[36]  Mark H. Johnson,et al.  Oxford Handbook of Face Perception , 2011 .

[37]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[38]  Nir Ailon,et al.  An Active Learning Algorithm for Ranking from Pairwise Preferences with an Almost Optimal Query Complexity , 2010, J. Mach. Learn. Res..

[39]  Jitendra Malik,et al.  Discriminative Decorrelation for Clustering and Classification , 2012, ECCV.

[40]  Robert D. Nowak,et al.  Active Ranking using Pairwise Comparisons , 2011, NIPS.

[41]  Ira Kemelmacher-Shlizerman,et al.  Exploring photobios , 2011, ACM Trans. Graph..

[42]  David Matsumoto,et al.  Cultural Similarities and Differences in Emblematic Gestures , 2012, Journal of Nonverbal Behavior.

[43]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[44]  Paul N. Bennett,et al.  Pairwise ranking aggregation in a crowdsourced setting , 2013, WSDM.

[45]  Albert Ali Salah,et al.  Are You Really Smiling at Me? Spontaneous versus Posed Enjoyment Smiles , 2012, ECCV.

[46]  J. Gross,et al.  Emotion elicitation using films , 1995 .

[47]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..