Automatic measurement of ad preferences from facial responses gathered over the Internet

Abstract We present an automated method for classifying “liking” and “desire to view again” of online video ads based on 3268 facial responses to media collected over the Internet. The results demonstrate the possibility for an ecologically valid, unobtrusive, evaluation of commercial “liking” and “desire to view again”, strong predictors of marketing success, based only on facial responses. The area under the curve for the best “liking” classifier was 0.82 when using a challenging leave-one-commercial-out testing regime (accuracy = 81%). We build on preliminary findings and show that improved smile detection can lead to a reduction in misclassifications. Comparison of the two smile detection algorithms showed that improved smile detection helps correctly classify responses recorded in challenging lighting conditions and those in which the expressions were subtle. Temporal discriminative approaches to classification performed most strongly showing that temporal information about an individual's response is important; it is not just how much a viewer smiles but when they smile. The technique could be employed in personalizing video content that is presented to people while they view videos over the Internet or in copy testing of ads to unobtrusively quantify ad effectiveness.

[1]  Nicu Sebe,et al.  Looking at the viewer: analysing facial activity to detect personal highlights of multimedia contents , 2010, Multimedia Tools and Applications.

[2]  Daniel McDuff,et al.  Crowdsourcing Facial Responses to Online Videos , 2012, IEEE Transactions on Affective Computing.

[3]  Trevor Darrell,et al.  Latent-Dynamic Discriminative Models for Continuous Gesture Recognition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Daniel McDuff,et al.  Affect valence inference from facial action unit spectrograms , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[5]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[6]  Daniel McDuff,et al.  Predicting online media effectiveness based on smile responses gathered over the Internet , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[7]  R. Pieters,et al.  Emotion-Induced Engagement in Internet Video Advertisements , 2012 .

[8]  Paul D. Bolls,et al.  The Effects of Message Valence and Listener Arousal on Attention, Memory, and Facial Muscular Responses to Radio Advertisements , 2001, Commun. Res..

[9]  Yale Song,et al.  Multi-signal gesture recognition using temporal smoothing hidden conditional random fields , 2011, Face and Gesture 2011.

[10]  Katherine B. Martin,et al.  Facial Action Coding System , 2015 .

[11]  J. Cohn,et al.  All Smiles are Not Created Equal: Morphology and Timing of Smiles Perceived as Amused, Polite, and Embarrassed/Nervous , 2009, Journal of nonverbal behavior.

[12]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Daniel McDuff,et al.  Affectiva-MIT Facial Expression Dataset (AM-FED): Naturalistic and Spontaneous Facial Expressions Collected "In-the-Wild" , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[15]  Hongxun Yao,et al.  Video classification and recommendation based on affective analysis of viewers , 2013, Neurocomputing.

[16]  P. Neijens,et al.  Effects of Advertising Likeability: A 10-Year Perspective , 2006, Journal of Advertising Research.

[17]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[18]  Anca C. Micu,et al.  Measurable Emotions: How Television Ads Really Work , 2010, Journal of Advertising Research.

[19]  Hatice Gunes,et al.  How to distinguish posed from spontaneous smiles using geometric features , 2007, ICMI '07.

[20]  Gwen Littlewort,et al.  Toward Practical Smile Detection , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Michel Wedel,et al.  Moment-to-Moment Optimal Branding in TV Commercials: Preventing Avoidance by Pulsing , 2010, Mark. Sci..

[22]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[23]  Simon Lucey,et al.  Registration Invariant Representations for Expression Detection , 2010, 2010 International Conference on Digital Image Computing: Techniques and Applications.

[24]  Trevor Darrell,et al.  Hidden Conditional Random Fields for Gesture Recognition , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[25]  Daniel McDuff,et al.  Crowdsourced data collection of facial responses , 2011, ICMI '11.

[26]  Russell I. Haley,et al.  The ARF Copy Research Validity Project , 2000, Journal of Advertising Research.

[27]  Daniel McDuff,et al.  Exploring Temporal Patterns in Classifying Frustrated and Delighted Smiles , 2012, IEEE Trans. Affect. Comput..

[28]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Caifeng Shan,et al.  Smile detection by boosting pixel differences , 2012, IEEE Transactions on Image Processing.

[30]  K. Kassam.  Assessment of emotional experience through facial expression , 2010 .

[31]  Richard L. Hazlett,et al.  Emotional response to television commercials: Facial EMG vs. self-report , 1999 .