Looking at the viewer: analysing facial activity to detect personal highlights of multimedia contents

This paper presents an approach to detect personal highlights in videos based on the analysis of facial activities of the viewer. Our facial activity analysis was based on the motion vectors tracked on twelve key points in the human face. In our approach, the magnitude of the motion vectors represented a degree of a viewer’s affective reaction to video contents. We examined 80 facial activity videos recorded for ten participants, each watching eight video clips in various genres. The experimental results suggest that useful motion vectors to detect personal highlights varied significantly across viewers. However, it was suggested that the activity in the upper part of face tended to be more indicative of personal highlights than the activity in the lower part.

[1]  Nicu Sebe,et al.  Semisupervised learning of classifiers: theory, algorithms, and their application to human-computer interaction , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Nicu Sebe,et al.  Guest Editors' Introduction: Human-Centered Computing--Toward a Human Revolution , 2007, Computer.

[3]  Nicu Sebe,et al.  Learning probabilistic classifiers for human–computer interaction applications , 2005, Multimedia Systems.

[4]  Mohammad Soleymani,et al.  Affective ranking of movie scenes using physiological signals and content analysis , 2008, MS '08.

[5]  Thomas S. Huang,et al.  Human-Centered Computing : Toward a Human Revolution , 2007 .

[6]  Harry W. Agius,et al.  Feasibility of Personalized Affective Video Summaries , 2008, Affect and Emotion in Human-Computer Interaction.

[7]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[8]  Takeo Kanade,et al.  Pose Robust Face Tracking by Combining Active Appearance Models and Cylinder Head Models , 2007, International Journal of Computer Vision.

[9]  Gareth J. F. Jones,et al.  Affect-based indexing and retrieval of films , 2005, MULTIMEDIA '05.

[10]  Alan Hanjalic,et al.  Affective video content representation and modeling , 2005, IEEE Transactions on Multimedia.

[11]  Rainer Lienhart,et al.  The Holy Grail of Multimedia Information Retrieval: So Close or Yet So Far Away? , 2008 .

[12]  Thomas S. Huang,et al.  Connected vibrations: a modal analysis approach for non-rigid motion tracking , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[13]  Daijin Kim,et al.  Natural facial expression recognition using differential-AAM and manifold learning , 2009, Pattern Recognit..

[14]  J. York,et al.  Bayesian Graphical Models for Discrete Data , 1995 .

[15]  Svetha Venkatesh,et al.  Affect computing in film through sound energy dynamics , 2001, MULTIMEDIA '01.

[16]  Nicu Sebe,et al.  Exploiting facial expressions for affective video summarisation , 2009, CIVR '09.

[17]  A. Hanjalic,et al.  Extracting moods from pictures and sounds: towards truly personalized TV , 2006, IEEE Signal Processing Magazine.

[18]  Harry W. Agius,et al.  Video summarisation: A conceptual framework and survey of the state of the art , 2008, J. Vis. Commun. Image Represent..

[19]  Loong Fah Cheong,et al.  Affective understanding in film , 2006, IEEE Trans. Circuits Syst. Video Technol..

[20]  A. Mehrabian.  Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in Temperament , 1996 .

[21]  Paul Over,et al.  The trecvid 2007 BBC rushes summarization evaluation pilot , 2007, TVS '07.

[22]  Hang-Bong Kang,et al.  Analysis of scene context related with emotional events , 2002, MULTIMEDIA '02.

[23]  Yi-Ping Phoebe Chen,et al.  Highlights for more complete sports video summarization , 2004, IEEE MultiMedia.

[24]  Alan F. Smeaton,et al.  Investigating Biometric Response for Information Retrieval Applications , 2006, ECIR.

[25]  Lawrence S. Chen,et al.  Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction , 2000 .

[26]  Min Xu,et al.  Affective content analysis in comedy and horror videos by audio emotional event detection , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[27]  Peter Y. K. Cheung,et al.  A computation method for video segmentation utilizing the pleasure-arousal-dominance emotional information , 2007, ACM Multimedia.

[28]  Nicu Sebe,et al.  Facial expression recognition from video sequences: temporal and static modeling , 2003, Comput. Vis. Image Underst..

[29]  Nicu Sebe,et al.  How to complete performance graphs in content-based image retrieval: add generality and normalize scope , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.