Using Viewer's Facial Expression and Heart Rate for Sports Video Highlights Detection

Viewer interest, evoked by video content, can potentially identify the highlights of the video. This paper explores the use of facial expressions (FE) and heart rate (HR) of viewers captured using camera and non-strapped sensor for identifying interesting video segments. The data from ten subjects with three videos showed that these signals are viewer dependent and not synchronized with the video contents. To address this issue, new algorithms are proposed to effectively combine FE and HR signals for identifying the time when viewer interest is potentially high. The results show that, compared with subjective annotation and match report highlights, 'non-neutral' FE and 'relatively higher and faster' HR is able to capture 60%-80% of goal, foul, and shot-on-goal soccer video events. FE is found to be more indicative than HR of viewer interest, but the fusion of these two modalities outperforms each of them.

[1]  Wei-Ta Chu,et al.  A User Experience Model for Home Video Summarization , 2009, MMM.

[2]  Wei-Ta Chu,et al.  Editing by Viewing: Automatic Home Video Summarization by Viewing Behavior Analysis , 2011, IEEE Transactions on Multimedia.

[3]  M. Bradley,et al.  Looking at pictures: affective, facial, visceral, and behavioral reactions. , 1993, Psychophysiology.

[4]  Tom Beckers,et al.  Psychophysiological Response Patterns to Affective Film Stimuli , 2013, PloS one.

[5]  Ioannis Konstas,et al.  Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance , 2009, ACM Multimedia.

[6]  Vinod Chandran,et al.  Facial expression recognition experiments with data from television broadcasts and the World Wide Web , 2014, Image Vis. Comput..

[7]  Quan Huynh-Thu,et al.  Physiological-Based Affect Event Detector for Entertainment Video Applications , 2012, IEEE Transactions on Affective Computing.

[8]  Mohammad Soleymani,et al.  Affective ranking of movie scenes using physiological signals and content analysis , 2008, MS '08.

[9]  Jakub Parák,et al.  Evaluation of wearable consumer heart rate monitors based on photopletysmography , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[10]  Mohammad Soleymani,et al.  Continuous emotion detection in response to music videos , 2011, Face and Gesture 2011.

[11]  Qiang Ji,et al.  Implicit video emotion tagging from audiences’ facial expression , 2013, Multimedia Tools and Applications.

[12]  Daniel McDuff,et al.  Measuring Voter's Candidate Preference Based on Affective Responses to Election Debates , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[13]  Nicu Sebe,et al.  Looking at the viewer: analysing facial activity to detect personal highlights of multimedia contents , 2010, Multimedia Tools and Applications.

[14]  Geoff Hulten,et al.  Measuring the engagement level of TV viewers , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[15]  Yi-Ping Phoebe Chen,et al.  The power of play-break for automatic detection and browsing of self-consumable sport video highlights , 2004, MIR '04.

[16]  Nicu Sebe,et al.  Exploiting facial expressions for affective video summarisation , 2009, CIVR '09.

[17]  Harry W. Agius,et al.  Analysing user physiological responses for affective video summarisation , 2009, Displays.

[18]  Mohammad Soleymani,et al.  Affective Characterization of Movie Scenes Based on Content Analysis and Physiological Changes , 2009, Int. J. Semantic Comput..