Generation of personalized video summaries by detecting viewer's emotion using electroencephalography

Abstract Video summaries produced by low level features are unaware of the viewer’s requirements and result in a semantic gap. Video content evokes certain emotions in a viewer, which can be measured and act as a strong source of information to generate summaries meeting viewer’s expectation. In this paper, we propose a personalized video summarization framework that classifies viewer’s emotion based on electroencephalography (EEG) signals, while watching a video to extract keyframes. Features are extracted from recorded EEG signals in time, frequency and wavelet domain to classify viewer’s emotions. Those frames are selected as keyframes from the video, where different emotions of viewer are evoked. Experiments are performed on 50 viewers and 50 video sequences to validate the effectiveness and efficiency of the proposed framework. It is evident from the results that the proposed method generates summaries with high precision, recall, F-measure, accuracy, and low error, hence reducing the semantic gap.

[1]  Chih-Jen Lin,et al.  Large-Scale Video Summarization Using Web-Image Priors , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Harry W. Agius,et al.  Analysing user physiological responses for affective video summarisation , 2009, Displays.

[3]  Arnaldo de Albuquerque Araújo,et al.  VSUMM: A mechanism designed to produce static video summaries and a novel evaluation method , 2011, Pattern Recognit. Lett..

[4]  Sankar K. Pal,et al.  Motion Frame Analysis and Scene Abstraction: Discrimination Ability of Fuzziness Measures , 1995, J. Intell. Fuzzy Syst..

[5]  Yukinobu Taniguchi,et al.  An intuitive and efficient access interface to real-time incoming video based on automatic indexing , 1995, MULTIMEDIA '95.

[6]  Yang Yi,et al.  Key frame extraction based on visual attention model , 2012, J. Vis. Commun. Image Represent..

[7]  Theodoros Iliou,et al.  Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011 , 2012, Artificial Intelligence Review.

[8]  Yan Liu,et al.  Full-Reference Quality Assessment for Video Summary , 2008, 2008 IEEE International Conference on Data Mining Workshops.

[9]  Thierry Pun,et al.  Multimodal Emotion Recognition in Response to Videos , 2012, IEEE Transactions on Affective Computing.

[10]  Chong-Wah Ngo,et al.  Automatic video summarization by graph modeling , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[11]  Wei Jiang,et al.  Memorable and rich video summarization , 2017, J. Vis. Commun. Image Represent..

[12]  Bao-Liang Lu,et al.  Emotional state classification from EEG data using machine learning approach , 2014, Neurocomputing.

[13]  F. B. Reguig,et al.  Emotion recognition from physiological signals , 2011, Journal of medical engineering & technology.

[14]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[15]  Ioannis Patras,et al.  Fusion of facial expressions and EEG for implicit affective tagging , 2013, Image Vis. Comput..

[16]  Sung Wook Baik,et al.  Efficient visual attention based framework for extracting key frames from videos , 2013, Signal Process. Image Commun..

[17]  Meng Wang,et al.  Event Driven Web Video Summarization by Tag Localization and Key-Shot Identification , 2012, IEEE Transactions on Multimedia.

[18]  Yong Jae Lee,et al.  Predicting Important Objects for Egocentric Video Summarization , 2015, International Journal of Computer Vision.

[19]  Yong Yu,et al.  Video summarization via transferrable structured learning , 2011, WWW.

[20]  Lan Li,et al.  Emotion Recognition Using Physiological Signals from Multiple Subjects , 2006, 2006 International Conference on Intelligent Information Hiding and Multimedia.

[21]  Harry W. Agius,et al.  Video summarisation: A conceptual framework and survey of the state of the art , 2008, J. Vis. Commun. Image Represent..

[22]  Guizhong Liu,et al.  A Multiple Visual Models Based Perceptive Analysis Framework for Multilevel Video Summarization , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[23]  Yan Liu,et al.  Rushes video summarization using audio-visual information and sequence alignment , 2008, TVS '08.

[24]  Sung Wook Baik,et al.  Divide-and-conquer based summarization framework for extracting affective video content , 2016, Neurocomputing.

[25]  Narasimhan Sundararajan,et al.  Classification of Mental Tasks from Eeg Signals Using Extreme Learning Machine , 2006, Int. J. Neural Syst..

[26]  Bin Zhao,et al.  Quasi Real-Time Summarization for Consumer Videos , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Wolfgang Effelsberg,et al.  Video abstracting , 1997, CACM.

[28]  Mohammad Hassan Moradi,et al.  A new approach for EEG feature extraction in P300-based lie detection , 2009, Comput. Methods Programs Biomed..

[29]  Chong-Wah Ngo,et al.  Summarizing Rushes Videos by Motion, Object, and Event Understanding , 2012, IEEE Transactions on Multimedia.

[30]  Jiang Peng,et al.  Keyframe-Based Video Summary Using Visual Attention Clues , 2010 .

[31]  Syed Muhammad Anwar,et al.  Emotion recognition in response to traditional and tactile enhanced multimedia using electroencephalography , 2018, Multimedia Tools and Applications.

[32]  Yong Jae Lee,et al.  Discovering important people and objects for egocentric video summarization , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[33]  Lie Lu,et al.  A generic framework of user attention model and its application in video summarization , 2005, IEEE Trans. Multim..

[34]  Shashidhar G. Koolagudi,et al.  Emotion recognition from speech using global and local prosodic features , 2013, Int. J. Speech Technol..

[35]  Zygmunt Pizlo,et al.  Automated video program summarization using speech transcripts , 2006, IEEE Transactions on Multimedia.

[36]  Marco Pellegrini,et al.  STIMO: STIll and MOving video storyboard for the web scenario , 2009, Multimedia Tools and Applications.

[37]  Changsheng Xu,et al.  Cross-Domain Feature Learning in Multimedia , 2015, IEEE Transactions on Multimedia.

[38]  Bernard Mérialdo,et al.  Multi-document video summarization , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[39]  Jiebo Luo,et al.  Towards Scalable Summarization of Consumer Videos Via Sparse Dictionary Selection , 2012, IEEE Transactions on Multimedia.

[40]  George Ghinea,et al.  User-centred personalised video abstraction approach adopting SIFT features , 2015, Multimedia Tools and Applications.

[41]  Xiang Ji,et al.  Clustering and retrieval of video shots based on natural stimulus fMRI , 2014, Neurocomputing.

[42]  Qiang Zhang,et al.  An Efficient Method of Key-Frame Extraction Based on a Cluster Algorithm , 2013, Journal of human kinetics.

[43]  Harry W. Agius,et al.  ELVIS: Entertainment-led video summaries , 2010, ACM Trans. Multim. Comput. Commun. Appl..

[44]  Sung Wook Baik,et al.  Feature aggregation based visual attention model for video summarization , 2014, Comput. Electr. Eng..

[45]  Syed Muhammad Anwar,et al.  Emotion recognition from facial expressions using hybrid feature descriptors , 2018, IET Image Process..

[46]  George Ghinea,et al.  What do you wish to see? A summarization system for movies based on user preferences , 2015, Inf. Process. Manag..

[47]  Syed Muhammad Anwar,et al.  Human emotion recognition and analysis in response to audio music using brain signals , 2016, Comput. Hum. Behav..

[48]  Syed Muhammad Anwar,et al.  Facial Expression Recognition Using Stationary Wavelet Transform Features , 2017 .