Incorporating social media comments in affective video retrieval

Affective video retrieval systems aim at finding video contents matching the desires and needs of users. Existing systems typically use the information contained in the video itself to specify its affect category. These systems either extract low-level features or build up higher-level attributes to train classification algorithms. However, using low-level features ignores global relations in data and constructing high-level features is time consuming and problem dependent. To overcome these drawbacks, an external source of information may be helpful. With the explosive growth and availability of social media, users’ comments could be such a valuable source of information. In this study, a new method for incorporating social media comments with the audio-visual contents of videos is proposed. Furthermore, for the combination stage a decision-level fusion method based on the Dempster–Shafer theory of evidence is presented. Experiments are carried out on the video clips of the DEAP (Database for Emotion Analysis using Physiological signals) dataset and their associated users’ comments on YouTube. Results show that the proposed system significantly outperforms the baseline method of using only the audio-visual contents for affective video retrieval.

[1]  K. Scherer,et al.  Multimodal expression of emotion , 2009 .

[2]  Christopher D. Manning,et al.  Baselines and Bigrams: Simple, Good Sentiment and Topic Classification , 2012, ACL.

[3]  Yi-Hsuan Yang,et al.  Quantitative Study of Music Listening Behavior in a Social and Affective Context , 2013, IEEE Transactions on Multimedia.

[4]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[5]  Tod Morrison,et al.  A NEW PARADIGM FOR ROBUST COMBINATORIAL OPTIMIZATION: USING PERSISTENCE AS A THEORY OF EVIDENCE , 2010 .

[6]  Hang-Bong Kang,et al.  Affective content detection using HMMs , 2003, ACM Multimedia.

[7]  Yaxin Bi,et al.  The combination of multiple classifiers using an evidential reasoning approach , 2008, Artif. Intell..

[8]  Yaser Sheikh,et al.  On the use of computable features for film classification , 2005, IEEE Transactions on Circuits and Systems for Video Technology.

[9]  Yonghwan Kim,et al.  Exploring characteristics of video consuming behaviour in different social media using K-pop videos , 2014, J. Inf. Sci..

[10]  Junqing Yu,et al.  Video Affective Content Representation and Recognition Using Video Affective Tree and Hidden Markov Models , 2007, ACII.

[11]  Jin-Woo Jeong,et al.  Automatic image annotation using affective vocabularies: Attribute-based learning approach , 2014, J. Inf. Sci..

[12]  Huan Liu,et al.  Exploiting social relations for sentiment analysis in microblogging , 2013, WSDM.

[13]  Hsinchun Chen,et al.  Text‐based video content classification for online video‐sharing sites , 2010, J. Assoc. Inf. Sci. Technol..

[14]  Kiyoharu Aizawa,et al.  Determination of emotional content of video clips by low-level audiovisual features , 2011, Multimedia Tools and Applications.

[15]  Yaxin Bi.  The impact of diversity on the accuracy of evidential classifier ensembles , 2012, Int. J. Approx. Reason..

[16]  Frank Hopfgartner,et al.  Understanding Affective Content of Music Videos through Learned Representations , 2014, MMM.

[17]  Nasser Ghasem-Aghaee,et al.  Sentiment Prediction Based on Dempster-Shafer Theory of Evidence , 2014 .

[18]  François-Régis Chaumartin,et al.  UPAR7: A knowledge-based system for headline sentiment tagging , 2007, Fourth International Workshop on Semantic Evaluations (SemEval-2007).

[19]  Zheng Li-xin,et al.  Block Matching Algorithms for Motion Estimation , 2005 .

[20]  Mohan S. Kankanhalli,et al.  Multimodal fusion for multimedia analysis: a survey , 2010, Multimedia Systems.

[21]  Touradj Ebrahimi,et al.  Multimedia content analysis for emotional characterization of music video clips , 2013, EURASIP J. Image Video Process..

[22]  Touradj Ebrahimi,et al.  Affective content analysis of music video clip , 2011, MM 2011.

[23]  Alan Hanjalic,et al.  Affective video content representation and modeling , 2005, IEEE Transactions on Multimedia.

[24]  Xiangjian He,et al.  Hierarchical affective content analysis in arousal and valence dimensions , 2013, Signal Process..

[25]  Ioannis Patras,et al.  Fusion of facial expressions and EEG for implicit affective tagging , 2013, Image Vis. Comput..

[26]  Yi-Hsuan Yang,et al.  LJ2M dataset: Toward better understanding of music listening behavior and user mood , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).

[27]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Nasser Ghasem-Aghaee,et al.  Exploiting reviewers’ comment histories for sentiment analysis , 2014, J. Inf. Sci..

[29]  Shiliang Zhang,et al.  Affective Visualization and Retrieval for Music Video , 2010, IEEE Transactions on Multimedia.

[30]  Irene Lopatovska,et al.  Toward a model of emotions and mood in the online information search process , 2014, J. Assoc. Inf. Sci. Technol..

[31]  Loong Fah Cheong,et al.  Affective understanding in film , 2006, IEEE Trans. Circuits Syst. Video Technol..

[32]  Yuji Matsumoto,et al.  Emotion Classification Using Massive Examples Extracted from the Web , 2008, COLING.

[33]  Andries Petrus Engelbrecht,et al.  A Cooperative approach to particle swarm optimization , 2004, IEEE Transactions on Evolutionary Computation.

[34]  Christopher S. G. Khoo,et al.  Aspect-based sentiment analysis of movie reviews on discussion boards , 2010, J. Inf. Sci..

[35]  Qiang Ji,et al.  Hybrid video emotional tagging using users’ EEG and video content , 2014, Multimedia Tools and Applications.

[36]  Huan Liu,et al.  Unsupervised sentiment analysis with emotional signals , 2013, WWW.

[37]  K. Scherer,et al.  Multimodal expression of emotion: affect programs or componential appraisal patterns? , 2007, Emotion.

[38]  Carolyn Penstein Rosé,et al.  Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning , 2008, Int. J. Comput. Support. Collab. Learn..

[39]  Athanasia Zlatintsi,et al.  A supervised approach to movie emotion tracking , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[40]  Thierry Pun,et al.  Multimodal Emotion Recognition in Response to Videos , 2012, IEEE Transactions on Affective Computing.

[41]  Jin Zhang,et al.  An empirical study of sentiment analysis for chinese documents , 2008, Expert Syst. Appl..

[42]  Hua Xu,et al.  Text-based emotion classification using emotion cause extraction , 2014, Expert Syst. Appl..

[43]  Touradj Ebrahimi,et al.  Affective content analysis of music video clips , 2011, MIRUM '11.