Collective indexing of emotions in videos

Purpose – The object of this empirical research study is emotion, as depicted and aroused in videos. This paper seeks to answer the questions: Are users able to index such emotions consistently? Are the users' votes usable for emotional video retrieval?Design/methodology/approach – The authors worked with a controlled vocabulary for nine basic emotions (love, happiness, fun, surprise, desire, sadness, anger, disgust and fear), a slide control for adjusting the emotions' intensity, and the approach of broad folksonomies. Different users tagged the same videos. The test persons had the task of indexing the emotions of 20 videos (reprocessed clips from YouTube). The authors distinguished between emotions which were depicted in the video and those that were evoked in the user. Data were received from 776 participants and a total of 279,360 slide control values were analyzed.Findings – The consistency of the users' votes is very high; the tag distributions for the particular videos' emotions are stable. The fi...

[1]  Mohammad Soleymani,et al.  Affective ranking of movie scenes using physiological signals and content analysis , 2008, MS '08.

[2]  M. Cabanac.  What is emotion? , 2002, Behavioural Processes.

[3]  P. Shaver,et al.  Emotion knowledge: further exploration of a prototype approach. , 1987, Journal of personality and social psychology.

[4]  S. Tomkins.  Illuminating and Stimulating. (Book Reviews: Affect, Imagery, Consciousness. vol. 1, The Positive Affects) , 1963 .

[5]  P. Kleinginna,et al.  A categorized list of emotion definitions, with suggestions for a consensual definition , 1981 .

[6]  Peter G. B. Enser,et al.  The evolution of visual information retrieval , 2008, J. Inf. Sci..

[7]  Kathrin Knautz,et al.  MEMOSE: search engine for emotions in multimedia documents , 2010, SIGIR.

[8]  Wolfgang Nejdl,et al.  Bridging the gap between tagging and querying vocabularies: Analyses and applications for enhancing multimedia IR , 2010, J. Web Semant..

[9]  William McDougall,et al.  An Outline Of Abnormal Psychology , 1927 .

[10]  Eero Sormunen,et al.  End-User Searching Challenges Indexing Practices in the Digital Newspaper Photo Archive , 2004, Information Retrieval.

[11]  Min Xu,et al.  Affective content analysis in comedy and horror videos by audio emotional event detection , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[12]  Dragutin Petkovic,et al.  Content-based representation and retrieval of visual media: A state-of-the-art review , 1996, Multimedia Tools and Applications.

[13]  P. Young,et al.  Emotion and personality , 1963 .

[14]  Mark S. Melenhorst,et al.  Incorporating user motivations to design for video tagging , 2009, Interact. Comput..

[15]  Valentin Robu,et al.  The complex dynamics of collaborative tagging , 2007, WWW '07.

[16]  Carroll E. Izard.  Die Emotionen des Menschen : eine Einführung in die Grundlagen der Emotionspsychologie , 1994 .

[17]  Wolfgang G. Stock,et al.  Folksonomies in Wissensrepräsentation und Information Retrieval , 2008 .

[18]  Wolfgang G. Stock,et al.  Wissensrepräsentation: Informationen auswerten und bereitstellen , 2008 .

[19]  J. Turner.  Human Emotions: A Sociological Theory , 2007 .

[20]  Achim Schützwohl,et al.  Einführung in die Emotionspsychologie/3 , 2006 .

[21]  Huanbo Luan,et al.  Content-based video retrieval: Three example systems from TRECVid , 2008 .

[22]  W. Mcdougall.  An introduction to social psychology , 1909 .

[23]  Marcel Worring,et al.  Concept-Based Video Retrieval , 2009, Found. Trends Inf. Retr..

[24]  Katrin Schweins.  Methoden zur Erschließung von Filmsequenzen , 1997 .

[25]  Yong Ho Kim,et al.  A two-step model for video key-frame determination , 2009, ASIST.

[26]  E. Rosch.  Cognitive reference points , 1975, Cognitive Psychology.

[27]  Corinne Jörgensen,et al.  Image Access, the Semantic Gap, and Social Tagging as a Paradigm Shift , 2007 .

[28]  K. Markey.  Interindexer consistency tests: a literature review and report of a test of consistency in indexing visual materials , 1984 .

[29]  Isabella Peters,et al.  Folksonomies - Indexing and Retrieval in Web 2.0 , 2009, Knowledge and Information.

[30]  Peter G. B. Enser,et al.  Activity theory and information seeking , 2008 .

[31]  Wolfgang G. Stock,et al.  Collective indexing of emotions in images. A study in emotional information retrieval , 2009 .

[32]  M. Erb,et al.  Are emotions contagious? Evoked emotions while viewing emotionally expressive faces: quality, quantity, time course and gender differences , 2001, Psychiatry Research.

[33]  Wolfgang G. Stock.  Folksonomies and science communication - A mash-up of professional science databases and Web 2.0 services , 2007, Inf. Serv. Use.

[34]  Jaak Panksepp,et al.  Toward a general psychobiological theory of emotions , 1982, Behavioral and Brain Sciences.

[35]  C. Izard.  The face of emotion , 1971 .

[36]  Steve Benford,et al.  EmoPlayer: A media player for video clips with affective annotations , 2008, Interact. Comput..

[37]  Achim Schützwohl,et al.  Einführung in die Emotionspsychologie /2 , 2006 .

[38]  A. Ortony,et al.  What's basic about basic emotions? , 1990, Psychological review.

[39]  Pauline Rafferty.  Informative Tagging of Images: The Importance of Modality in Interpretation , 2009 .

[40]  C. Darwin.  The Expression of the Emotions in Man and Animals , .

[41]  Abby Goodrum,et al.  Image Information Retrieval: An Overview of Current Research , 2000, Informing Sci. Int. J. an Emerg. Transdiscipl..

[42]  W. James.  II.—WHAT IS AN EMOTION ? , 1884 .

[43]  Wolfgang G. Stock,et al.  Folksonomy and information retrieval , 2008, ASIST.

[44]  Maria E. Jabon,et al.  Real-time classification of evoked emotions using facial feature tracking and physiological responses , 2008, Int. J. Hum. Comput. Stud..

[45]  Alberto Del Bimbo,et al.  Visual information retrieval , 1999 .

[46]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[47]  J. Russell,et al.  Concept of Emotion Viewed From a Prototype Perspective , 1984 .

[48]  Corinne Jörgensen,et al.  Attributes of Images in Describing Tasks , 1998, Inf. Process. Manag..

[49]  Jonathon S. Hare,et al.  Facing the reality of semantic image retrieval , 2007, J. Documentation.

[50]  S. Tomkins,et al.  Affect Imagery Consciousness: The Positive Affects , 1963 .

[51]  Wolfgang G. Stock,et al.  "Power tags" in information retrieval , 2010, Libr. Hi Tech.

[52]  Wolfgang G. Stock,et al.  Folksonomy: The Collaborative Knowledge Organization System , 2010 .

[53]  Mark S. Melenhorst,et al.  Tag-based information retrieval of video content , 2008, UXTV '08.

[54]  Paul Dourish,et al.  How emotion is made and measured , 2007, Int. J. Hum. Comput. Stud..

[55]  Andrew Large,et al.  ASIST 2009 : proceedings of the 72nd ASIS&T Annual Meeting : thriving on diversity : information opportunities in a pluralistic world , 2009 .

[56]  Mike Graham,et al.  Extracting information about emotions in films , 2003, ACM Multimedia.

[57]  Gary Geisler,et al.  Tagging video: conventions and strategies of the YouTube community , 2007, JCDL '07.

[58]  Alan Hanjalic,et al.  Affective video content representation and modeling , 2005, IEEE Transactions on Multimedia.

[59]  R. Plutchik.  A GENERAL PSYCHOEVOLUTIONARY THEORY OF EMOTION , 1980 .

[60]  Alberto Del Bimbo,et al.  Semantics in Visual Information Retrieval , 1999, IEEE Multim..

[61]  Daniela Petrelli,et al.  An examination of automatic video retrieval technology on access to the contents of an historical video archive , 2008, Program.

[62]  Mari Laine-Hernandez,et al.  Image semantics in the description and categorization of journalistic photographs , 2007, ASIST.

[63]  Gareth J. F. Jones,et al.  Affect-based indexing and retrieval of films , 2005, MULTIMEDIA '05.

[64]  Diane Neal,et al.  Toward Web 2.0 music information retrieval: Utilizing emotion-based, user-assigned descriptors , 2007, ASIST.

[65]  Wolfgang G. Stock.  On relevance distributions , 2006, J. Assoc. Inf. Sci. Technol..

[66]  Alan F. Smeaton.  Techniques used and open challenges to the analysis, indexing and retrieval of digital video , 2007, Inf. Syst..