Affective Recommendation of Movies Based on Selected Connotative Features

The apparent difficulty in assessing emotions elicited by movies and the undeniable high variability in subjects' emotional responses to film content have been recently tackled by exploring film connotative properties: the set of shooting and editing conventions that help in transmitting meaning to the audience. Connotation provides an intermediate representation that exploits the objectivity of audiovisual descriptors to predict the subjective emotional reaction of single users. This is done without the need of registering users' physiological signals. It is not done by employing other people's highly variable emotional rates, but by relying on the intersubjectivity of connotative concepts and on the knowledge of user's reactions to similar stimuli. This paper extends previous work by extracting audiovisual and film grammar descriptors and, driven by users' rates on connotative properties, creates a shared framework where movie scenes are placed, compared, and recommended according to connotation. We evaluate the potential of the proposed system by asking users to assess the ability of connotation in suggesting film content able to target their affective requests.

[1]  Maja Pantic,et al.  Implicit human-centered tagging [Social Sciences] , 2009, IEEE Signal Process. Mag..

[2]  B. S. Manjunath,et al.  Color and texture descriptors , 2001, IEEE Trans. Circuits Syst. Video Technol..

[3]  Chris H. Q. Ding,et al.  Minimum redundancy feature selection from microarray gene expression data , 2003, Computational Systems Bioinformatics. CSB2003. Proceedings of the 2003 IEEE Bioinformatics Conference. CSB2003.

[4]  Ioannis Arapakis,et al.  Theories, methods and current research on emotions in library and information science, information retrieval and human-computer interaction , 2011, Inf. Process. Manag..

[5]  Leonidas J. Guibas,et al.  The Earth Mover's Distance as a Metric for Image Retrieval , 2000, International Journal of Computer Vision.

[6]  D. Arijon,et al.  Grammar of Film Language , 1976 .

[7]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[8]  M. Kendall Rank Correlation Methods , 1949 .

[9]  Rongrong Ji,et al.  Video indexing and recommendation based on affective analysis of viewers , 2011, MM '11.

[10]  Clino T. Castelli Trini Diagram: imaging emotional identity 3D positioning tool , 1999, Electronic Imaging.

[11]  A. Hanjalic,et al.  Extracting moods from pictures and sounds: towards truly personalized TV , 2006, IEEE Signal Processing Magazine.

[12]  Alan Hanjalic,et al.  Affective video content representation and modeling , 2005, IEEE Transactions on Multimedia.

[13]  J. Russell A circumplex model of affect. , 1980 .

[14]  Jane Labadin,et al.  Feature selection based on mutual information , 2015, 2015 9th International Conference on IT in Asia (CITA).

[15]  Ajay Divakaran,et al.  MPEG-7 visual motion descriptors , 2001, IEEE Trans. Circuits Syst. Video Technol..

[16]  Rosalind W. Picard Affective Computing: From Laughter to IEEE , 2010 .

[17]  Kiyoharu Aizawa,et al.  Affective Audio-Visual Words and Latent Topic Driving Model for Realizing Movie Affective Scene Classification , 2010, IEEE Transactions on Multimedia.

[18]  E. Tan Film-induced affect as a witness emotion , 1995 .

[19]  Marko Tkalcic,et al.  Using affective parameters in a content-based recommender system for images , 2010, User Modeling and User-Adapted Interaction.

[20]  Loong Fah Cheong,et al.  Taxonomy of Directing Semantics for Film Shot Classification , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[21]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[22]  Thierry Pun,et al.  Multimodal Emotion Recognition in Response to Videos , 2012, IEEE Transactions on Affective Computing.

[23]  George Tzanetakis,et al.  Automatic Musical Genre Classification of Audio Signals , 2001, ISMIR.

[24]  Joemon M. Jose,et al.  A comparison of general vs personalised affective models for the prediction of topical relevance , 2010, SIGIR '10.

[25]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[26]  Thomas Sikora,et al.  The MPEG-7 visual standard for content description-an overview , 2001, IEEE Trans. Circuits Syst. Video Technol..

[27]  Kazimierz Choroś Video Shot Selection and Content-Based Scene Detection for Automatic Classification of TV Sports News , 2009 .

[28]  Tomlinson Holman,et al.  Sound for film and television , 1997 .

[29]  Marko Tkalcic,et al.  Affective recommender systems: The role of emotions in recommender systems , 2011 .

[30]  M. K. Luhandjula Studies in Fuzziness and Soft Computing , 2013 .

[31]  Loong Fah Cheong,et al.  Affective understanding in film , 2006, IEEE Trans. Circuits Syst. Video Technol..

[32]  Jurij F. Tasic,et al.  Impact of Implicit and Explicit Affective Labeling on a Recommender System's Performance , 2011, UMAP Workshops.

[33]  Alan Hanjalic,et al.  Extracting Moods from Pictures and Sounds , 2006 .

[34]  George Tzanetakis,et al.  Audio Analysis using the Discrete Wavelet Transform , 2001 .

[35]  Riccardo Leonardi,et al.  Emotional identity of movies , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[36]  Riccardo Leonardi,et al.  Extraction of Significant Video Summaries by Dendrogram Analysis , 2006, 2006 International Conference on Image Processing.

[37]  A. Mehrabian Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in Temperament , 1996 .

[38]  Greg M. Smith Film Structure and the Emotion System , 2003 .

[39]  Yi-Hsuan Yang,et al.  Music emotion ranking , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[40]  Yaser Sheikh,et al.  On the use of computable features for film classification , 2005, IEEE Transactions on Circuits and Systems for Video Technology.

[41]  F. Mora-Camino,et al.  Studies in Fuzziness and Soft Computing , 2011 .

[42]  Chng Eng Siong,et al.  Identify Sports Video Shots with "Happy" or "Sad" Emotions , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[43]  Riccardo Leonardi,et al.  Estimating cinematographic scene depth in movie shots , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[44]  C. Osgood,et al.  The Measurement of Meaning , 1958 .

[45]  Ling-Yu Duan,et al.  Hierarchical movie affective content analysis based on arousal and valence features , 2008, ACM Multimedia.

[46]  I. Daubechies Orthonormal bases of compactly supported wavelets II: variations on a theme , 1993 .

[47]  B. Salt Moving into Pictures: More on Film History, Style, and Analysis , 2006 .

[48]  Gediminas Adomavicius,et al.  Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions , 2005, IEEE Transactions on Knowledge and Data Engineering.

[49]  George Tzanetakis,et al.  Musical genre classification of audio signals , 2002, IEEE Trans. Speech Audio Process..

[50]  Riccardo Leonardi,et al.  A Connotative Space for Supporting Movie Affective Recommendation , 2011, IEEE Transactions on Multimedia.

[51]  M. Kendall,et al.  Rank Correlation Methods , 1949 .

[52]  Riccardo Leonardi,et al.  Affective analysis on patterns of shot types in movies , 2011, 2011 7th International Symposium on Image and Signal Processing and Analysis (ISPA).

[53]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Michael J. Pazzani,et al.  Content-Based Recommendation Systems , 2007, The Adaptive Web.

[55]  P. Valdez,et al.  Effects of color on emotions. , 1994, Journal of experimental psychology. General.

[56]  Joost van de Weijer,et al.  Author Manuscript, Published in "ieee Transactions on Image Processing Edge-based Color Constancy , 2022 .

[57]  Jian-Bo Yang,et al.  Feature selection for support vector regression using probabilistic prediction , 2010, KDD.

[58]  E. Hall,et al.  The Hidden Dimension , 1970 .

[59]  Min Xu,et al.  Affective content analysis in comedy and horror videos by audio emotional event detection , 2005, 2005 IEEE International Conference on Multimedia and Expo.