Movie Genre Classification by Exploiting MEG Brain Signals

Genre classification is an essential part of multimedia content recommender systems. In this study, we provide experimental evidence for the possibility of performing genre classification based on brain recorded signals. The brain decoding paradigm is employed to classify magnetoencephalography (MEG) data presented in [1] to four genre classes: Comedy, Romantic, Drama, and Horror. Our results show that: 1) there is a significant correlation between audio-visual features of movies and corresponding brain signals specially in the visual and temporal lobes; 2) the genre of movie clips can be classified with an accuracy significantly over the chance level using the MEG signal. On top of that we show that the combination of multimedia features and MEG-based features achieves the best accuracy. Our study provides a primary step towards user-centric media content retrieval using brain signals.

[1]  Diane J. Cook,et al.  Using Closed Captions and Visual Features to Classify Movies by Genre , 2006 .

[2]  Subramanian Ramanathan,et al.  DECAF: MEG-Based Multimodal Database for Decoding Affective Physiological Responses , 2015, IEEE Transactions on Affective Computing.

[3]  F. Tong,et al.  Decoding the visual and subjective contents of the human brain , 2005, Nature Neuroscience.

[4]  Robert Oostenveld,et al.  FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data , 2010, Comput. Intell. Neurosci..

[5]  Masaru Sugano,et al.  Shot genre classification using compressed audio-visual features , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[6]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[7]  Keiji Tanaka,et al.  Optical Imaging of Functional Organization in the Monkey Inferotemporal Cortex , 1996, Science.

[8]  Frank Tong,et al.  Decoding motion direction from activity in human visual cortex , 2010 .

[9]  R. Fisher.  Statistical methods for research workers , 1927, Protoplasma.

[10]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[11]  Xiangjian He,et al.  Hierarchical affective content analysis in arousal and valence dimensions , 2013, Signal Process..

[12]  Wen-Hsing Hsu,et al.  A Film Classifier Based on Low-level Visual Features , 2007, 2007 IEEE 9th Workshop on Multimedia Signal Processing.

[13]  Jeho Nam,et al.  Audio-visual content-based violent scene characterization , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[14]  Min Xu,et al.  Affective content analysis in comedy and horror videos by audio emotional event detection , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[15]  Mohammad Soleymani,et al.  A Bayesian framework for video affective representation , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[16]  T. Carlson,et al.  High temporal resolution decoding of object position and category. , 2011, Journal of vision.

[17]  James M. Rehg,et al.  Movie genre classification via scene categorization , 2010, ACM Multimedia.

[18]  David D. Cox,et al.  Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex , 2003, NeuroImage.

[19]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[20]  K. Obermayer,et al.  Geometry of orientation and ocular dominance columns in monkey striate cortex , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[21]  Yaser Sheikh,et al.  On the use of computable features for film classification , 2005 .

[22]  Ishwar K. Sethi,et al.  Classification of general audio data for content-based retrieval , 2001, Pattern Recognit. Lett..

[23]  Keiji Tanaka.  Mechanisms of visual object recognition: monkey and human studies , 1997, Current Opinion in Neurobiology.

[24]  Mohammad Soleymani,et al.  Affective Characterization of Movie Scenes Based on Multimedia Content Analysis and User's Physiological Emotional Responses , 2008, 2008 Tenth IEEE International Symposium on Multimedia.

[25]  Ling-Yu Duan,et al.  Hierarchical movie affective content analysis based on arousal and valence features , 2008, ACM Multimedia.

[26]  G. Rees,et al.  Neuroimaging: Decoding mental states from brain activity in humans , 2006, Nature Reviews Neuroscience.