Dynamic Time Warping of Multimodal Signals for Detecting Highlights in Movies

Affective computing has strong ties with literature and film studies, e.g. text sentiment analysis, affective tagging of movies. In this work we report on recent findings towards identifying highlights in movies on the basis of the synchronization of physiological and behavioral signals of people. The proposed architecture is utilizing dynamic time warping for measuring the distance among the multimodal signals of pairs of spectators. The reported results suggest that this distance can be indicative for the dynamics and existence of aesthetic moments in movies.

[1]  Srinivasan Jayaraman,et al.  Human Electrocardiogram for Biometrics Using DTW and FLDA , 2010, 2010 20th International Conference on Pattern Recognition.

[2]  Vinícius M. A. de Souza,et al.  Time Series Classification Using Compression Distance of Recurrence Plots , 2013, 2013 IEEE 13th International Conference on Data Mining.

[3]  Gilles Deleuze,et al.  The Time-Image , 1989 .

[4]  Elizabeth Cowie,et al.  The World Viewed , 2015, A Companion to Contemporary Documentary Film.

[5]  Sarah E. Worth,et al.  Philosophy of Mass Art , 2000 .

[6]  Emily Mower Provost,et al.  Say Cheese vs. Smile: Reducing Speech-Related Variability for Facial Emotion Recognition , 2014, ACM Multimedia.

[7]  M. Uzun.  The nature of fiction , 2007 .

[8]  Noël Carroll,et al.  A Philosophy of Mass Art , 1999 .

[9]  Gregory Currie,et al.  Image and mind : film, philosophy and cognitive science , 1995 .

[10]  Billur Barshan,et al.  Automated evaluation of physical therapy exercises using multi-template dynamic time warping on wearable sensor signals , 2014, Comput. Methods Programs Biomed..

[11]  Mohammad Soleymani,et al.  Highlight Detection in Movie Scenes Through Inter-users, Physiological Linkage , 2013, Social Media Retrieval.

[12]  Guillaume Chanel,et al.  Identifying aesthetic highlights in movies from clustering of physiological and behavioral signals , 2015, 2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX).

[13]  Andrew P. Bradley,et al.  The use of the area under the ROC curve in the evaluation of machine learning algorithms , 1997, Pattern Recognit..

[14]  David Bordwell,et al.  Film History: An Introduction , 1994 .

[15]  Touradj Ebrahimi,et al.  Multimedia content analysis for emotional characterization of music video clips , 2013, EURASIP J. Image Video Process..

[16]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[17]  Mohammad Soleymani,et al.  Continuous emotion detection using EEG signals and facial expressions , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).

[18]  Björn W. Schuller,et al.  Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge , 2011, Speech Commun..

[19]  Patrizia Lombardo.  Emotion et souvenir chez Aldo Rossi , 2011 .

[20]  Silvia Chiusano,et al.  Modeling Athlete Performance Using Clustering Techniques , 2010, isecs 2010.

[21]  John V. Guttag,et al.  Weighted Time Warping for Temporal Segmentation of Multi-parameter Physiological Signals , 2011, BIOSIGNALS.

[22]  Meinard Müller,et al.  Information retrieval for music and motion , 2007 .

[23]  Patrizia Lombardo,et al.  Bazin, Bresson and Scorsese: performative power and the impure art of cinema , 2010 .

[24]  K. Walton.  Transparent Pictures: On the Nature of Photographic Realism , 1984, Critical Inquiry.

[25]  Gilles Deleuze,et al.  The movement-image , 1986 .

[26]  Mohamed Chetouani,et al.  Automatic Imitation Assessment in Interaction , 2012, HBU.

[27]  Nicu Sebe,et al.  Looking at the viewer: analysing facial activity to detect personal highlights of multimedia contents , 2010, Multimedia Tools and Applications.

[28]  Darren Mitchell,et al.  Film as art , 2013 .

[29]  R. Stephenson.  A and V , 1962, The British journal of ophthalmology.

[30]  Julien Fleureau,et al.  Affective Benchmarking of Movies Based on the Physiological Responses of a Real Audience , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[31]  Alan Bundy,et al.  Dynamic Time Warping , 1984 .

[32]  Guillaume Chanel,et al.  Assessment of Computer-Supported Collaborative Processes Using Interpersonal Physiological and Eye-Movement Coupling , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[33]  Muttukrishnan Rajarajan,et al.  Exploring Worm Behaviors using DTW , 2014, SIN.

[34]  D. Heeger,et al.  Neurocinematics: The Neuroscience of Film , 2008 .

[35]  I. Newman.  The Philosophy of Horror or Paradoxes of the Heart , 1991 .