Remembering Winter Was Coming: Character-Oriented Video Summaries of TV Series.

Today's popular TV series tend to develop continuous, complex plots spanning several seasons, but are often viewed in controlled and discontinuous conditions. Consequently, most viewers need to be re-immersed in the story before watching a new season. Although discussions with friends and family can help, we observe that most viewers make extensive use of summaries to re-engage with the plot. Automatic generation of video summaries of TV series' complex stories requires, first, modeling the dynamics of the plot and, second, extracting relevant sequences. In this paper, we tackle plot modeling by considering the social network of interactions between the characters involved in the narrative: substantial, durable changes in a major character's social environment suggest a new development relevant for the summary. Once identified, these major stages in each character's storyline can be used as a basis for completing the summary with related sequences. Our algorithm combines such social network analysis with filmmaking grammar to automatically generate character-oriented video summaries of TV series from partially annotated data. We carry out evaluation with a user study in a real-world scenario: a large sample of viewers were asked to rank video summaries centered on five characters of the popular TV series Game of Thrones, a few weeks before the new, sixth season was released. Our results reveal the ability of character-oriented summaries to re-engage viewers in television series and confirm the contributions of modeling the plot content and exploiting stylistic patterns to identify salient sequences.

[1]  Alan Hanjalic,et al.  Affective video content representation and modeling , 2005, IEEE Transactions on Multimedia.

[2]  Christine Sénac,et al.  Toward plot de-interlacing in TV series using scenes clustering , 2012, 2012 10th International Workshop on Content-Based Multimedia Indexing (CBMI).

[3]  Georges Linarès,et al.  Extraction and Analysis of Dynamic Conversational Networks from TV Series , 2018, Social Network Based Big Data Analysis and Applications.

[4]  Changsheng Xu,et al.  Character-based movie summarization , 2010, ACM Multimedia.

[5]  Mark S. Daskin,et al.  Network and Discrete Location: Models, Algorithms and Applications , 1995 .

[6]  Wei-Ta Chu,et al.  RoleNet: Movie Analysis from the Perspective of Social Networks , 2009, IEEE Transactions on Multimedia.

[7]  Wei-Ta Chu,et al.  Action movies segmentation and summarization based on tempo analysis , 2004, MIR '04.

[8]  Xavier Bost.  A storytelling machine? : Automatic video summarization: the case of TV series. (Une machine à raconter des histoires ? / Une machine à raconter des histoires ? : Résumé automatique de vidéos : le cas des séries TV) , 2016 .

[9]  John R. Smith,et al.  Harnessing A.I. for Augmenting Creativity: Application to Movie Trailer Creation , 2017, ACM Multimedia.

[10]  David A. McAllester,et al.  A discriminatively trained, multiscale, deformable part model , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Hans Weda,et al.  Automated summarization of narrative video on a semantic level , 2007 .

[12]  Gérard Plateau,et al.  An algorithm for the solution of the 0–1 knapsack problem , 2005, Computing.

[13]  Georges Linarès,et al.  Narrative smoothing: Dynamic conversational network for the analysis of TV series plots , 2016, 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).

[14]  Sergios Theodoridis,et al.  Music tracking in audio streams from movies , 2008, 2008 IEEE 10th Workshop on Multimedia Signal Processing.

[15]  Gerald Friedland,et al.  Using Artistic Markers and Speaker Identification for Narrative-Theme Navigation of Seinfeld Episodes , 2009, 2009 11th IEEE International Symposium on Multimedia.

[16]  Tanaya Guha,et al.  Computationally deconstructing movie narratives: An informatics approach , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[17]  Gregory Gelly,et al.  Improving Speaker Diarization of TV Series using Talking-Face Detection and Clustering , 2016, ACM Multimedia.

[18]  Egon Balas,et al.  An Algorithm for Large Zero-One Knapsack Problems , 1980, Oper. Res..

[19]  Makarand Tapaswi,et al.  StoryGraphs: Visualizing Character Interactions as a Timeline , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Alan Hanjalic,et al.  Automated high-level movie segmentation for advanced video-retrieval systems , 1999, IEEE Trans. Circuits Syst. Video Technol..

[21]  Benoit Favre,et al.  A Scalable Global Model for Summarization , 2009, ILP 2009.

[22]  Alan F. Smeaton,et al.  Automatically selecting shots for action movie trailers , 2006, MIR '06.

[23]  Ryan T. McDonald.  A Study of Global Inference Algorithms in Multi-document Summarization , 2007, ECIR.

[24]  Boon-Lock Yeo,et al.  Segmentation of Video by Clustering and Graph Analysis , 1998, Comput. Vis. Image Underst..

[25]  Irena Koprinska,et al.  Temporal video segmentation: A survey , 2001, Signal Process. Image Commun..

[26]  Petri Toiviainen,et al.  A Matlab Toolbox for Music Information Retrieval , 2007, GfKl.