Visual Movie Analytics

The analysis of inherent structures of movies plays an important role in studying stylistic devices and specific, content-related questions. Examples are the analysis of personal constellations in movie scenes, dialogue-based content analysis, or the investigation of image-based features. We provide a visual analytics approach that supports the analytical reasoning process to derive higher level insights about the content on a semantic level. Combining automatic methods for semantic scene analysis based on script and subtitle text, we perform a low-level analysis of the data automatically. Our approach features an interactive visualization that allows a multilayer interpretation of descriptive features to characterize movie content. For semantic analysis, we extract scene information from movie scripts and match them with the corresponding subtitles. With text- and image-based query techniques, we facilitate an interactive comparison of different movie scenes on an image and on a semantic level. We demonstrate how our approach can be applied for content analysis on a popular Hollywood movie.

[1]  Nadir Weibel,et al.  ChronoViz: a system for supporting navigation of time-coded data , 2011, CHI Extended Abstracts.

[2]  Klaus Schöffmann,et al.  The video explorer: a tool for navigation and searching within a single video based on fast content analysis , 2010, MMSys '10.

[3]  Jeho Nam,et al.  Dynamic video summarization and visualization , 1999, MULTIMEDIA '99.

[4]  Li Li,et al.  A Survey on Visual Content-Based Video Indexing and Retrieval , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[5]  Michael Ryan,et al.  An Introduction to Film Analysis: Technique and Meaning in Narrative Film , 2012 .

[6]  Noah A. Smith,et al.  Frame-Semantic Parsing , 2014, CL.

[7]  Sheng Tang,et al.  A hierarchical framework for movie content analysis: Let computers watch films like humans , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[8]  Miriam R. L. Petruck FRAME SEMANTICS , 1996 .

[9]  Min Chen,et al.  SoundRiver: Semantically‐Rich Sound Illustration , 2010, Comput. Graph. Forum.

[10]  I. Kawachi,et al.  Women and smoking in Hollywood movies: a content analysis. , 2000, American journal of public health.

[11]  Shingo Uchihashi,et al.  An interactive comic book presentation for exploring video , 2000, CHI.

[12]  Klaus Schöffmann,et al.  Video Interaction Tools , 2015, ACM Comput. Surv..

[13]  Wolfgang Effelsberg,et al.  The MoCA Workbench: support for creativity in movie content analysis , 1996, Proceedings of the Third IEEE International Conference on Multimedia Computing and Systems.

[14]  Jhing-Fa Wang,et al.  A Novel Video Summarization Based on Mining the Story-Structure and Semantic Relations Among Concept Entities , 2009, IEEE Transactions on Multimedia.

[15]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Alan Hanjalic,et al.  Shot-boundary detection: unraveled and resolved? , 2002, IEEE Trans. Circuits Syst. Video Technol..

[17]  Marco Pellegrini,et al.  VISTO: visual storyboard for web video browsing , 2007, CIVR '07.

[18]  Wei-Ta Chu,et al.  RoleNet: Movie Analysis from the Perspective of Social Networks , 2009, IEEE Transactions on Multimedia.

[19]  Chia-Hung Yeh,et al.  Techniques for movie content analysis and skimming: tutorial and overview on video abstraction techniques , 2006, IEEE Signal Processing Magazine.

[20]  Takeo Kanade,et al.  Video skimming and characterization through the combination of image and language understanding , 1998, Proceedings 1998 IEEE International Workshop on Content-Based Access of Image and Video Database.

[21]  Karrie Karahalios,et al.  VCode and VData: illustrating a new framework for supporting the video annotation workflow , 2008, AVI '08.

[22]  William Ribarsky,et al.  Multimedia Analysis + Visual Analytics = Multimedia Analytics , 2010, IEEE Computer Graphics and Applications.

[23]  John S. Boreczky,et al.  Comparison of video shot boundary detection techniques , 1996, Electronic Imaging.

[24]  Wei-Ta Chu,et al.  Action movies segmentation and summarization based on tempo analysis , 2004, MIR '04.

[25]  Georges Linarès,et al.  Static and dynamic video summaries , 2011, MM '11.

[26]  Kuo C. Jay Video Content Analysis Using Multimodal Information: For Movie Content Extraction, Indexing and Representation , 2003 .

[27]  Gennady L. Andrienko,et al.  Exploratory spatio-temporal visualization: an analytical review , 2003, J. Vis. Lang. Comput..

[28]  Zhu Liu,et al.  Multimedia content analysis-using both audio and visual clues , 2000, IEEE Signal Process. Mag..

[29]  Marcel Worring,et al.  Where Is the User in Multimedia Retrieval? , 2012, IEEE Multim..

[30]  Ba Tu Truong,et al.  Video abstraction: A systematic review and classification , 2007, TOMCCAP.

[31]  Anan Liu,et al.  Watching, Thinking, Reacting: A Human-Centered Framework for Movie Content Analysis , 2010, J. Digit. Content Technol. its Appl..

[32]  Harry W. Agius,et al.  Video summarisation: A conceptual framework and survey of the state of the art , 2008, J. Vis. Commun. Image Represent..

[33]  László Böszörményi,et al.  State-of-the-art and future challenges in video scene detection: a survey , 2013, Multimedia Systems.

[34]  Min Chen,et al.  State of the Art Report on Video‐Based Graphics and Video Visualization , 2012, Comput. Graph. Forum.

[35]  Adel M. Alimi,et al.  IM(S)2: Interactive movie summarization system , 2010, J. Vis. Commun. Image Represent..

[36]  Marco Pellegrini,et al.  STIMO: STIll and MOving video storyboard for the web scenario , 2009, Multimedia Tools and Applications.

[37]  Nuno Vasconcelos,et al.  Statistical models of video structure for content analysis and characterization , 2000, IEEE Trans. Image Process..

[38]  Josef Ruppenhofer,et al.  FrameNet II: Extended theory and practice , 2006 .

[39]  Sandra Baringer,et al.  Hollywood's Indian: The Portrayal of the Native American in Film , 1998 .

[40]  Takeo Kanade,et al.  Intelligent Access to Digital Video: Informedia Project , 1996, Computer.

[41]  Andreas Dieberger,et al.  Hierarchical brushing in a collection of video data , 2001, Proceedings of the 34th Annual Hawaii International Conference on System Sciences.

[42]  Ben Taskar,et al.  Movie/Script: Alignment and Parsing of Video and Text Transcription , 2008, ECCV.

[43]  Mengchen Liu,et al.  StoryFlow: Tracking the Evolution of Stories , 2013, IEEE Transactions on Visualization and Computer Graphics.

[44]  Alan Hanjalic,et al.  Automated high-level movie segmentation for advanced video-retrieval systems , 1999, IEEE Trans. Circuits Syst. Video Technol..

[45]  Changsheng Xu,et al.  Character-based movie summarization , 2010, ACM Multimedia.

[46]  Atsuo Yoshitaka,et al.  A Survey on Content-Based Retrieval for Multimedia Databases , 1999, IEEE Trans. Knowl. Data Eng..

[47]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.