Optimized Comics-Based Storytelling for Temporal Image Sequences

We propose a system to transform any temporal image sequence into a comics-based presentation, as an effective and interesting storytelling manner. Three main components, including page allocation, layout selection, and speech balloon placement, are respectively formulated as optimization problems, and systematic approaches are proposed to find solutions. Page allocation is viewed as a labeling problem, and the best solution is determined by the genetic algorithm. Importance values of images and predefined layouts are both represented in vector forms, and the best layout is selected by finding the best match between vectors. Feasible solutions of speech balloons constitute a solution space, and the best solution that jointly describes the best locations of all balloons in a page is determined by the particle swarm optimization algorithm. Objective evaluation and subjective evaluation are designed from various perspectives to demonstrate effectiveness and superiority of the proposed system.

[1]  Shi-Min Hu,et al.  Global contrast based salient region detection , 2011, CVPR 2011.

[2]  Ruck Thawonmas,et al.  Pomics: A Computer-Aided Storytelling System with Automatic Picture-to-Comics Composition , 2012, 2012 Conference on Technologies and Applications of Artificial Intelligence.

[3]  Wei-Ta Chu,et al.  Enabling portable animation browsing by transforming animations into comics , 2012, IMMPD '12.

[4]  Alessandro Vinciarelli,et al.  Broadcast news story segmentation using social network analysis and hidden markov models , 2007, ACM Multimedia.

[5]  Shingo Uchihashi,et al.  Video Manga: generating semantically meaningful video summaries , 1999, MULTIMEDIA '99.

[6]  Gregory J. Wolff,et al.  Storytelling with digital photographs , 2000, CHI.

[7]  Boon-Lock Yeo,et al.  Video visualization for compact presentation and fast browsing of pictorial content , 1997, IEEE Trans. Circuits Syst. Video Technol..

[8]  Kwan-Liu Ma,et al.  Scientific Storytelling Using Visualization , 2012, IEEE Computer Graphics and Applications.

[9]  Yael Pritch,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2008 1 Non-Chronological Video , 2022 .

[10]  Shiyang Lu,et al.  Keypoint-Based Keyframe Selection , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[11]  Meng Wang,et al.  Movie2Comics: a feast of multimedia artwork , 2010, ACM Multimedia.

[12]  Emi Myodo,et al.  Automatic comic-like image layout system preserving image order and important regions , 2011, MM '11.

[13]  Andreas Girgensohn,et al.  A fast layout algorithm for visual video summaries , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[14]  Tao Mei,et al.  Dynamic Video Collage , 2010, MMM.

[15]  Luis Herranz,et al.  Effective Comic-Like Representations with Embedded Regions of Interest , 2012, PCM.

[16]  Ruck Thawonmas,et al.  Automatic storytelling in comics: a case study on World of Warcraft , 2009, CHI Extended Abstracts.

[17]  Rynson W. H. Lau,et al.  Automatic stylistic manga layout , 2012, ACM Trans. Graph..

[18]  Wei-Ta Chu,et al.  Optimized speech balloon placement for automatic comics generation , 2013, IMMPD '13.

[19]  Janko Calic,et al.  Efficient Layout of Comic-Like Video Summaries , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[20]  Lihi Zelnik-Manor,et al.  Context-Aware Saliency Detection , 2012, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Hwan-Gue Cho,et al.  An Automated Procedure for Word Balloon Placement in Cinema Comics , 2006, ISVC.

[22]  Jock D. Mackinlay,et al.  Storytelling: The Next Step for Visualization , 2013, Computer.

[23]  Masahiro Toyoura,et al.  Film Comic Reflecting Camera-Works , 2012, MMM.

[24]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[25]  Luigi Troiano,et al.  Web Form Page in Mobile Devices - Optimization of Layout with a Simple Genetic Algorithm , 2009, ICEIS.

[26]  Mubarak Shah,et al.  Scene detection in Hollywood movies and TV shows , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[27]  Meng Wang,et al.  Movie2Comics: Towards a Lively Video Content Presentation , 2012, IEEE Transactions on Multimedia.

[28]  Michael Austin.  Useful Fictions: Evolution, Anxiety, and the Origins of Literature , 2011 .

[29]  José María Martínez Sanchez,et al.  Scalable Comic-Like Video Summaries and Layout Disturbance , 2012, IEEE Transactions on Multimedia.

[30]  James Ze Wang,et al.  The Story Picturing Engine---a system for automatic text illustration , 2006, TOMCCAP.

[31]  Hiroaki Tobita.  DigestManga: interactive movie summarizing through comic visualization , 2010, CHI EA '10.

[32]  Mary Czerwinski,et al.  PhotoTOC: automatic clustering for browsing personal photographs , 2003, Fourth International Conference on Information, Communications and Signal Processing, 2003 and the Fourth Pacific Rim Conference on Multimedia. Proceedings of the 2003 Joint.

[33]  Jun-Cheng Chen,et al.  Tiling slideshow , 2006, MM '06.

[34]  Hiroaki Tobita.  Comic Engine: interactive system for creating and browsing comic books with attention cuing , 2010, AVI.

[35]  Masahiro Toyoura,et al.  Film Comic Generation with Eye Tracking , 2013, MMM.

[36]  Jeffrey Heer,et al.  Narrative Visualization: Telling Stories with Data , 2010, IEEE Transactions on Visualization and Computer Graphics.

[37]  Raimondo Schettini,et al.  Dynamic storyboards for video content summarization , 2006, MIR '06.

[38]  Gjermund Bø Brabrand.  Dynamic Layout Optimization for Newspaper Web Sites using a Controlled Annealed Genetic Algorithm , 2008 .

[39]  Zhu Liu,et al.  Multimedia content analysis-using both audio and visual clues , 2000, IEEE Signal Process. Mag..