Inferring artistic intention in comic art through viewer gaze

Comics are a compelling, though complex, visual storytelling medium. Researchers are interested in the process of comic art creation to be able to automatically tell new stories, and also, summarize videos and catalog large collections of photographs for example. A primary organizing principle used by artists to lay out the components of comic art (panels, word bubbles, objects inside each panel) is to lead the viewer's attention along a deliberate visual route that reveals the narrative. If artists are successful in leading viewer attention, then their intended visual route would be accessible through recorded viewer attention, i.e., eyetracking data. In this paper, we conduct an experiment to verify if artists are successful in their goal of leading viewer gaze. We eyetrack viewers on images taken from comic books, as well as photographs taken by experts, amateur photographers and a robot. Our data analyses show that there is increased consistency in viewer gaze for comic pictures versus photographs taken by a robot and by amateur photographers, thus confirming that comic artists do indeed direct the flow of viewer attention.

[1]  Iain D. Gilchrist,et al.  Visual correlates of fixation selection: effects of scale and time , 2005, Vision Research.

[2]  Taku Komura,et al.  Automatic Panel Extraction of Color Comic Images , 2007, PCM.

[3]  A. L. I︠A︡rbus.  Eye Movements and Vision , 1967 .

[4]  Marcus Nyström,et al.  A vector-based, multidimensional scanpath similarity measure , 2010, ETRA.

[5]  Ariel Shamir,et al.  Generating comics from 3D interactive computer graphics , 2006, IEEE Computer Graphics and Applications.

[6]  Frédo Durand,et al.  Learning to predict where humans look , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[7]  Laurent Itti,et al.  The role of memory in guiding attention during natural vision. , 2006, Journal of vision.

[8]  Asha Iyer,et al.  Components of bottom-up gaze allocation in natural images , 2005, Vision Research.

[9]  Laurent Itti,et al.  Attention, Movie Cuts, and Natural Vision: A Functional Perspective , 2008 .

[10]  S. Chiba,et al.  Dynamic programming algorithm optimization for spoken word recognition , 1978 .

[11]  W. Eisner.  Comics and Sequential Art , 1985 .

[12]  Takeo Kanade,et al.  Image matching in large scale indoor environment , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[13]  Masahiro Toyoura,et al.  Film Comic Reflecting Camera-Works , 2012, MMM.

[14]  A. L. Yarbus,et al.  Eye Movements and Vision , 1967, Springer US.

[15]  William H. Bares.  Panel Beat: Layout and Timing of Comic Panels , 2008, Smart Graphics.

[16]  Kohei Arai,et al.  Automatic E-Comic Content Adaptation , 2010 .

[17]  Ana Paiva,et al.  So tell me what happened: turning agent-based interactive drama into comics , 2008, AAMAS.

[18]  Takahide Omori・Takeharu Igaki.  Eye catchers in comics: Controlling eye movements in reading pictorial and textual media , 2005 .

[19]  David S Wooding,et al.  Eye movements of large populations: II. Deriving regions of interest, coverage, and similarity using fixation maps , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[20]  Emi Myodo,et al.  Automatic comic-like image layout system preserving image order and important regions , 2011, MM '11.

[21]  Meng Wang,et al.  Movie2Comics: a feast of multimedia artwork , 2010, ACM Multimedia.

[22]  Thomas Martinetz,et al.  Variability of eye movements when viewing dynamic natural scenes. , 2010, Journal of vision.

[23]  Masahiro Toyoura,et al.  Using eye-tracking data for automatic film comic creation , 2012, ETRA '12.

[24]  C. Koch,et al.  Computational modelling of visual attention , 2001, Nature Reviews Neuroscience.

[25]  Hiroaki Tobita.  Comic Engine: interactive system for creating and browsing comic books with attention cuing , 2010, AVI.

[26]  Harish Katti,et al.  Affective Video Summarization and Story Board Generation Using Pupillary Dilation and Eye Gaze , 2011, 2011 IEEE International Symposium on Multimedia.

[27]  H. Ritter,et al.  Disambiguating Complex Visual Information: Towards Communication of Personal Views of a Scene , 1996, Perception.