Gaze-Driven Video Re-Editing

Given the current profusion of devices for viewing media, video content created at one aspect ratio is often viewed on displays with different aspect ratios. Many previous solutions address this problem by retargeting or resizing the video, but a more general solution would re-edit the video for the new display. Our method employs the three primary editing operations: pan, cut, and zoom. We let viewers implicitly reveal what is important in a video by tracking their gaze as they watch the video. We present an algorithm that optimizes the path of a cropping window based on the collected eyetracking data, finds places to cut, and computes the size of the cropping window. We present results on a variety of video clips, including close-up and distant shots, and stationary and moving cameras. We conduct two experiments to evaluate our results. First, we eyetrack viewers on the result videos generated by our algorithm, and second, we perform a subjective assessment of viewer preference. These experiments show that viewer gaze patterns are similar on our result videos and on the original video clips, and that viewers prefer our results to an optimized crop-and-warp algorithm.

[1]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[2]  Steven K. Feiner,et al.  Computer graphics: principles and practice (2nd ed.) , 1990 .

[3]  Yoshio Ohno,et al.  Computer Graphics : Principles and Practice, 2nd edition, J.D. Foley, A.van Dam, S.K. Feiner, J.F. Hughes, Addison-Wesley, 1990 , 1991 .

[4]  Steven D. Katz,et al.  Shot by shot , 1996 .

[5]  Edgar Erdfelder,et al.  GPOWER: A general power analysis program , 1996 .

[6]  Thomas Ertl,et al.  Computer Graphics - Principles and Practice, 3rd Edition , 2014 .

[7]  DeCarloDoug,et al.  Stylization and abstraction of photographs , 2002 .

[8]  Douglas DeCarlo,et al.  Stylization and abstraction of photographs , 2002, ACM Trans. Graph..

[9]  John Hart,et al.  ACM Transactions on Graphics , 2004, SIGGRAPH 2004.

[10]  Mohan S. Kankanhalli,et al.  Video content representation on tiny devices , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[11]  David Salesin,et al.  Gaze-based interaction for semi-automatic photo cropping , 2006, CHI.

[12]  Michael Gleicher,et al.  Video retargeting: automating pan and scan , 2006, MM '06.

[13]  M. Angela Sasse,et al.  How low can you go? The effect of low resolutions on shot types in mobile TV , 2006, Multimedia Tools and Applications.

[14]  Larry S. Davis,et al.  Multi-scale video cropping , 2007, ACM Multimedia.

[15]  S. Avidan,et al.  Seam carving for content-aware image resizing , 2007, SIGGRAPH 2007.

[16]  Jiaya Jia,et al.  Active Window Oriented Dynamic Video Retargeting , 2007 .

[17]  Eli Peli,et al.  Where people look when watching movies: Do all viewers look at the same place? , 2007, Comput. Biol. Medicine.

[18]  Ariel Shamir,et al.  Improved seam carving for video retargeting , 2008, ACM Trans. Graph..

[19]  Christel Chamaret,et al.  Attention-based video reframing: Validation using eye-tracking , 2008, 2008 19th International Conference on Pattern Recognition.

[20]  Hermann Ney,et al.  Pan, zoom, scan — Time-coherent, trained automatic video cropping , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  J. Henderson,et al.  Edit Blindness: The relationship between attention and global change blindness in dynamic scenes. , 2008 .

[22]  O. Sorkine,et al.  Optimized scale-and-stretch for image resizing , 2008, SIGGRAPH 2008.

[23]  B. Velichkovsky,et al.  Eye typing in application: A comparison of two interfacing systems with ALS patients , 2008 .

[24]  O. Sorkine-Hornung,et al.  Optimized scale-and-stretch for image resizing , 2008, SIGGRAPH Asia '08.

[25]  H. Seidel,et al.  Motion-aware temporal coherence for video resizing , 2009, SIGGRAPH 2009.

[26]  Olga Sorkine-Hornung,et al.  Visual media retargeting , 2009, SIGGRAPH ASIA Courses.

[27]  Hans-Peter Seidel,et al.  Motion-aware temporal coherence for video resizing , 2009, ACM Trans. Graph..

[28]  Markus H. Gross,et al.  A system for retargeting of streaming video , 2009, ACM Trans. Graph..

[29]  Xueqing Li,et al.  Warp propagation for video resizing , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[30]  John M. Henderson,et al.  Clustering of Gaze During Dynamic Scene Viewing is Predicted by Motion , 2011, Cognitive Computation.

[31]  Thomas Martinetz,et al.  Variability of eye movements when viewing dynamic natural scenes. , 2010, Journal of vision.

[32]  O. Sorkine,et al.  Motion-based video retargeting with optimized crop-and-warp , 2010, ACM Trans. Graph..

[33]  Mohan S. Kankanhalli,et al.  Automated aesthetic enhancement of videos , 2010, ACM Multimedia.

[34]  Tong-Yee Lee,et al.  Motion-based video retargeting with optimized crop-and-warp , 2010, SIGGRAPH 2010.

[35]  Daniel Cohen-Or,et al.  Optimizing Photo Composition , 2010, Comput. Graph. Forum.

[36]  Wolfgang Effelsberg,et al.  Algorithms for video retargeting , 2010, Multimedia Tools and Applications.

[37]  Mohan S. Kankanhalli,et al.  Video retargeting for aesthetic enhancement , 2010, ACM Multimedia.

[38]  John Paulin Hansen,et al.  Evaluation of a low-cost open-source gaze tracker , 2010, ETRA.

[39]  Diego Gutierrez,et al.  Using eye-tracking to assess different image retargeting methods , 2011, APGV '11.

[40]  A Aldo Faisal,et al.  Ultra-low cost eyetracking as an high-information throughput alternative to BMIs , 2011, BMC Neuroscience.

[41]  Tong-Yee Lee,et al.  Scalable and coherent video resizing with per-frame optimization , 2011, SIGGRAPH 2011.

[42]  N. Spruston,et al.  Slow integration leads to persistent action potential firing in distal axons of coupled interneurons , 2011, Nature Neuroscience.

[43]  L. Itti,et al.  Mechanisms of top-down attention , 2011, Trends in Neurosciences.

[44]  Christopher G. Healey,et al.  On the limits of resolution and visual angle in visualization , 2012, TAP.

[45]  Christof Koch,et al.  Learning visual saliency by combining feature maps in a nonlinear manner using AdaBoost. , 2012, Journal of vision.

[46]  Lihi Zelnik-Manor,et al.  Crowdsourcing Gaze Data Collection , 2012, ArXiv.

[47]  Yaser Sheikh,et al.  Inferring artistic intention in comic art through viewer gaze , 2012, SAP.

[48]  Frédo Durand,et al.  A Benchmark of Computational Models of Saliency to Predict Human Fixations , 2012 .

[49]  Harish Katti,et al.  Online Estimation of Evolving Human Visual Interest , 2014, TOMM.

[50]  John Preston Isenhour,et al.  in Film Editing , 2016 .