Schematic storyboarding for video visualization and editing

We present a method for visualizing short video clips in a single static image, using the visual language of storyboards. These schematic storyboards are composed from multiple input frames and annotated using outlines, arrows, and text describing the motion in the scene. The principal advantage of this storyboard representation over standard representations of video -- generally either a static thumbnail image or a playback of the video clip in its entirety -- is that it requires only a moment to observe and comprehend but at the same time retains much of the detail of the source video. Our system renders a schematic storyboard layout based on a small amount of user interaction. We also demonstrate an interaction technique to scrub through time using the natural spatial dimensions of the storyboard. Potential applications include video editing, surveillance summarization, assembly instructions, composition of graphic novels, and illustration of camera technique for film studies.

[1]  Michael Rubin Droidmaker: George Lucas and the Digital Revolution , 2005 .

[2]  Andrew Zisserman,et al.  Learning Layered Motion Segmentation of Video , 2005, ICCV.

[3]  Chi-Keung Tang,et al.  Eliminating structure and intensity misalignment in image stitching , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[4]  P. Perona,et al.  Squaring the circle in panoramas , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[5]  Denis Simakov,et al.  Space-time scene manifolds , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[6]  Daniel Cohen-Or,et al.  Action synopsis: pose selection and illustration , 2005, ACM Trans. Graph..

[7]  Irfan A. Essa,et al.  Video-based nonphotorealistic and expressive illustration of motion , 2005, International 2005 Computer Graphics.

[8]  Dominique Barba,et al.  Grouping video shots into scenes based on 1D mosaic descriptors , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[9]  David Salesin,et al.  Interactive digital photomontage , 2004, ACM Trans. Graph..

[10]  Vladimir Kolmogorov,et al.  "GrabCut": interactive foreground extraction using iterated graph cuts , 2004, ACM Trans. Graph..

[11]  Loong Fah Cheong,et al.  Shot Change Detection Using Scene-Based Constraint , 2001, Multimedia Tools and Applications.

[12]  Brendan J. Frey,et al.  Joint Design of Data Analysis Algorithms and User Interfaces for Video Applications , 2003 .

[13]  Kunio Kondo,et al.  Cartoon blur: nonphotorealistic motion blur , 2003, Proceedings Computer Graphics International 2003.

[14]  Jürgen Döllner,et al.  Dynamic Glyphs - Depicting Dynamics in Images of 3D Scenes , 2003, Smart Graphics.

[15]  William T. Freeman,et al.  Shape-time photography , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[16]  J. Cutting Representing Motion in a Static Image: Constraints and Parallels in Art, Science, and Popular Culture , 2002, Perception.

[17]  King Ngi Ngan,et al.  An Object-Based Shot Boundary Detection Using Edge Tracing and Tracking , 2001, J. Vis. Commun. Image Represent..

[18]  Ying Li,et al.  An Overview of Video Abstraction Techniques , 2001 .

[19]  Maic Masuch,et al.  Speedlines: depicting motion in motionless pictures , 1999, SIGGRAPH '99.

[20]  John Hart The Art of the Storyboard: Storyboarding for Film, TV, and Animation , 1999 .

[21]  Seong-Whan Lee,et al.  Automatic video parsing using shot boundary detection and camera operation analysis , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[22]  Michal Irani,et al.  Video indexing based on mosaic representations , 1998, Proc. IEEE.

[23]  Yukinobu Taniguchi,et al.  PanoramaExcerpts: extracting and packing panoramas for video browsing , 1997, MULTIMEDIA '97.

[24]  David Salesin,et al.  Multiperspective panoramas for cel animation , 1997, SIGGRAPH.

[25]  Walter Bender,et al.  Salient Stills: Process and Practice , 1996, IBM Syst. J..

[26]  Walter Bender,et al.  Salient video stills: content and context preserved , 1993, MULTIMEDIA '93.

[27]  A. D. Manning,et al.  Understanding Comics: The Invisible Art , 1993 .

[28]  S. Umeyama,et al.  Least-Squares Estimation of Transformation Parameters Between Two Point Patterns , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Steven D. Katz Film directing, shot by shot : visualizing from concept to screen , 1991 .

[30]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using orthonormal matrices , 1988 .

[31]  D. F. Fisher,et al.  Perception and pictorial representation , 1979 .