Content aware video manipulation

Content aware video manipulation (CAVM) is a method for the analysis and recomposition of video footage, by means of content analysis and adaptive video warping. One main motivation of CAVM is ''video retargeting'', a process that visually alters an existing video while considering the relative importance of its various regions. CAVM video retargeting aims at preserving the viewers' experience by maintaining the information content of important regions in the frame, while altering the video dimensions. Other applications include commercial real-estate allocations, time and space content summary, and content deletion (in both time and spatial domain). In this paper we introduce an efficient algorithm for the implementation of CAVM. It consists of two stages. First, the video is analyzed to detect the importance of each pixel in the frame, based on local saliency, motion detection and object detectors. Then, a transformation manipulates the video content according to the aforementioned analysis and application dependent constraints. The visual performance of the proposed algorithm is demonstrated on a variety of video sequences, and compared to the state-of-the-art in image retargeting.

[1]  Yael Pritch,et al.  Making a Long Video Short: Dynamic Video Synopsis , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[2]  Nebojsa Jojic,et al.  Interactive Montages of Sprites for Indexing and Summarizing Security Video , 2005, CVPR.

[3]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[4]  Michael Gleicher,et al.  Video retargeting: automating pan and scan , 2006, MM '06.

[5]  Ramesh Raskar,et al.  Automatic image retargeting , 2005, MUM '05.

[6]  Ariel Shamir,et al.  Improved seam carving for video retargeting , 2008, ACM Trans. Graph..

[7]  Frédo Durand,et al.  ACM SIGGRAPH Asia 2009 Papers , 2009 .

[8]  Daniel Cohen-Or,et al.  Non-homogeneous Content-driven Video-retargeting , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[9]  Daniel Cohen-Or,et al.  Semi-automatic stereo extraction from video footage , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[10]  Michael Gleicher,et al.  Automatic image retargeting with fisheye-view warping , 2005, UIST.

[11]  Shyang Chang,et al.  Statistical change detection with moments under time-varying illumination , 1998, IEEE Trans. Image Process..

[12]  Benjamin B. Bederson,et al.  Automatic thumbnail cropping and its effectiveness , 2003, UIST '03.

[13]  Ralph R. Martin,et al.  Shrinkability Maps for Content‐Aware Video Resizing , 2008, Comput. Graph. Forum.

[14]  Timothy A. Davis,et al.  A column pre-ordering strategy for the unsymmetric-pattern multifrontal method , 2004, TOMS.

[15]  Olga Sorkine-Hornung,et al.  Optimized scale-and-stretch for image resizing , 2008, SIGGRAPH Asia '08.

[16]  Shih-Fu Chang,et al.  Scene change detection in an MPEG-compressed video sequence , 1995, Electronic Imaging.

[17]  P. Anandan,et al.  Efficient representations of video sequences and their applications , 1996, Signal Process. Image Commun..

[18]  S. Avidan,et al.  Seam carving for content-aware image resizing , 2007, SIGGRAPH 2007.

[19]  Carlos D. Correa,et al.  Dynamic video narratives , 2010, ACM Trans. Graph..

[20]  Adam Finkelstein,et al.  Video tapestries with continuous temporal zoom , 2010, ACM Trans. Graph..

[21]  Daniel Cohen-Or,et al.  Feature-aware texturing , 2006, EGSR '06.

[22]  O. Sorkine,et al.  Motion-based video retargeting with optimized crop-and-warp , 2010, ACM Trans. Graph..

[23]  O. Sorkine,et al.  Optimized scale-and-stretch for image resizing , 2008, SIGGRAPH 2008.

[24]  Jianhua Lu,et al.  A simple and efficient search algorithm for block-matching motion estimation , 1997, IEEE Trans. Circuits Syst. Video Technol..

[25]  Hans-Peter Seidel,et al.  Motion-aware temporal coherence for video resizing , 2009, ACM Trans. Graph..

[26]  Markus H. Gross,et al.  A system for retargeting of streaming video , 2009, ACM Trans. Graph..

[27]  Christof Koch,et al.  Comparison of feature combination strategies for saliency-based visual attention systems , 1999, Electronic Imaging.

[28]  Olga Sorkine-Hornung,et al.  Visual media retargeting , 2009, SIGGRAPH ASIA Courses.