Video retargeting with nonlinear spatial-temporal saliency fusion

Video retargeting (resolution adaptation) is a challenging problem for its highly subjective nature. In this paper, a nonlinear saliency fusing approach, that considers human perceptual characteristics for automatic video retargeting, is being proposed. First, we incorporate features from phase spectrum of quaternion Fourier Transform (PQFT) in spatial domain and global motion residual based on matched feature points by the Kanade-Lucas-Tomasi (KLT) tracker in temporal domain. In addition, under a cropping-and-scaling retargeting framework, we propose content-aware information loss metrics and a hierarchical search to find optimal cropping window parameters. Results show the success of our approach on detecting saliency regions and retargeting on images and videos.

[1]  Denis Simakov,et al.  Summarizing visual data using bidirectional similarity , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Liming Zhang,et al.  Spatio-temporal Saliency detection using phase spectrum of quaternion fourier transform , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Stephen J. Sangwine,et al.  Hypercomplex Fourier Transforms of Color Images , 2001, IEEE Transactions on Image Processing.

[4]  Peter J. Rousseeuw,et al.  Robust regression and outlier detection , 1987 .

[5]  Daniel Cohen-Or,et al.  Non-homogeneous Content-driven Video-retargeting , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[6]  Michael Gleicher,et al.  Video retargeting: automating pan and scan , 2006, MM '06.

[7]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Liqing Zhang,et al.  Saliency Detection: A Spectral Residual Approach , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Gang Hua,et al.  Efficient Scale-Space Spatiotemporal Saliency Tracking for Distortion-Free Video Retargeting , 2009, ACCV.

[10]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[11]  Yu Huang,et al.  Video retargeting: A visual-friendly dynamic programming approach , 2010, 2010 IEEE International Conference on Image Processing.

[12]  Trevor Darrell,et al.  Combining object and feature dynamics in probabilistic tracking , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[13]  Peter J. Rousseeuw,et al.  Robust Regression and Outlier Detection , 2005, Wiley Series in Probability and Statistics.

[14]  Stephen J. Sangwine,et al.  Hypercomplex Fourier Transforms of Color Images , 2007, IEEE Trans. Image Process..

[15]  Ariel Shamir,et al.  Seam Carving for Content-Aware Image Resizing , 2007, ACM Trans. Graph..