A Survey on Content-Aware Image and Video Retargeting

This survey introduces the current state of the art in image and video retargeting and describes important ideas and technologies that have influenced the recent work. Retargeting is the process of adapting an image or video from one screen resolution to another to fit different displays, for example, when watching a wide screen movie on a normal television screen or a mobile device. As there has been considerable work done in this field already, this survey provides an overview of the techniques. It is meant to be a starting point for new research in the field. We include explanations of basic terms and operators, as well as the basic workflow of the different methods.

[1]  Wolfgang Effelsberg,et al.  Warping-based video retargeting for stereoscopic video , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[2]  Diego Gutierrez,et al.  Objective Quality Prediction of Image Retargeting Algorithms , 2017, IEEE Transactions on Visualization and Computer Graphics.

[3]  Lin Mei,et al.  Video Retargeting Based on Spatiotemporal Saliency Model , 2014 .

[4]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[5]  HongJiang Zhang,et al.  Contrast-based image attention analysis by using fuzzy growing , 2003, MULTIMEDIA '03.

[6]  Wolfgang Effelsberg,et al.  GPU video retargeting with parallelized SeamCrop , 2014, MMSys '14.

[7]  Wolfgang Effelsberg,et al.  Saliency detection for stereoscopic video , 2013, MMSys.

[8]  Michael Gleicher,et al.  Video retargeting: automating pan and scan , 2006, MM '06.

[9]  Weisi Lin,et al.  Objective Quality Assessment for Image Retargeting Based on Perceptual Geometric Distortion and Information Loss , 2014, IEEE Journal of Selected Topics in Signal Processing.

[10]  David Salesin,et al.  Gaze-based interaction for semi-automatic photo cropping , 2006, CHI.

[11]  Qi Tian,et al.  Codebook Guided Feature-Preserving for Recognition-Oriented Image Retargeting , 2017, IEEE Transactions on Image Processing.

[12]  Wonjun Kim,et al.  Spatiotemporal Saliency Detection Using Textural Contrast and Its Applications , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[13]  Chang Wen Chen,et al.  Deformable Shape Preserving Video Retargeting With Salient Curve Matching , 2014, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[14]  Jeffrey Ho,et al.  CASAIR: Content and Shape-Aware Image Retargeting and Its Applications , 2016, IEEE Transactions on Image Processing.

[15]  Tong-Yee Lee,et al.  Summarization-Based Image Resizing by Intelligent Object Carving , 2014, IEEE Transactions on Visualization and Computer Graphics.

[16]  Huchuan Lu,et al.  Adaptive Metric Learning for Saliency Detection , 2015, IEEE Transactions on Image Processing.

[17]  Xiaodong Wu,et al.  Optimal multiple surfaces searching for video/image resizing - a graph-theoretic approach , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[18]  Sabine Süsstrunk,et al.  Frequency-tuned salient region detection , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Yi Liu,et al.  A retargeting method for stereoscopic 3D video , 2015, Computational Visual Media.

[20]  Henrik I. Christensen,et al.  Computational visual attention systems and their cognitive foundations: A survey , 2010, TAP.

[21]  Wu-chi Feng,et al.  Aesthetics-Based Stereoscopic Photo Cropping for Heterogeneous Displays , 2012, IEEE Transactions on Multimedia.

[22]  Touradj Ebrahimi,et al.  Perceptual Video Compression: A Survey , 2012, IEEE Journal of Selected Topics in Signal Processing.

[23]  Wei Tsang Ooi,et al.  Crowdsourced automatic zoom and scroll for video retargeting , 2010, ACM Multimedia.

[24]  Lei Guo,et al.  An Object-Oriented Visual Saliency Detection Framework Based on Sparse Coding Representations , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[25]  Ramesh Raskar,et al.  Automatic image retargeting , 2005, MUM '05.

[26]  Matthew Turk,et al.  On Preserving Structure in Stereo Seam Carving , 2015, 2015 International Conference on 3D Vision.

[27]  Olga Sorkine-Hornung,et al.  Robust Image Retargeting via Axis‐Aligned Deformation , 2012, Comput. Graph. Forum.

[28]  Wolfgang Effelsberg,et al.  FSCAV: fast seam carving for size adaptation of videos , 2009, ACM Multimedia.

[29]  Daniel Cohen-Or,et al.  Feature-aware texturing , 2006, EGSR '06.

[30]  Xuelong Li,et al.  Semantic Photo Retargeting Under Noisy Image Labels , 2016, TOMM.

[31]  Lihi Zelnik-Manor,et al.  Context-aware saliency detection , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[32]  Olga Sorkine-Hornung,et al.  A comparative study of image retargeting , 2010, ACM Trans. Graph..

[33]  Hari Kalva,et al.  Image retargeting for wearable devices , 2016, 2016 IEEE International Conference on Consumer Electronics (ICCE).

[34]  David A. Clausi,et al.  Structure-Guided Statistical Textural Distinctiveness for Salient Region Detection in Natural Images , 2015, IEEE Trans. Image Process..

[35]  Victor A. Mateescu,et al.  Visual Attention Retargeting , 2022, IEEE MultiMedia.

[36]  Shao-Yi Chien,et al.  Content-aware image resizing using perceptual seam carving with human attention model , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[37]  Takeshi Naemura,et al.  Seam carving for stereo images , 2010, 2010 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[38]  Chao-Hung Lin,et al.  Content-Aware Video Retargeting Using Object-Preserving Warping , 2013 .

[39]  Jian Shi,et al.  Image Retargeting Using Mesh Parametrization , 2009, IEEE Transactions on Multimedia.

[40]  C.-C. Jay Kuo,et al.  Compressed-Domain Video Retargeting , 2014, IEEE Transactions on Image Processing.

[41]  Shang-Hong Lai,et al.  Fast JND-Based Video Carving With GPU Acceleration for Real-Time Video Retargeting , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[42]  Shuyuan Yang,et al.  Salient Region Detection via Discriminative Dictionary Learning and Joint Bayesian Inference , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[43]  Sung-Jea Ko,et al.  Improved seam carving using a modified energy function based on wavelet decomposition , 2009, 2009 IEEE 13th International Symposium on Consumer Electronics.

[44]  Xing Xie,et al.  Automatic browsing of large pictures on mobile devices , 2003, MULTIMEDIA '03.

[45]  Weisi Lin,et al.  Aspect Ratio Similarity (ARS) for image retargeting quality assessment , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[46]  Natasha Gelfand,et al.  A survey of image retargeting techniques , 2010, Optical Engineering + Applications.

[47]  Mei Han,et al.  Discontinuous seam-carving for video retargeting , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[48]  Xin Fan,et al.  Image Retargeting Via Adaptive Scaling With Geometry Preservation , 2014, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[49]  Bohyung Han,et al.  Seam carving with forward gradient difference maps , 2012, ACM Multimedia.

[50]  Bo Yang,et al.  Graph based spatiotemporal saliency detection incorporating low and high level features , 2015, 2015 Visual Communications and Image Processing (VCIP).

[51]  Zhang Yan,et al.  Improved Adaptive Seam Carving for Image Retargeting , 2013, 2013 Fourth International Conference on Digital Manufacturing & Automation.

[52]  Hans-Peter Seidel,et al.  Motion-aware temporal coherence for video resizing , 2009, ACM Trans. Graph..

[53]  Markus H. Gross,et al.  A system for retargeting of streaming video , 2009, ACM Trans. Graph..

[54]  Feiyue Huang,et al.  Measuring and Predicting Visual Importance of Similar Objects , 2016, IEEE Transactions on Visualization and Computer Graphics.

[55]  Meng Wang,et al.  Improved seam carving combining with 3D saliency for image retargeting , 2015, Neurocomputing.

[56]  Ariel Shamir,et al.  Cropping Scaling Seam carving Warping Multi-operator , 2009 .

[57]  Qing Zhang,et al.  Video retargeting combining warping and summarizing optimization , 2013, The Visual Computer.

[58]  Lixia Zhang,et al.  Seam warping: a new approach for image retargeting for small displays , 2017, Soft Comput..

[59]  Benjamin B. Bederson,et al.  Automatic thumbnail cropping and its effectiveness , 2003, UIST '03.

[60]  Wolfgang Effelsberg,et al.  Algorithms for video retargeting , 2010, Multimedia Tools and Applications.

[61]  J.-L. Wu,et al.  Video Adaptation for Small Display Based on Content Recomposition , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[62]  Wolfgang Effelsberg,et al.  Seam carving with improved edge preservation , 2010, Electronic Imaging.

[63]  Hiroshi Ishikawa,et al.  Globally and locally consistent image completion , 2017, ACM Trans. Graph..

[64]  Wen Gao,et al.  Depth-Preserving Warping for Stereo Image Retargeting , 2015, IEEE Transactions on Image Processing.

[65]  Wen Gao,et al.  Spatiotemporal Grid Flow for Video Retargeting , 2014, IEEE Transactions on Image Processing.

[66]  Yael Pritch,et al.  Shift-map image editing , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[67]  Wolfgang Effelsberg,et al.  Adaptation of web pages and images for mobile applications , 2009, Electronic Imaging.

[68]  Tong-Yee Lee,et al.  Scalable and coherent video resizing with per-frame optimization , 2011, SIGGRAPH 2011.

[69]  Yan Liu,et al.  Image Retargeting by Combining Region Warping and Occlusion , 2013, PCM.

[70]  Daniel Cohen-Or,et al.  Combining color and depth for enhanced image segmentation and retargeting , 2011, The Visual Computer.

[71]  Shadrokh Samavi,et al.  Image retargeting using nonparametric semantic segmentation , 2014, Multimedia Tools and Applications.

[72]  Weisi Lin,et al.  Stereoscopic image retargeting based on 3D saliency detection , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[73]  Ja-Ling Wu,et al.  Seam Carving for Color-Plus-Depth 3D Image , 2014, 2014 IEEE International Symposium on Multimedia.

[74]  Wolfgang Effelsberg,et al.  SeamCrop: changing the size and aspect ratio of videos , 2012, MoVid '12.

[75]  Wolfgang Effelsberg,et al.  Improved Image Retargeting by Distinguishing between Faces in Focus and Out of Focus , 2012, 2012 IEEE International Conference on Multimedia and Expo Workshops.

[76]  O. Sorkine,et al.  Motion-based video retargeting with optimized crop-and-warp , 2010, ACM Trans. Graph..

[77]  Bo Yan,et al.  Seam Searching-Based Pixel Fusion for Image Retargeting , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[78]  Wolfgang Effelsberg,et al.  Seam carving for stereoscopic video , 2013, IVMSP 2013.

[79]  Lizhuang Ma,et al.  A survey for image resizing , 2014, Journal of Zhejiang University SCIENCE C.

[80]  Mei Yu,et al.  Stereoscopic Visual Attention Guided Seam Carving for Stereoscopic Image Retargeting , 2016, Journal of Display Technology.

[81]  Wen Gao,et al.  Video retargeting with multi-scale trajectory optimization , 2010, MIR '10.

[82]  Ralph R. Martin,et al.  Shrinkability Maps for Content‐Aware Video Resizing , 2008, Comput. Graph. Forum.

[83]  Kwan-Liu Ma,et al.  Stereoscopic Thumbnail Creation via Efficient Stereo Saliency Detection , 2017, IEEE Transactions on Visualization and Computer Graphics.

[84]  Weiming Dong,et al.  Fast Multi-Operator Image Resizing and Evaluation , 2012, Journal of Computer Science and Technology.

[85]  Jiwon Choi,et al.  Sparse Seam-Carving for Structure Preserving Image Retargeting , 2016, J. Signal Process. Syst..

[86]  Xing Mei,et al.  Image Retargeting by Texture-Aware Synthesis , 2016, IEEE Transactions on Visualization and Computer Graphics.

[87]  Min Xu,et al.  Context-Aware Video Retargeting via Graph Model , 2013, IEEE Transactions on Multimedia.

[88]  Daniel Cohen-Or,et al.  Non-homogeneous Content-driven Video-retargeting , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[89]  Junle Wang,et al.  Perceptual Relevance Based Image Retargeting , 2015, IEEE Signal Processing Letters.

[90]  Michael Gleicher,et al.  Automatic image retargeting with fisheye-view warping , 2005, UIST.

[91]  O. Sorkine,et al.  Optimized scale-and-stretch for image resizing , 2008, SIGGRAPH 2008.

[92]  Ariel Shamir,et al.  Improved seam carving for video retargeting , 2008, SIGGRAPH 2008.

[93]  S. Avidan,et al.  Seam carving for content-aware image resizing , 2007, SIGGRAPH 2007.

[94]  King Ngi Ngan,et al.  Image Retargeting Quality Assessment: A Study of Subjective Scores and Objective Metrics , 2012, IEEE Journal of Selected Topics in Signal Processing.

[95]  Wen-Jiin Tsai,et al.  Hybrid image retargeting , 2015, 2015 Visual Communications and Image Processing (VCIP).

[96]  Shang-Hong Lai,et al.  Compressibility-Aware Media Retargeting With Structure Preserving , 2011, IEEE Transactions on Image Processing.

[97]  Ling-Yu Duan,et al.  Consumer video retargeting: context assisted spatial-temporal grid optimization , 2009, ACM Multimedia.

[98]  C.-C. Jay Kuo,et al.  Region-adaptive texture-aware image resizing , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[99]  Yan Liu,et al.  How important is location information in saliency detection of natural images , 2015, Multimedia Tools and Applications.

[100]  Tao Mei,et al.  Adaptive Content Condensation Based on Grid Optimization for Thumbnail Image Generation , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[101]  Edoardo Ardizzone,et al.  Physical Metaphor for Streaming Media Retargeting , 2014, IEEE Transactions on Multimedia.

[102]  Bo Yan,et al.  Effective Video Retargeting With Jittery Assessment , 2014, IEEE Transactions on Multimedia.

[103]  Min Xu,et al.  A Hybrid Image Retargeting Approach via Combining Seam Carving and Grid Warping , 2014, J. Multim..

[104]  Qi Tian,et al.  Retargeting Semantically-Rich Photos , 2015, IEEE Transactions on Multimedia.

[105]  Wolfgang Effelsberg,et al.  Automatic scaling and cropping of videos for devices with limited screen resolution , 2006, MM '06.

[106]  Shai Avidan,et al.  Stereo Seam Carving a Geometrically Consistent Approach , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[107]  Xiangchu Feng,et al.  Salient edge and region aware image retargeting , 2014, Signal Process. Image Commun..

[108]  Feng Wu,et al.  Depth-Preserving Stereo Image Retargeting Based on Pixel Fusion , 2017, IEEE Transactions on Multimedia.

[109]  Toshihiko Yamasaki,et al.  Fast volume seam carving with multi-pass dynamic programming , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[110]  Rongrong Ji,et al.  Towards perceptual video cropping with curve fitting , 2014, Multimedia Tools and Applications.

[111]  Toshihiko Yamasaki,et al.  Fast Volume Seam Carving With Multipass Dynamic Programming , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[112]  Zhang Yan,et al.  A Study of Image Retargeting Based on Seam Carving , 2014, 2014 Sixth International Conference on Measuring Technology and Mechatronics Automation.

[113]  Yan Liu,et al.  Image retargeting based on global energy optimization , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[114]  Yong-Jin Liu,et al.  Optimal-Scaling-Factor Assignment for Patch-wise Image Retargeting , 2013, IEEE Computer Graphics and Applications.

[115]  Li Xu,et al.  Hierarchical Image Saliency Detection on Extended CSSD , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[116]  Wolfgang Effelsberg,et al.  SeamCrop for image retargeting , 2012, Electronic Imaging.