Semantic Segmentation Guided Pixel Fusion for Image Retargeting

Image retargeting aims to obtain high visual quality of target images for human vision. Through semantic segmentation and understanding of input images, we can better preserve the important semantic regions, so as to effectively improve the performance of image retargeting. Benefit from the successful application of deep neural network in the field of semantic segmentation, in this paper, we propose a novel image retargeting approach using semantic segmentation and pixel fusion. Compared with existing image retargeting methods, our approach can effectively reduce geometric distortion during image retargeting by finely reallocating scaling factors for each region based on the semantic segmentation results. Experimental results demonstrate that the proposed approach can well preserve important semantic regions while leaving less unnatural geometric distortion. Our approach also shows the important role of semantic segmentation and understanding of scenes in image retargeting in detail.

[1]  Olga Sorkine-Hornung,et al.  A comparative study of image retargeting , 2010, ACM Trans. Graph..

[2]  Bo Yan,et al.  Seam Searching-Based Pixel Fusion for Image Retargeting , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[3]  Jian Shi,et al.  Image Retargeting Using Mesh Parametrization , 2009, IEEE Transactions on Multimedia.

[4]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Qi Tian,et al.  Image Retargeting for Preserving Robust Local Feature: Application to Mobile Visual Search , 2016, IEEE Transactions on Multimedia.

[6]  Markus H. Gross,et al.  A system for retargeting of streaming video , 2009, ACM Trans. Graph..

[7]  Xuelong Li,et al.  Depth-Aware Image Seam Carving , 2013, IEEE Transactions on Cybernetics.

[8]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Xing Mei,et al.  Image Retargeting by Texture-Aware Synthesis , 2016, IEEE Transactions on Visualization and Computer Graphics.

[10]  Min Xu,et al.  Context-Aware Video Retargeting via Graph Model , 2013, IEEE Transactions on Multimedia.

[11]  Yi Yang,et al.  Attention to Scale: Scale-Aware Semantic Image Segmentation , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Chia-Wen Lin,et al.  Maintaining Temporal Coherence in Video Retargeting Using Mosaic-Guided Scaling , 2011, IEEE Transactions on Image Processing.

[13]  Daniel Cohen-Or,et al.  Non-homogeneous Content-driven Video-retargeting , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[14]  Ian D. Reid,et al.  RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[16]  Feng Wu,et al.  Depth-Preserving Stereo Image Retargeting Based on Pixel Fusion , 2017, IEEE Transactions on Multimedia.

[17]  Lok Ming Lui,et al.  Image Retargeting via Beltrami Representation , 2018, IEEE Transactions on Image Processing.

[18]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[19]  S. Avidan,et al.  Seam carving for content-aware image resizing , 2007, SIGGRAPH 2007.

[20]  Olga Sorkine-Hornung,et al.  Robust Image Retargeting via Axis‐Aligned Deformation , 2012, Comput. Graph. Forum.

[21]  Yong-Jin Liu,et al.  Image Retargeting Quality Assessment , 2011, Comput. Graph. Forum.

[22]  Michael Gleicher,et al.  Video retargeting: automating pan and scan , 2006, MM '06.

[23]  Luc Van Gool,et al.  The Pascal Visual Object Classes Challenge: A Retrospective , 2014, International Journal of Computer Vision.

[24]  Bo Yan,et al.  Lowcomplexity content-aware image retargeting , 2012, 2012 19th IEEE International Conference on Image Processing.

[25]  Chao-Hung Lin,et al.  Patch-Based Image Warping for Content-Aware Retargeting , 2013, IEEE Transactions on Multimedia.

[26]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Xiaogang Wang,et al.  Context Encoding for Semantic Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  Qi Tian,et al.  Retargeting Semantically-Rich Photos , 2015, IEEE Transactions on Multimedia.

[29]  Olga Sorkine-Hornung,et al.  Visual media retargeting , 2009, SIGGRAPH ASIA Courses.

[30]  Bo Yan,et al.  Effective Video Retargeting With Jittery Assessment , 2014, IEEE Transactions on Multimedia.

[31]  Ariel Shamir,et al.  Cropping Scaling Seam carving Warping Multi-operator , 2009 .

[32]  Iasonas Kokkinos,et al.  Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs , 2014, ICLR.

[33]  Truong Q. Nguyen,et al.  Depth-Adaptive Deep Neural Network for Semantic Segmentation , 2017, IEEE Transactions on Multimedia.

[34]  Gang Yu,et al.  Learning a Discriminative Feature Network for Semantic Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.