Learning quality assessment of retargeted images

Abstract Content-aware image resizing (or image retargeting) enables images to be fit to different display devices having different aspect ratios while preserving salient image content. There are many approaches to retargeting, although no “best” method has been agreed upon. Therefore, finding ways to assess the quality of image retargeting has become a prominent challenge. Traditional image quality assessment methods are not directly applicable to image retargeting because the retargeted image size is not same as the original one. In this paper, we propose an open framework for image retargeting quality assessment, where the quality prediction engine is a trained Radial Basis Function (RBF) neural network. Broadly, our approach is motivated by the observation that no single method can be expected to perform well on all types of content. We train the network on ten perceptually relevant features, including a saliency-weighted, SIFT-directed complex wavelet structural similarity (CW-SSIM) index, and a new image aesthetics evaluation method. These two features and eight other features are used by the neural network to learn to assess the quality of retargeted images. The accuracy of the new model is extensively verified by simulations.

[1]  W. Chu.  Studying Aesthetics in Photographic Images Using a Computational Approach , 2013 .

[2]  C.-C. Jay Kuo,et al.  An Objective Quality of Experience (QoE) Assessment Index for Retargeted Images , 2014, ACM Multimedia.

[3]  Olga Sorkine-Hornung,et al.  A comparative study of image retargeting , 2010, ACM Trans. Graph..

[4]  Xin Zhang,et al.  A face detection algorithm based on modified skin-color model , 2013, Proceedings of the 32nd Chinese Control Conference.

[5]  Weisi Lin,et al.  Image Quality Assessment Using Multi-Method Fusion , 2013, IEEE Transactions on Image Processing.

[6]  B. S. Manjunath,et al.  Color and texture descriptors , 2001, IEEE Trans. Circuits Syst. Video Technol..

[7]  Michael Werman,et al.  Fast and robust Earth Mover's Distances , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[8]  Craig Gotsman,et al.  Energy‐Based Image Deformation , 2009, Comput. Graph. Forum.

[9]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[10]  Alan C. Bovik,et al.  Experiments in segmenting texton patterns using localized spatial filters , 1989, Pattern Recognit..

[11]  Yael Pritch,et al.  Shift-map image editing , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[12]  C. Spearman.  The proof and measurement of association between two things. , 2015, International journal of epidemiology.

[13]  Akio Yamada,et al.  The MPEG-7 color layout descriptor: a compact image feature description for high-speed image/video segment retrieval , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[14]  Naila Murray,et al.  AVA: A large-scale database for aesthetic visual analysis , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  O. Sorkine,et al.  Optimized scale-and-stretch for image resizing , 2008, SIGGRAPH 2008.

[16]  Markus Gross,et al.  A system for retargeting of streaming video , 2009, SIGGRAPH 2009.

[17]  Diego Gutierrez,et al.  Objective Quality Prediction of Image Retargeting Algorithms , 2017, IEEE Transactions on Visualization and Computer Graphics.

[18]  Yan Ke,et al.  The Design of High-Level Features for Photo Quality Assessment , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[19]  Alan C. Bovik,et al.  Objective quality assessment of multiply distorted images , 2012, 2012 Conference Record of the Forty Sixth Asilomar Conference on Signals, Systems and Computers (ASILOMAR).

[20]  Friedhelm Schwenker,et al.  Three learning phases for radial-basis-function networks , 2001, Neural Networks.

[21]  Ariel Shamir,et al.  Cropping Scaling Seam carving Warping Multi-operator , 2009 .

[22]  Weisi Lin,et al.  Visual Quality Assessment by Machine Learning , 2015 .

[23]  Weisi Lin,et al.  Backward Registration-Based Aspect Ratio Similarity for Image Retargeting Quality Assessment , 2016, IEEE Transactions on Image Processing.

[24]  Stephanie Thalberg.  Contemporary Quilts Design Surface And Stitch , 2016 .

[25]  Ariel Shamir,et al.  Improved seam carving for video retargeting , 2008, SIGGRAPH 2008.

[26]  Weisi Lin,et al.  Objective Quality Assessment for Image Retargeting Based on Perceptual Geometric Distortion and Information Loss , 2014, IEEE Journal of Selected Topics in Signal Processing.

[27]  Sandra Meech.  Contemporary Quilts: Design, Surface and Stitch , 2004 .

[28]  D. Broomhead,et al.  Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks , 1988 .

[29]  Jingrui He,et al.  Classification of Digital Photos Taken by Photographers or Home Users , 2004, PCM.

[30]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[31]  Eli Shechtman,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, ACM Trans. Graph..

[32]  Alan C. Bovik,et al.  Automatic Prediction of Perceptual Image and Video Quality , 2013, Proceedings of the IEEE.

[33]  Weiming Dong,et al.  Optimized image resizing using seam carving and scaling , 2009, SIGGRAPH 2009.

[34]  Andrew D. Back,et al.  Radial Basis Functions , 2001 .

[35]  Anil K. Jain,et al.  Artificial neural networks and statistical pattern recognition : old and new connections , 1991 .

[36]  Zhou Wang,et al.  Video quality assessment based on structural distortion measurement , 2004, Signal Process. Image Commun..

[37]  Denis Simakov,et al.  Summarizing visual data using bidirectional similarity , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  Alexander C. Loui,et al.  Measuring the perceived aesthetic quality of photographic images , 2009, 2009 International Workshop on Quality of Multimedia Experience.

[39]  Daniel Cohen-Or,et al.  Non-homogeneous Content-driven Video-retargeting , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[40]  Yong-Jin Liu,et al.  Image Retargeting Quality Assessment , 2011, Comput. Graph. Forum.

[41]  Kai Zeng,et al.  Objective Quality Assessment for Image Retargeting Based on Structural Similarity , 2014, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[42]  Alexander C. Loui,et al.  Automatic aesthetic value assessment in photographic images , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[43]  Antonio Torralba,et al.  SIFT Flow: Dense Correspondence across Different Scenes , 2008, ECCV.

[44]  Alan C. Bovik,et al.  Visual quality assessment algorithms: what does the future hold? , 2010, Multimedia Tools and Applications.

[45]  Ariel Shamir,et al.  Seam Carving for Content-Aware Image Resizing , 2007, ACM Trans. Graph..

[46]  Ling Shao,et al.  Perceptually Guided Photo Retargeting , 2017, IEEE Transactions on Cybernetics.

[47]  Alan C. Bovik,et al.  A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms , 2006, IEEE Transactions on Image Processing.

[48]  King Ngi Ngan,et al.  Image Retargeting Quality Assessment: A Study of Subjective Scores and Objective Metrics , 2012, IEEE Journal of Selected Topics in Signal Processing.

[49]  M. Kendall.  A NEW MEASURE OF RANK CORRELATION , 1938 .

[50]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[51]  Zhou Wang,et al.  Complex Wavelet Structural Similarity: A New Image Similarity Index , 2009, IEEE Transactions on Image Processing.