A Deep Evaluator for Image Retargeting Quality by Geometrical and Contextual Interaction

An image is compressed or stretched during the multidevice displaying, which will have a very big impact on perception quality. In order to solve this problem, a variety of image retargeting methods have been proposed for the retargeting process. However, how to evaluate the results of different image retargeting is a very critical issue. In various application systems, the subjective evaluation method cannot be applied on a large scale. So we put this problem in the accurate objective-quality evaluation. Currently, most of the image retargeting quality assessment algorithms use simple regression methods as the last step to obtain the evaluation result, which are not corresponding with the perception simulation in the human vision system (HVS). In this paper, a deep quality evaluator for image retargeting based on the segmented stacked AutoEnCoder (SAE) is proposed. Through the help of regularization, the designed deep learning framework can solve the overfitting problem. The main contributions in this framework are to simulate the perception of retargeted images in HVS. Especially, it trains two separated SAE models based on geometrical shape and content matching. Then, the weighting schemes can be used to combine the obtained scores from two models. Experimental results in three well-known databases show that our method can achieve better performance than traditional methods in evaluating different image retargeting results.

[1]  ShamirAriel,et al.  Seam carving for content-aware image resizing , 2007 .

[2]  Xuelong Li,et al.  Depth-Aware Image Seam Carving , 2013, IEEE Transactions on Cybernetics.

[3]  King Ngi Ngan,et al.  Image Retargeting Quality Assessment: A Study of Subjective Scores and Objective Metrics , 2012, IEEE Journal of Selected Topics in Signal Processing.

[4]  Svetlana Lazebnik,et al.  Iterative quantization: A procrustean approach to learning binary codes , 2011, CVPR 2011.

[5]  Alan C. Bovik,et al.  A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms , 2006, IEEE Transactions on Image Processing.

[6]  Markus H. Gross,et al.  A system for retargeting of streaming video , 2009, ACM Trans. Graph..

[7]  Andreas Uhl,et al.  Modeling the Marginal Distributions of Complex Wavelet Coefficient Magnitudes for the Classification of Zoom-Endoscopy Images , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[8]  Yong-Jin Liu,et al.  Image Retargeting Quality Assessment , 2011, Comput. Graph. Forum.

[9]  Denis Simakov,et al.  Summarizing visual data using bidirectional similarity , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Antonio Torralba,et al.  Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope , 2001, International Journal of Computer Vision.

[11]  Antonio Torralba,et al.  SIFT Flow: Dense Correspondence across Scenes and Its Applications , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  King Ngi Ngan,et al.  No-Reference Retargeted Image Quality Assessment Based on Pairwise Rank Learning , 2016, IEEE Transactions on Multimedia.

[13]  Svetlana Lazebnik,et al.  Iterative quantization: A procrustean approach to learning binary codes , 2011, CVPR 2011.

[14]  Jasper Snoek,et al.  Nonparametric guidance of autoencoder representations using label information , 2012, J. Mach. Learn. Res..

[15]  Jessica A. Turner,et al.  Deep Independence Network Analysis of Structural Brain Imaging: Application to Schizophrenia , 2016, IEEE Transactions on Medical Imaging.

[16]  S. Avidan,et al.  Seam carving for content-aware image resizing , 2007, SIGGRAPH 2007.

[17]  Olga Sorkine-Hornung,et al.  A comparative study of image retargeting , 2010, ACM Trans. Graph..

[18]  Daniel Cohen-Or,et al.  Non-homogeneous Content-driven Video-retargeting , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[19]  Jianping Fan,et al.  HD-MTL: Hierarchical Deep Multi-Task Learning for Large-Scale Visual Recognition , 2017, IEEE Transactions on Image Processing.

[20]  Christophe Charrier,et al.  Blind Image Quality Assessment: A Natural Scene Statistics Approach in the DCT Domain , 2012, IEEE Transactions on Image Processing.

[21]  Craig Gotsman,et al.  Energy‐Based Image Deformation , 2009, Comput. Graph. Forum.

[22]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[23]  Kai Zeng,et al.  Objective Quality Assessment for Image Retargeting Based on Structural Similarity , 2014, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[24]  Weisi Lin,et al.  SVD-Based Quality Metric for Image and Video Using Machine Learning , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[25]  Franco Scarselli,et al.  On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Ja-Ling Wu,et al.  Quality Assessment of Stereoscopic 3D Image Compression by Binocular Integration Behaviors , 2014, IEEE Transactions on Image Processing.

[27]  King Ngi Ngan,et al.  Reduced-Reference Video Quality Assessment of Compressed Video Sequences , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[28]  Wenjing Jin,et al.  Enhanced Restricted Boltzmann Machine With Prognosability Regularization for Prognostics and Health Assessment , 2016, IEEE Transactions on Industrial Electronics.

[29]  Jiying Zhao,et al.  An Image Quality Evaluation Method Based on Digital Watermarking , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[30]  R. A. Bradley,et al.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS , 1952 .

[31]  Gangyi Jiang,et al.  Learning Sparse Representation for Objective Image Retargeting Quality Assessment , 2018, IEEE Transactions on Cybernetics.

[32]  O. Sorkine,et al.  Optimized scale-and-stretch for image resizing , 2008, SIGGRAPH 2008.

[33]  B. S. Manjunath,et al.  Color and texture descriptors , 2001, IEEE Trans. Circuits Syst. Video Technol..

[34]  Weisi Lin,et al.  Objective Quality Assessment for Image Retargeting Based on Perceptual Geometric Distortion and Information Loss , 2014, IEEE Journal of Selected Topics in Signal Processing.

[35]  Yabin Zhang,et al.  Backward Registration-Based Aspect Ratio Similarity for Image Retargeting Quality Assessment. , 2016, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[36]  Xuelong Li,et al.  Sparse representation for blind image quality assessment , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Feiniu Yuan,et al.  Optimized Multioperator Image Retargeting Based on Perceptual Similarity Measure , 2017, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[38]  R. A. Bradley,et al.  Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons , 1952 .

[39]  Alan C. Bovik,et al.  Making image quality assessment robust , 2012, 2012 Conference Record of the Forty Sixth Asilomar Conference on Signals, Systems and Computers (ASILOMAR).

[40]  Matti Pietikäinen,et al.  Gray Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2000, ECCV.

[41]  Sungho Jo,et al.  Approximate Bayesian MLP regularization for regression in the presence of noise , 2016, Neural Networks.

[42]  Lei Zhang,et al.  Blind Image Quality Assessment Using Joint Statistics of Gradient Magnitude and Laplacian Features , 2014, IEEE Transactions on Image Processing.

[43]  Yael Pritch,et al.  Shift-map image editing , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[44]  Michael Werman,et al.  Fast and robust Earth Mover's Distances , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[45]  Ling Shao,et al.  Perceptually Guided Photo Retargeting , 2017, IEEE Transactions on Cybernetics.

[46]  Weisi Lin,et al.  Image retargeting quality assessment based on support vector regression , 2015, Signal Process. Image Commun..

[47]  Gary J. Sullivan,et al.  Video Quality Evaluation Methodology and Verification Testing of HEVC Compression Performance , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[48]  Björn W. Schuller,et al.  Autoencoder-based Unsupervised Domain Adaptation for Speech Emotion Recognition , 2014, IEEE Signal Processing Letters.

[49]  Zhou Wang,et al.  Video quality assessment based on structural distortion measurement , 2004, Signal Process. Image Commun..

[50]  Thomas Hofmann,et al.  Greedy Layer-Wise Training of Deep Networks , 2007 .

[51]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[52]  Yoshua Bengio,et al.  An empirical evaluation of deep architectures on problems with many factors of variation , 2007, ICML '07.

[53]  Diego Gutierrez,et al.  Objective Quality Prediction of Image Retargeting Algorithms , 2017, IEEE Transactions on Visualization and Computer Graphics.

[54]  Ariel Shamir,et al.  Cropping Scaling Seam carving Warping Multi-operator , 2009 .

[55]  Lawrence H. Staib,et al.  Shape-based 3D surface correspondence using geodesics and local geometry , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[56]  Alan C. Bovik,et al.  Image information and visual quality , 2006, IEEE Trans. Image Process..

[57]  Weiming Dong,et al.  Optimized image resizing using seam carving and scaling , 2009, SIGGRAPH 2009.