Visual saliency in image quality assessment

Advances in image quality assessment have shown the benefits of modelling functional components of the human visual system in image quality metrics. Visual saliency, a crucial aspect of the human visual system, is increasingly investigated recently. Current applications of visual saliency in image quality metrics are limited by our knowledge on the relation between visual saliency and quality perception. Issues regarding how to simulate and integrate visual saliency in image quality metrics remain. This thesis presents psychophysical experiments and computational models relevant to the perceptually-optimised use of visual saliency in image quality metrics. We first systematically validated the capability of computational saliency in improving image quality metrics. Practical guidance regarding how to select suitable saliency models, which image quality metrics can benefit from saliency integration, and how the added value of saliency depends on image distortion type were provided. To better understand the relation between saliency and image quality, an eye-tracking experiment with a reliable experimental methodology was first designed to obtain ground truth fixation data. Significant findings on the interactions between saliency and visual distortion were then discussed. Based on these findings, a saliency integration approach taking into account the impact of distortion on the saliency deployment was proposed. We also devised an algorithm which adaptively incorporate saliency in image quality metrics based on saliency dispersion. Moreover, we further investigated the plausibility of measuring image quality based on the deviation of saliency induced by distortion. An image quality metric based on measuring saliency deviation was devised. This thesis demonstrates that the added value of saliency in image quality metrics can be optimised by taking into account the interactions between saliency and visual distortion. This thesis also demonstrates that the deviation of fixation deployment due to distortion can be used as a proxy for the prediction of image quality.

[1]  Alan C. Bovik,et al.  Visual Importance Pooling for Image Quality Assessment , 2009, IEEE Journal of Selected Topics in Signal Processing.

[2]  D. Chandler.  Seven Challenges in Image Quality Assessment: Past, Present, and Future Research , 2013 .

[3]  Patrick Le Callet,et al.  Visual Attention and Applications in Multimedia Technologies , 2013, Proceedings of the IEEE.

[4]  Weisi Lin,et al.  Image Quality Assessment Based on Gradient Similarity , 2012, IEEE Transactions on Image Processing.

[5]  Judith Redi,et al.  Effects of task and image properties on visual-attention deployment in image-quality assessment , 2015, J. Electronic Imaging.

[6]  Pietro Perona,et al.  Graph-Based Visual Saliency , 2006, NIPS.

[7]  Matei Mancas,et al.  Memorability of natural scenes: The role of attention , 2013, 2013 IEEE International Conference on Image Processing.

[8]  Liming Zhang,et al.  A Novel Multiresolution Spatiotemporal Saliency Detection Model and Its Applications in Image and Video Compression , 2010, IEEE Transactions on Image Processing.

[9]  Stefan Winkler,et al.  Analysis of Public Image and Video Databases for Quality Assessment , 2012, IEEE Journal of Selected Topics in Signal Processing.

[10]  Eric C. Larson,et al.  Can visual fixation patterns improve image fidelity assessment? , 2008, 2008 15th IEEE International Conference on Image Processing.

[11]  J. M. Foley,et al.  Contrast masking in human vision. , 1980, Journal of the Optical Society of America.

[12]  Zhou Wang,et al.  Group MAD Competition? A New Methodology to Compare Objective Image Quality Models , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Andrew B. Watson,et al.  Digital images and human vision , 1993 .

[14]  Martin Reisslein,et al.  Objective Video Quality Assessment Methods: A Classification, Review, and Performance Comparison , 2011, IEEE Transactions on Broadcasting.

[15]  Eric C. Larson,et al.  Unveiling relationships between regions of interest and image fidelity metrics , 2008, Electronic Imaging.

[16]  S. Süsstrunk,et al.  Frequency-tuned salient region detection , 2009, CVPR 2009.

[17]  Nanning Zheng,et al.  Automatic salient object segmentation based on context and shape prior , 2011, BMVC.

[18]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[19]  Alan C. Bovik,et al.  GAFFE: A Gaze-Attentive Fixation Finding Engine , 2008, IEEE Transactions on Image Processing.

[20]  B. Breitmeyer,et al.  Mechanisms of visual attention revealed by saccadic eye movements , 1987, Neuropsychologia.

[21]  Louis K. H. Chan,et al.  Dimension-specific signal modulation in visual search: evidence from inter-stimulus surround suppression. , 2012, Journal of vision.

[22]  Tim K Marks,et al.  SUN: A Bayesian framework for saliency using natural statistics. , 2008, Journal of vision.

[23]  Antonio Torralba,et al.  Modeling global scene factors in attention. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[24]  Zhou Wang,et al.  Information Content Weighting for Perceptual Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[25]  Dominic W. Hughes,et al.  Eye-Tracking Patterns in Schizophrenia , 1973, Science.

[26]  A. Torralba,et al.  Fixations on low-resolution images. , 2010, Journal of vision.

[27]  Eric C. Larson,et al.  Most apparent distortion: full-reference image quality assessment and the role of strategy , 2010, J. Electronic Imaging.

[28]  John K. Tsotsos,et al.  Saliency, attention, and visual search: an information theoretic approach. , 2009, Journal of vision.

[29]  J. Hoffman,et al.  The role of visual attention in saccadic eye movements , 1995, Perception & psychophysics.

[30]  Ingrid Heynderickx,et al.  Comparative Study of Fixation Density Maps , 2013, IEEE Transactions on Image Processing.

[31]  Glen P. Abousleman,et al.  A no-reference perceptual image sharpness metric based on saliency-weighted foveal pooling , 2008, 2008 15th IEEE International Conference on Image Processing.

[32]  Lihi Zelnik-Manor,et al.  Context-Aware Saliency Detection , 2012, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  Eric N. Wiebe,et al.  Eye-Tracking Students' Attention to PowerPoint Photographs in a Science Education Setting , 2005 .

[34]  Xiongkuo Min,et al.  Influence of compression artifacts on visual attention , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).

[35]  Lina J. Karam,et al.  A No-Reference Objective Image Sharpness Metric Based on the Notion of Just Noticeable Blur (JNB) , 2009, IEEE Transactions on Image Processing.

[36]  H R BLACKWELL,et al.  Contrast thresholds of the human eye. , 1946, Journal of the Optical Society of America.

[37]  Aline Roumy,et al.  Prediction of the inter-observer visual congruency (IOVC) and application to image ranking , 2011, ACM Multimedia.

[38]  Alan C. Bovik,et al.  Image information and visual quality , 2006, IEEE Trans. Image Process..

[39]  Wilson S. Geisler,et al.  Image quality assessment based on a degradation model , 2000, IEEE Trans. Image Process..

[40]  Margaret H. Pinson,et al.  A new standardized method for objectively measuring video quality , 2004, IEEE Transactions on Broadcasting.

[41]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[42]  Robert V. Brill,et al.  Applied Statistics and Probability for Engineers , 2004, Technometrics.

[43]  Asha Iyer,et al.  Components of bottom-up gaze allocation in natural images , 2005, Vision Research.

[44]  A. Oliva,et al.  Coarse Blobs or Fine Edges? Evidence That Information Diagnosticity Changes the Perception of Complex Visual Stimuli , 1997, Cognitive Psychology.

[45]  Weisi Lin,et al.  Saliency Detection in the Compressed Domain for Adaptive Image Retargeting , 2012, IEEE Transactions on Image Processing.

[46]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[47]  Patrick Le Callet,et al.  Considering Temporal Variations of Spatial Visual Distortions in Video Quality Assessment , 2009, IEEE Journal of Selected Topics in Signal Processing.

[48]  Ivan V. Bajic,et al.  Eye-Tracking Database for a Set of Standard Video Sequences , 2012, IEEE Transactions on Image Processing.

[49]  Patrick Le Callet,et al.  Subjective quality assessment IRCCyN/IVC database , 2004 .

[50]  Peyman Milanfar,et al.  Static and space-time visual saliency detection by self-resemblance. , 2009, Journal of vision.

[51]  Antonio Torralba,et al.  Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. , 2006, Psychological review.

[52]  Paul L. Rosin.  A simple method for detecting salient regions , 2009, Pattern Recognit..

[53]  Sheila S. Hemami,et al.  VSNR: A Wavelet-Based Visual Signal-to-Noise Ratio for Natural Images , 2007, IEEE Transactions on Image Processing.

[54]  Ihor O. Kirenko,et al.  A no-reference blocking artifact measure for adaptive video processing , 2005, 2005 13th European Signal Processing Conference.

[55]  Tao Liu,et al.  Saliency Inspired Full-Reference Quality Metrics for Packet-Loss-Impaired Video , 2011, IEEE Transactions on Broadcasting.

[56]  Stefan Winkler,et al.  Review of Existing Objective QoE Methodologies , 2015 .

[57]  Tao Liu,et al.  Saliency based objective quality assessment of decoded video affected by packet losses , 2008, 2008 15th IEEE International Conference on Image Processing.

[58]  Weisi Lin,et al.  Saliency-based stereoscopic image retargeting , 2016, Inf. Sci..

[59]  A. Greenwald.  Within-subjects designs: To use or not to use? , 1976 .

[60]  Neil D. B. Bruce,et al.  Factors underlying inter-observer agreement in gaze patterns: predictive modelling and analysis , 2016, ETRA.

[61]  Stefan Winkler,et al.  A no-reference perceptual blur metric , 2002, Proceedings. International Conference on Image Processing.

[62]  Ali Borji,et al.  Salient Object Detection: A Benchmark , 2015, IEEE Transactions on Image Processing.

[63]  Nikolay N. Ponomarenko,et al.  Color image database TID2013: Peculiarities and preliminary results , 2013, European Workshop on Visual Information Processing (EUVIP).

[64]  D. Jameson,et al.  An opponent-process theory of color vision. , 1957, Psychological review.

[65]  Iain D. Gilchrist,et al.  Visual correlates of fixation selection: effects of scale and time , 2005, Vision Research.

[66]  André Kaup,et al.  Temporal Trajectory Aware Video Quality Measure , 2009, IEEE Journal of Selected Topics in Signal Processing.

[67]  Alan C. Bovik,et al.  Motion Tuned Spatio-Temporal Quality Assessment of Natural Videos , 2010, IEEE Transactions on Image Processing.

[68]  Mert R. Sabuncu,et al.  Entropy-based Image Registration , 2006 .

[69]  Stefan Winkler,et al.  Vision models and quality metrics for image processing applications , 2001 .

[70]  David S Wooding,et al.  Eye movements of large populations: II. Deriving regions of interest, coverage, and similarity using fixation maps , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[71]  S Ullman,et al.  Shifts in selective visual attention: towards the underlying neural circuitry. , 1985, Human neurobiology.

[72]  K. Guo,et al.  How does image noise affect actual and predicted human gaze allocation in assessing image quality? , 2015, Vision Research.

[73]  Nicolas Riche,et al.  RARE2012: A multi-scale rarity-based saliency detection with its comparative statistical analysis , 2013, Signal Process. Image Commun..

[74]  Weisi Lin,et al.  Modeling visual attention's modulatory aftereffects on visual sensitivity and quality evaluation , 2005, IEEE Transactions on Image Processing.

[75]  B. Wandell.  Foundations of vision , 1995 .

[76]  Zhou Wang,et al.  Spatial Pooling Strategies for Perceptual Image Quality Assessment , 2006, 2006 International Conference on Image Processing.

[77]  Ingrid Heynderickx,et al.  Visual Attention in Objective Image Quality Assessment: Based on Eye-Tracking Data , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[78]  Liqing Zhang,et al.  Dynamic visual attention: searching for coding length increments , 2008, NIPS.

[79]  Sugato Chakravarty,et al.  Methodology for the subjective assessment of the quality of television pictures , 1995 .

[80]  Brian Scassellati,et al.  A Context-Dependent Attention System for a Social Robot , 1999, IJCAI.

[81]  Andrew T Duchowski,et al.  A breadth-first survey of eye-tracking applications , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[82]  Abdelhakim Saadane,et al.  Blind Quality Metric using a Perceptual Importance Map for JPEG-20000 Compressed Images , 2006, 2006 International Conference on Image Processing.

[83]  Nikolay N. Ponomarenko,et al.  Image database TID2013: Peculiarities, results and perspectives , 2015, Signal Process. Image Commun..

[84]  Judith Redi,et al.  Interactions of visual attention and quality perception , 2011, Electronic Imaging.

[85]  Scott J. Daly,et al.  Application of a Noise Adaptive Contrast Sensitivity Function to Image Data Compression , 1989, Photonics West - Lasers and Applications in Science and Engineering.

[86]  Hans-Jürgen Zepernick,et al.  Framework for optimal region of interest-based quality assessment in wireless imaging , 2010, J. Electronic Imaging.

[87]  Stefan Winkler,et al.  Overview of Eye tracking Datasets , 2013, 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX).

[88]  Liming Zhang,et al.  Image quality assessment with visual attention , 2008, 2008 19th International Conference on Pattern Recognition.

[89]  Aykut Erdem,et al.  Visual saliency estimation by nonlinearly integrating features using region covariances. , 2013, Journal of vision.

[90]  Alan C. Bovik,et al.  Massive Online Crowdsourced Study of Subjective and Objective Picture Quality , 2015, IEEE Transactions on Image Processing.

[91]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[92]  Ali Borji,et al.  State-of-the-Art in Visual Attention Modeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[93]  Ali Borji,et al.  Exploiting local and global patch rarities for saliency detection , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[94]  K. Rayner.  The 35th Sir Frederick Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search , 2009, Quarterly journal of experimental psychology.

[95]  Touradj Ebrahimi,et al.  The JPEG2000 still image coding system: an overview , 2000, IEEE Trans. Consumer Electron..

[96]  Christof Koch,et al.  Modeling attention to salient proto-objects , 2006, Neural Networks.

[97]  A. M. Rohaly,et al.  Automatic detection of regions of interest in complex video sequences , 2001, IS&T/SPIE Electronic Imaging.

[98]  Mylène C. Q. Farias,et al.  Video quality assessment using visual attention computational models , 2014, J. Electronic Imaging.

[99]  H.R. Wu,et al.  A generalized block-edge impairment metric for video coding , 1997, IEEE Signal Processing Letters.

[100]  Hongyu Li,et al.  VSI: A Visual Saliency-Induced Index for Perceptual Image Quality Assessment , 2014, IEEE Transactions on Image Processing.

[101]  Peter M. Corcoran,et al.  Real-time eye gaze tracking for gaming design and consumer electronics systems , 2012, IEEE Transactions on Consumer Electronics.

[102]  E.C.L. Vu,et al.  Visual Fixation Patterns when Judging Image Quality: Effects of Distortion Type, Amount, and Subject Experience , 2008, 2008 IEEE Southwest Symposium on Image Analysis and Interpretation.

[103]  Liming Zhang,et al.  Spatio-temporal Saliency detection using phase spectrum of quaternion fourier transform , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[104]  Alan C. Bovik,et al.  A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms , 2006, IEEE Transactions on Image Processing.

[105]  Lambert Schomaker,et al.  Prediction of human eye fixations using symmetry , 2009 .

[106]  Patrick Le Callet,et al.  Does where you Gaze on an Image Affect your Perception of Quality? Applying Visual Attention to Image Quality Metric , 2007, 2007 IEEE International Conference on Image Processing.

[107]  Frédo Durand,et al.  Learning to predict where humans look , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[108]  Laurent Itti,et al.  An Integrated Model of Top-Down and Bottom-Up Attention for Optimizing Detection Speed , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[109]  Lihi Zelnik-Manor,et al.  Salient Edges: A Multi Scale Approach , 2010 .

[110]  Zhou Wang,et al.  Modern Image Quality Assessment , 2006, Modern Image Quality Assessment.

[111]  Alan C. Bovik,et al.  Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures , 2009, IEEE Signal Processing Magazine.

[112]  L. Ma,et al.  Visual saliency detection in image using ant colony optimisation and local phase coherence , 2010 .

[113]  Nanning Zheng,et al.  Learning to Detect a Salient Object , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[114]  Stefan Winkler,et al.  Video quality measurement standards — Current status and trends , 2009, 2009 7th International Conference on Information, Communications and Signal Processing (ICICS).

[115]  Judith Redi,et al.  Quantifying the importance of preserving video quality in visually important regions at the expense of background content , 2015, Signal Process. Image Commun..

[116]  Shang-Hong Lai,et al.  Fusing generic objectness and visual saliency for salient object detection , 2011, 2011 International Conference on Computer Vision.

[117]  Benoit M. Macq,et al.  A Rarity-Based Visual Attention Map - Application to Texture Description , 2006, 2006 International Conference on Image Processing.

[118]  Ulrich Engelke,et al.  Visual Attention in Quality Assessment , 2011, IEEE Signal Processing Magazine.

[119]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[120]  Patrick Le Callet,et al.  Overt visual attention for free-viewing and quality assessment tasks Impact of the regions of interest on a video quality metric , 2010 .

[121]  Tommy Strandvall,et al.  Eye Tracking in Human-Computer Interaction and Usability Research , 2009, INTERACT.

[122]  Christof Koch,et al.  Image Signature: Highlighting Sparse Salient Regions , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[123]  Liqing Zhang,et al.  Saliency Detection: A Spectral Residual Approach , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[124]  Pietro Perona,et al.  Selective visual attention enables learning and recognition of multiple objects in cluttered scenes , 2005, Comput. Vis. Image Underst..

[125]  Joseph H. Goldberg,et al.  Identifying fixations and saccades in eye-tracking protocols , 2000, ETRA.

[126]  Martin D. Levine,et al.  Saliency Detection Based on Frequency and Spatial Domain Analyses , 2011, BMVC.

[127]  Koen E. A. van de Sande,et al.  Evaluating Color Descriptors for Object and Scene Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[128]  P. Sterling,et al.  How Much the Eye Tells the Brain , 2006, Current Biology.

[129]  Wei Zhang,et al.  The Application of Visual Saliency Models in Objective Image Quality Assessment: A Statistical Evaluation , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[130]  Yanjiao Chen,et al.  From QoS to QoE: A Tutorial on Video Quality Assessment , 2015, IEEE Communications Surveys & Tutorials.

[131]  G. Keren,et al.  Between- or Within-Subjects Design: A Methodological Dilemma , 1993 .

[132]  Mylène C. Q. Farias,et al.  On performance of image quality metrics enhanced with visual attention computational models , 2012 .

[133]  Antón García-Díaz,et al.  Saliency from hierarchical adaptation through decorrelation and variance normalization , 2012, Image Vis. Comput..

[134]  Ali Borji,et al.  Quantitative Analysis of Human-Model Agreement in Visual Saliency Modeling: A Comparative Study , 2013, IEEE Transactions on Image Processing.

[135]  Zhou Wang,et al.  Multiscale structural similarity for image quality assessment , 2003, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.

[136]  Judith Redi,et al.  Studying the effect of optimizing image quality in salient regions at the expense of background content , 2013, J. Electronic Imaging.

[137]  Ingrid Heynderickx,et al.  How Does Image Content Affect the Added Value of Visual Attention in Objective Image Quality Assessment? , 2013, IEEE Signal Processing Letters.

[138]  Nicolas Riche,et al.  Saliency and Human Fixations: State-of-the-Art and Study of Comparison Metrics , 2013, 2013 IEEE International Conference on Computer Vision.

[139]  Zhou Wang,et al.  Applications of Objective Image Quality Assessment Methods , 2011 .

[140]  Nicolas Riche,et al.  Bottom-Up Saliency Models for Videos: A Practical Review , 2016 .