Survey of full-reference image quality metrics

This paper aims to give a survey of full-reference image quality metrics, including metrics specifically designed to evaluate image quality, but also metrics for image difference, image fidelity, and more. These metrics have in common that they try to predict the perceived difference between an original image and a modified version of it, this modification can typically be compression, halftoning and blurring. They output one numerical value and/or an image difference map. More than 100 image quality metrics have been reviewed and categorized, and short descriptions and analysis of all metrics and their relationships are given. This should prove valuable to researchers in various fields, to find the most appropriate metric for their application, and to give a better understanding of the state of the art of the field of image quality metrics.

[1]  B. Duval.  Commission internationale de l’éclairage (CIE) , 2001, Optique Photonique.

[2]  Ravi Sankar,et al.  An algorithm for image quality assessment , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[3]  Norimichi Tsumura,et al.  Perceptual color difference metric for complex images based on Mahalanobis distance , 2001, J. Electronic Imaging.

[4]  Mark D. Fairchild,et al.  Darwinism of Color Image Difference Models , 2001, CIC.

[5]  R. J. Safranek,et al.  A perceptually tuned sub-band image coder with image dependent quantization and post-quantization data compression , 1989, International Conference on Acoustics, Speech, and Signal Processing,.

[6]  Joyce E. Farrell,et al.  Perceptual quality metric for digitally coded color images , 1996, 1996 8th European Signal Processing Conference (EUSIPCO 1996).

[7]  E. Peli.  Contrast in complex images. , 1990, Journal of the Optical Society of America. A, Optics and image science.

[8]  Yangli Hector Yee,et al.  A perceptual metric for production testing , 2004, SIGGRAPH '04.

[9]  Patrick C. Teo,et al.  Perceptual image distortion , 1994, Proceedings of 1st International Conference on Image Processing.

[10]  C.-C. Jay Kuo,et al.  New image compression artifact measure using wavelets , 1997, Electronic Imaging.

[11]  Pasi Fränti.  Blockwise distortion measure for statistical and structural errors in digital images , 1998, Signal Process. Image Commun..

[12]  Takahiko Horiuchi,et al.  Image Quality Assessment for Color Halftone Images Based on Color Structural Similarity , 2008, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[13]  Alexander Toet,et al.  A new universal colour image fidelity metric , 2003 .

[14]  Mark D. Fairchild,et al.  A top down description of S-CIELAB and CIEDE2000 , 2003 .

[15]  Paul S. Fisher,et al.  Image quality measures and their performance , 1995, IEEE Trans. Commun..

[16]  Christine D. Piatko,et al.  Comparing Real and Synthetic Images: Some Ideas about Metrics , 1995, Rendering Techniques.

[17]  Rui Seara,et al.  A comparison of image quality metric performances under practical conditions , 2005, IEEE International Conference on Image Processing 2005.

[18]  Zhou Wang,et al.  Translation insensitive image similarity in complex wavelet domain , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[19]  Michael Elad,et al.  Space-dependent color gamut mapping: a variational approach , 2005, IEEE Transactions on Image Processing.

[20]  Norman B. Nill,et al.  A Visual Model Weighted Cosine Transform for Image Compression and Quality Assessment , 1985, IEEE Trans. Commun..

[21]  Andrew P. Bradley,et al.  A wavelet visible difference predictor , 1999, IEEE Trans. Image Process..

[22]  Rik Van de Walle,et al.  Multichannel distribution for universal multimedia access in home media gateways , 2006 .

[23]  A. Beghdadi,et al.  Color Image Assessment Using Spatial Extension to CIE DE2000 , 2008, 2008 Digest of Technical Papers - International Conference on Consumer Electronics.

[24]  Andrew B. Watson,et al.  DCTune: A TECHNIQUE FOR VISUAL OPTIMIZATION OF DCT QUANTIZATION MATRICES FOR INDIVIDUAL IMAGES. , 1993 .

[25]  David L. Neuhoff,et al.  Least-squares model-based halftoning , 1999, IEEE Trans. Image Process..

[26]  Rui Seara,et al.  A measure for perceptual image quality assessment , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[27]  S.A. Karunasekera,et al.  A distortion measure for image artifacts based on human visual sensitivity , 1994, Proceedings of ICASSP '94. IEEE International Conference on Acoustics, Speech and Signal Processing.

[28]  Richard J. Herrnstein,et al.  A source book in the history of psychology , 1965 .

[29]  Arne Valberg,et al.  Towards a uniform color space: A better formula to describe the munsell and OSA color scales , 1986 .

[30]  Patrick C. Teo,et al.  A model of perceptual image fidelity , 1995, Proceedings., International Conference on Image Processing.

[31]  Eric C. Larson,et al.  Unveiling relationships between regions of interest and image fidelity metrics , 2008, Electronic Imaging.

[32]  Thrasyvoulos N. Pappas,et al.  Structural Similarity Quality Metrics in a Coding Context: Exploring the Space of Realistic Distortions , 2006, IEEE Transactions on Image Processing.

[33]  David J. Sakrison,et al.  The effects of a visual fidelity criterion of the encoding of images , 1974, IEEE Trans. Inf. Theory.

[34]  Alan C. Bovik,et al.  Image information and visual quality , 2006, IEEE Trans. Image Process..

[35]  Peter G. J. Barten,et al.  Evaluation of Subjective Image Quality with the Square Root Integral Method , 1990, Applied Vision.

[36]  Hans-Peter Seidel,et al.  Visible difference predicator for high dynamic range images , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[37]  Eric P. Lam,et al.  An image similarity measure using homogeneity regions and structure , 2008, Electronic Imaging.

[38]  Mong-Shu Lee,et al.  Image Similarity Comparison Using Dual-Tree Wavelet Transform , 2006, PSIVT.

[39]  V. Ralph Algazi,et al.  Objective picture quality scale (PQS) for image coding , 1998, IEEE Trans. Commun..

[40]  Jean-Charles Pinoli,et al.  A model for logarithmic image processing , 1988 .

[41]  Ahmet M. Eskicioglu,et al.  Multidimensional image quality measure using singular value decomposition , 2003, IS&T/SPIE Electronic Imaging.

[42]  S. L. Guth,et al.  Heterochromatic additivity, foveal spectral sensitivity, and a new color model. , 1973, Journal of the Optical Society of America.

[43]  Ján Morovic,et al.  What Differences Do Observers See In Colour Image Reproduction Experiments? , 2002, CGIV.

[44]  Marius Pedersen,et al.  Importance of region-of-interest on image difference metrics , 2007 .

[45]  Azeddine Beghdadi,et al.  Image Quality Assessment Using the Joint Spatial/Spatial-Frequency Representation , 2006, EURASIP J. Adv. Signal Process..

[46]  Anil C. Kokaram,et al.  Perceptual distortion measure for edgelike artifacts in image sequences , 1998, Electronic Imaging.

[47]  B. Wandell,et al.  Appearance of colored patterns: pattern-color separability. , 1993, Journal of the Optical Society of America. A, Optics, image science, and vision.

[48]  Fjj Frans Blommaert,et al.  Perceptual error measure and its application to sampled and interpolated single-edged images , 1997 .

[49]  Weisi Lin,et al.  Perceived visual quality metric based on error spread and contrast , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[50]  Robert J. Beaton.  Quantitative Models of Image Quality , 1983 .

[51]  Fabio Pellacini,et al.  Functional difference predictors (FDPs): measuring meaningful image differences , 2003, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.

[52]  Peter G. Engeldrum,et al.  Psychometric Scaling: A Toolkit for Imaging Systems Development , 2000 .

[53]  M. Luo,et al.  The development of the CIE 2000 Colour Difference Formula , 2001 .

[54]  Andrew P. Bradley,et al.  Perceptual quality metrics applied to still image compression , 1998, Signal Process..

[55]  R. E. Miller,et al.  Image halftoning using a visual model in error diffusion , 1993 .

[56]  M. Ronnier Luo,et al.  Experimental Filters for Estimating Image Differences , 2008, CGIV/MCS.

[57]  Chun-Ling Yang,et al.  Gradient-Based Structural Similarity for Image Quality Assessment , 2006, 2006 International Conference on Image Processing.

[58]  Kuo-Cheng Liu,et al.  A Fidelity Metric for Assessing Visual Quality of Color Images , 2007, 2007 16th International Conference on Computer Communications and Networks.

[59]  Nikolay N. Ponomarenko,et al.  Color image database for evaluation of image quality metrics , 2008, 2008 IEEE 10th Workshop on Multimedia Signal Processing.

[60]  Lai-Man Po,et al.  Edge-Based Structural Similarity for Image Quality Assessment , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[61]  Gaurav Sharma.  Digital Color Imaging Handbook , 2002 .

[62]  Mark D. Fairchild,et al.  Perception and Modeling of Halftone Image Quality Using a High-Resolution LCD , 2005, Color Imaging Conference.

[63]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[64]  Patrick Le Callet,et al.  An image quality assessment method based on perception of structural information , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[65]  Azeddine Beghdadi,et al.  A new image distortion measure based on wavelet decomposition , 2003, Seventh International Symposium on Signal Processing and Its Applications, 2003. Proceedings..

[66]  P. Mahalanobis.  On the generalized distance in statistics , 1936 .

[67]  Alan C. Bovik,et al.  Image quality assessment using natural scene statistics , 2004 .

[68]  M R Luo,et al.  New algorithm for calculating perceived colour difference of images , 2006 .

[69]  Michel Herbin,et al.  Are existing procedures enough? Image and video quality assessment: review of subjective and objective metrics , 2008, Electronic Imaging.

[70]  Gary W. Meyer,et al.  Comparison of two image quality models , 1998, Electronic Imaging.

[71]  Jon Yngve Hardeberg,et al.  SHAME: A new spatial hue angle metric for perceptual image difference , 2010 .

[72]  Thrasyvoulos N. Pappas,et al.  Using Structural Similarity Quality Metrics to Evaluate Image Compression Techniques , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[73]  Jon Y. Hardeberg,et al.  Using gaze information to improve image difference metrics , 2008, Electronic Imaging.

[74]  Jon Y. Hardeberg,et al.  A New Spatial Hue Angle Metric for Perceptual Image Difference , 2009, CCIW.

[75]  Brian A. Wandell,et al.  Color image fidelity metrics evaluated using image distortion maps , 1998, Signal Process..

[76]  Hans-Peter Seidel,et al.  Predicting visible differences in high dynamic range images: model and its calibration , 2005, IS&T/SPIE Electronic Imaging.

[77]  Dirk W. Hertel.  Exploring s-CIELAB as a scanner metric for print uniformity , 2005, IS&T/SPIE Electronic Imaging.

[78]  Mark D. Fairchild,et al.  iCAM framework for image appearance, differences, and quality , 2004, J. Electronic Imaging.

[79]  C.-C. Jay Kuo,et al.  Perceptual fidelity measure of digital color images , 1998, Electronic Imaging.

[80]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[81]  Kevin J. Parker,et al.  A New Metric for Color Halftone Visibility , 1998, PICS.

[82]  Yalr Kipman.  Image Quality Metrics for Printers and Media , 1998, PICS.

[83]  Teresa H. Meng,et al.  Psycho-visual based distortion measures for monochrome image and video compression , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[84]  Rui Seara,et al.  Perceptual image quality assessment based on Bayesian networks , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[85]  Ahmet M. Eskicioglu,et al.  A full-reference color image quality measure in the DWT domain , 2005, 2005 13th European Signal Processing Conference.

[86]  Qian Lin.  Halftone image quality analysis based on a human vision model , 1993, Electronic Imaging.

[87]  A. Beghdadi,et al.  Image quality assessment using a neural network approach , 2004, Proceedings of the Fourth IEEE International Symposium on Signal Processing and Information Technology, 2004..

[88]  H.-J. Zepernick,et al.  Perceptual-based Quality Metrics for Image and Video Services: A Survey , 2007, 2007 Next Generation Internet Networks.

[89]  J. Lubin.  A human vision system model for objective picture quality measurements , 1997 .

[90]  Michael Elad,et al.  A Variational Framework for Retinex , 2002, IS&T/SPIE Electronic Imaging.

[91]  Stéphane Mallat,et al.  Singularity detection and processing with wavelets , 1992, IEEE Trans. Inf. Theory.

[92]  David Connah,et al.  Can gamut mapping quality be predicted by colour image difference formulae? , 2005, IS&T/SPIE Electronic Imaging.

[93]  Gary W. Meyer,et al.  Visual difference metric for realistic image synthesis , 1999, Electronic Imaging.

[94]  Wilson S. Geisler,et al.  Image quality assessment based on a degradation model , 2000, IEEE Trans. Image Process..

[95]  Azeddine Beghdadi,et al.  A Wigner-Ville distribution-based image dissimilarity measure , 2001, Proceedings of the Sixth International Symposium on Signal Processing and its Applications (Cat.No.01EX467).

[96]  K. Mullen.  The contrast sensitivity of human colour vision to red‐green and blue‐yellow chromatic gratings. , 1985, The Journal of physiology.

[97]  Nick G. Kingsbury,et al.  A distortion measure for blocking artifacts in images based on human visual sensitivity , 1995, IEEE Trans. Image Process..

[98]  Theophano Mitsa,et al.  Evaluation of contrast sensitivity functions for the formulation of quality measures incorporated in halftoning algorithms , 1993, 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[99]  Zygmunt Pizlo,et al.  Color Image Fidelity Assessor , 2001, PICS.

[100]  Tomas Akenine-Möller,et al.  High dynamic range texture compression for graphics hardware , 2006, SIGGRAPH 2006.

[101]  Sos S. Agaian,et al.  Quantifying image similarity using measure of enhancement by entropy , 2007, SPIE Defense + Commercial Sensing.

[102]  Reginald L. Lagendijk,et al.  Perceptual image quality based on a multiple channel HVS model , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[103]  Andrew B. Watson,et al.  DCT quantization matrices visually optimized for individual images , 1993, Electronic Imaging.

[104]  Zhou Wang,et al.  Multi-scale structural similarity for image quality assessment , 2003 .

[105]  M. Ronnier Luo,et al.  Testing Color-Difference Formulae on Complex Images Using a CRT Monitor , 2000, Color Imaging Conference.

[106]  Xiao-Fan Feng,et al.  Halftone Quality Evaluation Using Color Visual Models , 2002, PICS.

[107]  Lindsay W. MacDonald,et al.  Colour Difference Metrics and Image Sharpness , 2000, Color Imaging Conference.

[108]  Kevin J. Parker,et al.  Quality issues in blue noise halftoning , 1998, Electronic Imaging.

[109]  Gary W. Meyer,et al.  A perceptually based adaptive sampling algorithm , 1998, SIGGRAPH.

[110]  Xue Dong Yang,et al.  Existing and emerging image quality metrics , 2005, Canadian Conference on Electrical and Computer Engineering, 2005..

[111]  Nikolay N. Ponomarenko,et al.  A NEW FULL-REFERENCE QUALITY METRICS BASED ON HVS , 2006 .

[112]  Mirko Guarnera,et al.  A perceptual quality metric for color-interpolated images , 2005, IS&T/SPIE Electronic Imaging.

[113]  Olof Bryngdahl,et al.  Texture metric of halftone images , 1996 .

[114]  Mark D. Fairchild,et al.  Meet iCAM: A Next-Generation Color Appearance Model , 2002, Color Imaging Conference.

[115]  Jon Y. Hardeberg,et al.  Rank Order and Image Difference Metrics , 2008, CGIV/MCS.

[116]  Jean Philippe Farrugia,et al.  A Perceptual Adaptive Image Metric for Computer Graphics , 2004 .

[117]  Maryellen L. Giger,et al.  Evaluating the EM algorithm for image processing using a human visual fidelity criterion , 1991, [Proceedings] ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing.

[118]  Elaine W. Jin,et al.  The Development of A Color Visual Difference Model (CVDM) , 1998, PICS.

[119]  László Neumann,et al.  Perception Based Color Image Difference , 1998, Comput. Graph. Forum.

[120]  Fredrik Nilsson.  Objective quality measures for halftoned images , 1999 .

[121]  Mark D. Fairchild,et al.  Statistical Summaries of iCAM Image-Difference Maps , 2003, PICS.

[122]  Robyn A. Owens,et al.  A New Metric for Grey-Scale Image Comparison , 1997, International Journal of Computer Vision.

[123]  R. Nasanen.  Visibility of halftone dot textures , 1984, IEEE Transactions on Systems, Man, and Cybernetics.

[124]  Chun-Hsien Chou,et al.  A perceptually tuned subband image coder based on the measure of just-noticeable-distortion profile , 1995, IEEE Trans. Circuits Syst. Video Technol..

[125]  Norimichi Tsumura,et al.  Evaluation of Image Corrected by Retinex Method Based on S-CIELAB and Gazing Information , 2006, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[126]  Jon Yngve Hardeberg,et al.  Perceptual evaluation of color gamut mapping algorithms , 2008 .

[127]  Brian Bouzas,et al.  Objective image quality measure derived from digital image power spectra , 1992 .

[128]  Hocine Cherifi,et al.  A comparison of image quality models and metrics based on human visual sensitivity , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[129]  Jon Yngve Hardeberg,et al.  Evaluating colour image difference metrics for gamut‐mapped images , 2008 .

[130]  V. Ralph Algazi,et al.  Evaluation of a human-vision-system-based image fidelity metric for image compression , 1999, Optics & Photonics.

[131]  Michael H. Brill,et al.  Perceptual Scaling of Quality Metrics for Hardcopy Image Evaluation , 1999 .

[132]  Zygmunt Pizlo,et al.  The Image Fidelity Assessor , 1998, PICS.

[133]  Elaine Jin,et al.  A Groundtruth Database for Testing Objective Metrics for Image Difference , 2003, PICS.

[134]  Gustavo de Veciana,et al.  An information fidelity criterion for image quality assessment using natural scene statistics , 2005, IEEE Transactions on Image Processing.

[135]  Maeng-Sub Cho,et al.  Experimental Approach for Human Perception Based Image Quality Assessment , 2006, ICEC.

[136]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[137]  Mark D. Fairchild,et al.  The iCAM Framework for Image Appearance, Image Differences, and Image Quality , 2002 .

[138]  Hans Brettel,et al.  Evaluation of Spatial Gamut Mapping Algorithms , 2006, Color Imaging Conference.

[139]  Scott J. Daly,et al.  Visible differences predictor: an algorithm for the assessment of image fidelity , 1992, Electronic Imaging.

[140]  Ahmet M. Eskicioglu,et al.  Quality measurement for monochrome compressed images in the past 25 years , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).

[141]  M. Ronnier Luo,et al.  Colour Differences for Complex Images , 2003, Color Imaging Conference.

[142]  Ján Morovic,et al.  Visual Differences In Colour Reproduction And Their Colorimetric Correlates , 2002, Color Imaging Conference.

[143]  Alain Fournier,et al.  Multiscale edge analysis of halftoned images , 1998, Electronic Imaging.

[144]  D. Amnon Silverstein,et al.  DCT image fidelity metric and its application to a text-based scheme for image display , 1993, Electronic Imaging.

[145]  Olof Bryngdahl,et al.  SPATIALLY DEPENDENT TEXTURE ANALYSIS AND CONTROL IN DIGITAL HALFTONING , 1997 .

[146]  Stefan Winkler,et al.  Issues in vision modeling for perceptual video quality assessment , 1999, Signal Process..

[147]  S. Triantaphillidou,et al.  Predicting image quality using a modular image difference model , 2008, Electronic Imaging.

[148]  Brian A. Wandell,et al.  A spatial extension of CIELAB for digital color‐image reproduction , 1997 .

[149]  Stephane Pefferkorn,et al.  Perceptual quality metric of color quantization errors on still images , 1998, Electronic Imaging.

[150]  Tao Wang,et al.  A Content-Based Image Quality Metric , 2005, RSFDGrC.

[151]  Xiaoxia Wan,et al.  Quality evaluation of the halftone by halftoning algorithm-based methods and adaptive method , 2007, Electronic Imaging.

[152]  S. Nakauchi,et al.  Color gamut mapping based on a perceptual image difference measure , 1999 .

[153]  Ahmet M. Eskicioglu,et al.  An SVD-based grayscale image quality measure for local and global assessment , 2006, IEEE Transactions on Image Processing.

[154]  Hua Li,et al.  Image Quality Assessment Using Rough Fuzzy Integrals , 2007, 27th International Conference on Distributed Computing Systems Workshops - Supplements (ICDCSW'07).

[155]  Alan C. Bovik,et al.  A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms , 2006, IEEE Transactions on Image Processing.

[156]  Takahiko Horiuchi,et al.  Digital Color Image Halftone : Hybrid Error Diffusion Using the Mask Perturbation and Quality Verification , 2007 .

[157]  Nick Holliman,et al.  Stereoscopic image quality metrics and compression , 2008, Electronic Imaging.

[158]  Brian A. Wandell,et al.  Applications of a spatial extension to CIELAB , 1997, Electronic Imaging.

[159]  Rodney L. Miller,et al.  Design of minimum visual modulation halftone patterns , 1991, IEEE Trans. Syst. Man Cybern..

[160]  Brian A. Wandell,et al.  Color image quality metric S-CIELAB and its application on halftone texture visibility , 1997, Proceedings IEEE COMPCON 97. Digest of Papers.

[161]  Weisi Lin,et al.  Image Quality Measure using Curvature Similarity , 2007, 2007 IEEE International Conference on Image Processing.