Introduction: state of the play and challenges of visual quality assessment

Quality of visual signals perceived by human observers has always been a critical issue, so has been the measurement of the signal quality throughout a process chain of acquisition/reproduction, encoding, transmission or storage, decoding, and visualization/display associated with a designated application or service in either analogue or digital form. Digital visual signals compressed using various coding techniques exhibit coding distortions which differ from those known to be associated with analogue visual signals and, therefore, require provision of both subjective and objective distortion or quality measures which quantitatively assess and evaluate the visual picture quality for the purposes of system or service evaluation and optimization. A number of fundamental issues are examined to put the current discussions and activities into perspective and context, including relationship between picture quality assessment and coding designs, how to measure effectiveness of visual signal compression performance, different scales used for visual quality assessment and their intended applications, picture distortion or quality ratings for rate-perceptual-distortion (RpD) optimization.

[1]  C. Harrison Experiments with linear prediction in television , 1952 .

[2]  Shuai Wan,et al.  Bitstream-based quality assessment for networked video: a review , 2012, IEEE Communications Magazine.

[3]  King Ngi Ngan,et al.  Spatio-Temporal Just Noticeable Distortion Profile for Grey Scale Image/Video in DCT Domain , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[4]  R. J. Safranek,et al.  A perceptually tuned sub-band image coder with image dependent quantization and post-quantization data compression , 1989, International Conference on Acoustics, Speech, and Signal Processing,.

[5]  C.E. Shannon,et al.  Communication in the Presence of Noise , 1949, Proceedings of the IRE.

[6]  Weisi Lin,et al.  Computational Models for Just-Noticeable Difference , 2017 .

[7]  Michael Yuen,et al.  A survey of hybrid MC/DPCM/DCT video coding distortions , 1998, Signal Process..

[8]  Patrick C. Teo,et al.  Perceptual image distortion , 1994, Proceedings of 1st International Conference on Image Processing.

[9]  Vincent Ricordel,et al.  Experimenting texture similarity metric STSIM for intra prediction mode selection and block partitioning in HEVC , 2014, 2014 19th International Conference on Digital Signal Processing.

[10]  Alan C. Bovik,et al.  Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures , 2009, IEEE Signal Processing Magazine.

[11]  Hong Ren Wu,et al.  Perceptually lossless medical image coding , 2006, IEEE Transactions on Medical Imaging.

[12]  J A Solomon,et al.  Model of visual contrast gain control and pattern masking. , 1997, Journal of the Optical Society of America. A, Optics, image science, and vision.

[13]  Brian A. Wandell,et al.  Color image fidelity metrics evaluated using image distortion maps , 1998, Signal Process..

[14]  Hong Ren Wu,et al.  No-Reference Quality Assessment for Networked Video via Primary Analysis of Bit Stream , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[15]  Guangming Shi,et al.  Structural uncertainty based just noticeable difference estimation , 2014, 2014 19th International Conference on Digital Signal Processing.

[16]  Jeffrey Lubin,et al.  The use of psychophysical data and models in the analysis of display system performance , 1993 .

[17]  J. R. Pierce,et al.  A new type of high-frequency amplifier , 1949, Bell Syst. Tech. J..

[18]  Margaret H. Pinson,et al.  Audiovisual Quality Components , 2011, IEEE Signal Processing Magazine.

[19]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[20]  Robert J. Safranek,et al.  Signal compression based on models of human perception , 1993, Proc. IEEE.

[21]  Andrew B. Watson,et al.  The cortex transform: rapid computation of simulated neural images , 1987 .

[22]  Michael W. Marcellin,et al.  Visually Lossless Encoding for JPEG2000 , 2013, IEEE Transactions on Image Processing.

[23]  R. R. Clarke Transform coding of images , 1985 .

[24]  Luigi Atzori,et al.  QoE management in emerging multimedia services , 2012, IEEE Commun. Mag..

[25]  Mohammed Hassan,et al.  Structural Similarity Measure for Color Images , 2012 .

[26]  Jelena Kovacevic,et al.  Wavelets and Subband Coding , 2013, Prentice Hall Signal Processing Series.

[27]  Andrew B. Watson,et al.  Digital images and human vision , 1993 .

[28]  David J. Sakrison,et al.  The effects of a visual fidelity criterion of the encoding of images , 1974, IEEE Trans. Inf. Theory.

[29]  Alan C. Bovik,et al.  Image information and visual quality , 2006, IEEE Trans. Image Process..

[30]  Martin Reisslein,et al.  Objective Video Quality Assessment Methods: A Classification, Review, and Performance Comparison , 2011, IEEE Transactions on Broadcasting.

[31]  J. B. O'Neal,et al.  Predictive quantizing systems (differential pulse code modulation) for the transmission of television signals , 1966 .

[32]  Masayuki Tanimoto,et al.  Multiview Imaging and 3DTV , 2007, IEEE Signal Processing Magazine.

[33]  Heidi A. Peterson,et al.  Luminance-model-based DCT quantization for color image compression , 1992, Electronic Imaging.

[34]  Nikolay N. Ponomarenko,et al.  A NEW FULL-REFERENCE QUALITY METRICS BASED ON HVS , 2006 .

[35]  D. Chandler Seven Challenges in Image Quality Assessment: Past, Present, and Future Research , 2013 .

[36]  Weisi Lin,et al.  Perceptual Visual Signal Compression and Transmission , 2013, Proceedings of the IEEE.

[37]  V. Ralph Algazi,et al.  Objective picture quality scale (PQS) for image coding , 1998, IEEE Trans. Commun..

[38]  Edward H. Adelson,et al.  Shiftable multiscale transforms , 1992, IEEE Trans. Inf. Theory.

[39]  Hong Ren Wu,et al.  Perceptual coding of digital monochrome images , 2004, IEEE Signal Processing Letters.

[40]  Marcus Barkowsky,et al.  Video quality assessment: From 2D to 3D — Challenges and future trends , 2010, 2010 IEEE International Conference on Image Processing.

[41]  Andrew B. Watson,et al.  DCTune: A TECHNIQUE FOR VISUAL OPTIMIZATION OF DCT QUANTIZATION MATRICES FOR INDIVIDUAL IMAGES. , 1993 .

[42]  Zhou Wang,et al.  Introduction to the Issue on Visual Media Quality Assessment , 2009, IEEE J. Sel. Top. Signal Process..

[43]  M. Miyahara Quality assessments for visual service , 1988, IEEE Communications Magazine.

[44]  C. Lambrecht Perceptual models and architectures for video coding applications , 1996 .

[45]  M. G. Ramos,et al.  Suprathreshold wavelet coefficient quantization in complex stimuli: psychophysical evaluation and analysis. , 2001, Journal of the Optical Society of America. A, Optics, image science, and vision.

[46]  Scott J. Daly,et al.  Visible differences predictor: an algorithm for the assessment of image fidelity , 1992, Electronic Imaging.

[47]  Makoto Miyahara,et al.  Philosophy of Picture Quality Scale , 2005 .

[48]  Weisi Lin,et al.  Perceptual visual quality metrics: A survey , 2011, J. Vis. Commun. Image Represent..

[49]  Hong Ren Wu,et al.  Multiple reference impairment scale subjective assessment method for digital video , 2002, 2002 14th International Conference on Digital Signal Processing Proceedings. DSP 2002 (Cat. No.02TH8628).

[50]  Sheila S. Hemami,et al.  No-reference image and video quality estimation: Applications and human-motivated design , 2010, Signal Process. Image Commun..

[51]  Zhen Liu,et al.  JPEG2000 encoding with perceptual distortion control , 2006, IEEE Transactions on Image Processing.

[52]  Pascal Frossard,et al.  Advanced Solutions for Quality-Oriented Multimedia Broadcasting , 2008 .

[53]  Robert J. Safranek,et al.  JPEG compliant encoder utilizing perceptually based quantization , 1994, Electronic Imaging.

[54]  Bernd Girod,et al.  What's wrong with mean-squared error? , 1993 .

[55]  Zhenghua Yu,et al.  Vision-model-based impairment metric to evaluate blocking artifacts in digital video , 2002, Proc. IEEE.

[56]  W. M. Goodall Television by pulse code modulation , 1951 .

[57]  Manoranjan Paul,et al.  Just Noticeable Difference for Images With Decomposition Model for Separating Edge and Textured Regions , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[58]  J. M. Foley,et al.  Human luminance pattern-vision mechanisms: masking experiments require a new model. , 1994, Journal of the Optical Society of America. A, Optics, image science, and vision.

[59]  Hong Ren Wu,et al.  Digital Video Image Quality and Perceptual Coding , 2005 .

[60]  Susu Yao,et al.  Just noticeable distortion model and its applications in video coding , 2005, Signal Process. Image Commun..

[61]  P. Green Fiber Optic Networks , 1992 .

[62]  Alan C. Bovik,et al.  A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms , 2006, IEEE Transactions on Image Processing.

[63]  Richard H. Sherman,et al.  Chaotic communications in the presence of noise , 1993, Optics & Photonics.

[64]  Hong Ren Wu,et al.  Perceptual Color Image Coding With JPEG2000 , 2010, IEEE Transactions on Image Processing.

[65]  B. Wandell Foundations of vision , 1995 .

[66]  Kuo-Cheng Liu,et al.  A Perceptually Tuned Watermarking Scheme for Color Images , 2010, IEEE Transactions on Image Processing.

[67]  Albert J. Ahumada,et al.  Improved detection model for DCT coefficient quantization , 1993, Electronic Imaging.

[68]  David M. Hoffman,et al.  Perceptual Issues in Stereoscopic Signal Processing , 2011, IEEE Transactions on Broadcasting.

[69]  David L. Neuhoff,et al.  Image Analysis: Focus on Texture Similarity , 2013, Proceedings of the IEEE.

[70]  David M. Rouse,et al.  Estimating the usefulness of distorted natural images using an image contour degradation measure. , 2011, Journal of the Optical Society of America. A, Optics, image science, and vision.

[71]  Michael Anthony Isnardi,et al.  HVS Based Perceptual Video Encoders , 2017 .

[72]  King Ngi Ngan,et al.  Rate-perceptual-distortion optimization (RpDO) based picture coding — Issues and challenges , 2014, 2014 19th International Conference on Digital Signal Processing.

[73]  Stefan Winkler,et al.  Stereo/multiview picture quality: Overview and recent advances , 2013, Signal Process. Image Commun..

[74]  Gary J. Sullivan,et al.  Comparison of the Coding Efficiency of Video Coding Standards—Including High Efficiency Video Coding (HEVC) , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[75]  Stephen D. Voran,et al.  Objective video quality assessment system based on human perception , 1993, Electronic Imaging.

[76]  Chun-Hsien Chou,et al.  A perceptually optimized 3-D subband codec for video communication over wireless channels , 1996, IEEE Trans. Circuits Syst. Video Technol..

[77]  Lew B. Stelmach,et al.  All subjective scales are not created equal: The effects of context on different scales , 1999, Signal Process..

[78]  Richard A. Johnson,et al.  Statistics: Principles and Methods , 1985 .

[79]  Andrew B. Watson Receptive Fields And Visual Representations , 1989, Photonics West - Lasers and Applications in Science and Engineering.

[80]  Zhou Wang,et al.  Information Content Weighting for Perceptual Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[81]  Weisi Lin,et al.  Estimating Just-Noticeable Distortion for Video , 2006, IEEE Transactions on Circuits and Systems for Video Technology.

[82]  Stefan Winkler,et al.  Perceptual distortion metric for digital color video , 1999, Electronic Imaging.

[83]  Eric C. Larson,et al.  Most apparent distortion: full-reference image quality assessment and the role of strategy , 2010, J. Electronic Imaging.

[84]  D. Tolhurst,et al.  The effects of amplitude-spectrum statistics on foveal and peripheral discrimination of changes in natural images, and a multi-resolution model , 2005, Vision Research.

[85]  David L. Neuhoff,et al.  Structural Texture Similarity Metrics for Image Analysis and Retrieval , 2013, IEEE Transactions on Image Processing.

[86]  Michel Barlaud,et al.  Image coding using wavelet transform , 1992, IEEE Trans. Image Process..

[87]  Z. L. Budrikis,et al.  Visual fidelity criterion and modeling , 1972 .

[88]  Andrew B. Watson,et al.  Image quality and entropy masking , 1997, Electronic Imaging.

[89]  Zhou Wang,et al.  Complex Wavelet Structural Similarity: A New Image Similarity Index , 2009, IEEE Transactions on Image Processing.

[90]  Sheila S. Hemami,et al.  VSNR: A Wavelet-Based Visual Signal-to-Noise Ratio for Natural Images , 2007, IEEE Transactions on Image Processing.

[91]  J. Astola,et al.  ON BETWEEN-COEFFICIENT CONTRAST MASKING OF DCT BASIS FUNCTIONS , 2007 .

[92]  Zhou Wang,et al.  Video quality assessment based on structural distortion measurement , 2004, Signal Process. Image Commun..

[93]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[94]  Toby Berger,et al.  Lossy Source Coding , 1998, IEEE Trans. Inf. Theory.

[95]  Philip J. Corriveau Video Quality Testing , 2005 .

[96]  Eero P. Simoncelli,et al.  Image denoising using a local Gaussian scale mixture model in the wavelet domain , 2000, SPIE Optics + Photonics.

[97]  Sebastian Möller,et al.  Multimedia Quality Assessment Standards in ITU-T SG12 , 2011, IEEE Signal Processing Magazine.

[98]  Patrick Le Callet,et al.  Objective quality assessment of color images based on a generic perceptual reduced reference , 2008, Signal Process. Image Commun..

[99]  N. Jayant,et al.  Digital Coding of Waveforms: Principles and Applications to Speech and Video , 1990 .

[100]  Sheila S. Hemami,et al.  Dynamic contrast-based quantization for lossy wavelet image compression , 2005, IEEE Transactions on Image Processing.

[101]  M. Luo,et al.  The development of the CIE 2000 Colour Difference Formula , 2001 .

[102]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[103]  Eero P. Simoncelli,et al.  Random Cascades on Wavelet Trees and Their Use in Analyzing and Modeling Natural Images , 2001 .

[104]  Lina J. Karam,et al.  Locally adaptive perceptual image coding , 2000, IEEE Trans. Image Process..

[105]  Tao Chen,et al.  3D-TV Content Storage and Transmission , 2011, IEEE Transactions on Broadcasting.

[106]  J. O. Limb Source-receiver encoding of television signals , 1967 .

[107]  Andrew F. Inglis,et al.  Video Engineering , 1992 .

[108]  Stefan Winkler,et al.  The Evolution of Video Quality Measurement: From PSNR to Hybrid Metrics , 2008, IEEE Transactions on Broadcasting.

[109]  Levent Onural,et al.  Television in 3-D: What Are the Prospects? , 2007 .

[110]  Weisi Lin,et al.  Introduction to the Special Issue on New Subjective and Objective Methodologies for Audio and Visual Signal Processing , 2012, IEEE J. Sel. Top. Signal Process..

[111]  Alan C. Bovik,et al.  Automatic Prediction of Perceptual Image and Video Quality , 2013, Proceedings of the IEEE.

[112]  Chun-Hsien Chou,et al.  A perceptually tuned subband image coder based on the measure of just-noticeable-distortion profile , 1995, IEEE Trans. Circuits Syst. Video Technol..

[113]  Zhou Wang,et al.  Video quality assessment using a statistical model of human visual speed perception. , 2007, Journal of the Optical Society of America. A, Optics, image science, and vision.

[114]  Peter Kauff,et al.  Production Rules for Stereo Acquisition , 2011, Proceedings of the IEEE.

[115]  Margaret H. Pinson,et al.  A new standardized method for objectively measuring video quality , 2004, IEEE Transactions on Broadcasting.