Issues in vision modeling for perceptual video quality assessment

Lossy compression algorithms used in digital video systems produce artifacts whose visibility strongly depends on the actual image content. Simple error measures such as RMSE or PSNR, albeit popular, ignore this important fact and are only a mediocre predictor of perceived quality. Many applications require more reliable assessment methods. This paper discusses issues in vision modeling for perceptual video quality assessment (PVQA). Its purpose is not to describe a particular model or system, but rather to summarize and to provide pointers to up-to-date knowledge of important characteristics of the human visual system, to explain how these characteristics may be incorporated in vision models for PVQA, to give a brief overview of the state-of-the-art and current efforts in this field, and to outline directions for future research. Verlustbehaftete Kompressionsalgorithmen, wie sie in digitalen Video-Systemen verwendet werden, erzeugen Artefakte, deren Sichtbarkeit stark vom Bildinhalt ab

[1]  E. Peli Contrast in complex images. , 1990, Journal of the Optical Society of America. A, Optics and image science.

[2]  J. Daugman Two-dimensional spectral analysis of cortical receptive field profiles , 1980, Vision Research.

[3]  Anil C. Kokaram,et al.  Perceptual distortion measure for edgelike artifacts in image sequences , 1998, Electronic Imaging.

[4]  J J Koenderink,et al.  Spatiotemporal contrast detection threshold surface is bimodal. , 1979, Optics letters.

[5]  Wa James Tam,et al.  Static and dynamic spatial resolution in image coding: an investigation of eye movements , 1991, Electronic Imaging.

[6]  Yuukou Horita,et al.  Objective picture quality scale for video coding , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[7]  C. A. Burbeck,et al.  Spatiotemporal characteristics of visual mechanisms: excitatory-inhibitory model. , 1980, Journal of the Optical Society of America.

[8]  C. Stromeyer,et al.  Visual interactions with luminance and chromatic stimuli. , 1990, Journal of the Optical Society of America. A, Optics and image science.

[9]  Stanley A. Klein,et al.  Image quality and image compression: a psychophysicist's viewpoint , 1993 .

[10]  D. Pollen,et al.  Spatial and temporal frequency selectivity of neurones in visual cortical areas V1 and V2 of the macaque monkey. , 1985, The Journal of physiology.

[11]  J A Solomon,et al.  Model of visual contrast gain control and pattern masking. , 1997, Journal of the Optical Society of America. A, Optics, image science, and vision.

[12]  B. Wandell,et al.  Pattern—color separable pathways predict sensitivity to simple colored patterns , 1996, Vision Research.

[13]  Konstantin Bogdanov,et al.  Optics of the Eye , 2000 .

[14]  Jaj Jacques Roufs,et al.  PERCEPTUAL IMAGE QUALITY: CONCEPT AND MEASUREMENT , 1992 .

[15]  D. H. Kelly Motion and vision. II. Stabilized spatio-temporal threshold surface. , 1979, Journal of the Optical Society of America.

[16]  R. F. Hess,et al.  Temporal properties of human visual filters: number, shapes and spatial covariation , 1992, Vision Research.

[17]  Michael P. Eckert,et al.  The significance of eye movements and image acceleration for coding television image sequences , 1993 .

[18]  J. Ross,et al.  Contrast adaptation and contrast masking in human vision , 1991, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[19]  Patrick C. Teo,et al.  Perceptual image distortion , 1994, Proceedings of 1st International Conference on Image Processing.

[20]  Andrew B. Watson,et al.  The cortex transform: rapid computation of simulated neural images , 1987 .

[21]  S. R. Lehky,et al.  Temporal properties of visual channels measured by masking. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[22]  Jean-Bernard Martens,et al.  Image quality prediction in a multidimensional perceptual space , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[23]  Albert J. Ahumada,et al.  Computational image quality metrics: A review , 1993 .

[24]  Jeffrey Lubin,et al.  A VISUAL DISCRIMINATION MODEL FOR IMAGING SYSTEM DESIGN AND EVALUATION , 1995 .

[25]  M. Webster,et al.  Adaptation and the color statistics of natural images , 1997, Vision Research.

[26]  Scott Daly,et al.  Engineering observations from spatiovelocity and spatiotemporal visual models , 1998, Electronic Imaging.

[27]  Toshikazu Matsui,et al.  New human vision system model for spatio-temporal image signals , 1991, Electronic Imaging.

[28]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[29]  Albert A. Michelson,et al.  Studies in Optics , 1995 .

[30]  Jian Yang,et al.  Do Gabor functions provide appropriate descriptions of visual cortical receptive fields?: comment , 1992 .

[31]  A. Milam,et al.  Distribution and morphology of human cone photoreceptors stained with anti‐blue opsin , 1991, The Journal of comparative neurology.

[32]  O. Schade Optical and photoelectric analog of the eye. , 1956, Journal of the Optical Society of America.

[33]  R. Hess,et al.  Temporal detection in human vision: dependence on stimulus energy. , 1997, Journal of the Optical Society of America. A, Optics, image science, and vision.

[34]  J. Daugman Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[35]  Per Lindh,et al.  Efficient spatio-temporal decomposition for perceptual processing of video sequences , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[36]  Limin Wang,et al.  Visual masking at video scene cuts , 1995, Electronic Imaging.

[37]  R. Vimal,et al.  ORIENTATION TUNING OF THE SPATIAL-FREQUENCY-TUNED MECHANISMS OF THE RED-GREEN CHANNEL , 1997 .

[38]  Stephane Pefferkorn,et al.  Perceptual quality metric of color quantization errors on still images , 1998, Electronic Imaging.

[39]  Hideaki Haneishi,et al.  Analysis of the Eye Movements and its Applications to Image Evaluation , 1994, Color Imaging Conference.

[40]  Olivier Verscheure,et al.  Perceptual quality measure using a spatiotemporal model of the human visual system , 1996, Electronic Imaging.

[41]  J. V. van Hateren,et al.  Independent component filters of natural images compared with simple cells in primary visual cortex , 1998, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[42]  H. Wilson Spatiotemporal characterization of a transient mechanism in the human visual system , 1980, Vision Research.

[43]  Albert J. Ahumada,et al.  Image quality: a multidimensional problem , 1993 .

[44]  J. Rovamo,et al.  Foveal optical modulation transfer function of the human eye at various pupil sizes. , 1998, Journal of the Optical Society of America. A, Optics, image science, and vision.

[45]  William Y. Zou,et al.  Performance Evaluation: From NTSC to Digitally Compressed Video , 1994 .

[46]  Eero P. Simoncelli,et al.  Non-separable extensions of quadrature mirror filters to multiple dimensions , 1990, Proc. IEEE.

[47]  Mohammed Ghanbari,et al.  Recency effect in the subjective assessment of digitally-coded television pictures , 1995 .

[48]  Eero P. Simoncelli Shiftable multi-scale transforms [or "What's wrong with orthonormal wavelets"] , 1992 .

[49]  Robert Eriksson,et al.  Spatiotemporal discrimination model predicts temporal masking functions , 1998, Electronic Imaging.

[50]  D. Jameson,et al.  Some Quantitative Aspects of an Opponent-Colors Theory. I. Chromatic Responses and Spectral Saturation , 1955 .

[51]  Walter Makous,et al.  Implicit masking constrained by spatial inhomogeneities , 1997, Vision Research.

[52]  Chuan Yi Tang,et al.  A 2.|E|-Bit Distributed Algorithm for the Directed Euler Trail Problem , 1993, Inf. Process. Lett..

[53]  Stefan Winkler,et al.  Computing isotropic local contrast from oriented pyramid decompositions , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[54]  H. Wilson,et al.  Spatial frequency adaptation and contrast gain control , 1993, Vision Research.

[55]  Stefan Winkler,et al.  Perceptual distortion metric for digital color video , 1999, Electronic Imaging.

[56]  J P Thomas,et al.  Effect of pattern adaptation on spatial frequency discrimination. , 1992, Journal of the Optical Society of America. A, Optics and image science.

[57]  Huib de Ridder,et al.  Perceptually optimal color reproduction , 1998, Electronic Imaging.

[58]  Stephen D. Voran,et al.  Objective video quality assessment system based on human perception , 1993, Electronic Imaging.

[59]  Michael Yuen,et al.  A survey of hybrid MC/DPCM/DCT video coding distortions , 1998, Signal Process..

[60]  Paul J. Hearty Achieving and confirming optimum image quality , 1993 .

[61]  Brian A. Wandell,et al.  Color image fidelity metrics evaluated using image distortion maps , 1998, Signal Process..

[62]  D. Jameson,et al.  An opponent-process theory of color vision. , 1957, Psychological review.

[63]  J. Robson Spatial and Temporal Contrast-Sensitivity Functions of the Visual System , 1966 .

[64]  A B Watson,et al.  Perceptual-components architecture for digital video. , 1990, Journal of the Optical Society of America. A, Optics and image science.

[65]  A. J. Seyler,et al.  Measurements of Temporal Adaptation to Spatial Detail Vision , 1959, Nature.

[66]  Quick Rf A vector-magnitude model of contrast detection. , 1974 .

[67]  D G Stork,et al.  Do Gabor functions provide appropriate descriptions of visual cortical receptive fields? , 1990, Journal of the Optical Society of America. A, Optics and image science.

[68]  M. Webster,et al.  Contrast adaptation and the spatial structure of natural images. , 1997, Journal of the Optical Society of America. A, Optics, image science, and vision.

[69]  A Bradley,et al.  Contrast dependence and mechanisms of masking interactions among chromatic and luminance gratings. , 1988, Journal of the Optical Society of America. A, Optics and image science.

[70]  F. Campbell,et al.  Optical quality of the human eye , 1966, The Journal of physiology.

[71]  R. Hess,et al.  Estimating multiple temporal mechanisms in human vision , 1998, Vision Research.

[72]  William T. Freeman,et al.  Presented at: 2nd Annual IEEE International Conference on Image , 1995 .

[73]  Zigmantas L. Budrikis,et al.  Detail perception after scene changes in television image presentations , 1965, IEEE Trans. Inf. Theory.

[74]  Anthony J. Maeder,et al.  Limiting human perception for image sequences , 1996, Electronic Imaging.

[75]  Murat Kunt,et al.  Motion rendition quality metric for MPEG coded video , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[76]  A. Hendrickson,et al.  Human photoreceptor topography , 1990, The Journal of comparative neurology.

[77]  R. Snowden,et al.  Spatial frequency adaptation: Threshold elevation and perceived contrast , 1996, Vision Research.

[78]  D J Field,et al.  Relations between the statistics of natural images and the response properties of cortical cells. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[79]  Charles A. Poynton,et al.  Rehabilitation of gamma , 1998, Electronic Imaging.

[80]  B. Wandell,et al.  Appearance of colored patterns: pattern-color separability. , 1993, Journal of the Optical Society of America. A, Optics, image science, and vision.

[81]  A. T. Smith,et al.  Two temporal channels or three? A re-evaluation , 1992, Vision Research.

[82]  K. Mullen,et al.  The spatial tuning of chromatic mechanisms identified by simultaneous masking , 1994, Vision Research.

[83]  D. G. Albrecht,et al.  Motion selectivity and the contrast-response function of simple cells in the visual cortex , 1991, Visual Neuroscience.

[84]  KIM T. BLACKWELL,et al.  PII: S0042-6989(97)00130-2 , 2003 .

[85]  J. Cohen,et al.  Color Science: Concepts and Methods, Quantitative Data and Formulas , 1968 .

[86]  Mark E. Gorzynski,et al.  CRT colorimetry. part I: Theory and practice , 1993 .

[87]  G Westheimer,et al.  Optical performances of human eyes derived from double-pass measurements. , 1995, Journal of the Optical Society of America. A, Optics, image science, and vision.

[88]  Walter Makous,et al.  Spatiotemporal separability in contrast sensitivity , 1994, Vision Research.

[89]  Stanley A. Klein,et al.  Visual masking near spatiotemporal edges , 1996, Electronic Imaging.

[90]  Bernd Girod,et al.  The Information Theoretical Significance of Spatial and Temporal Masking in Video Signals , 1989, Photonics West - Lasers and Applications in Science and Engineering.

[91]  G. Wyszecki,et al.  Color Science Concepts and Methods , 1982 .

[92]  Edward H. Adelson,et al.  Shiftable multiscale transforms , 1992, IEEE Trans. Inf. Theory.

[93]  H. Spekreijse,et al.  An improved mathematical description of the foveal visual point spread function with parameters for age, pupil size and pigmentation , 1993, Vision Research.

[94]  Y. L. Grand,et al.  Optics of the Eye , 1980 .

[95]  Wa James Tam,et al.  Processing image sequences based on eye movements , 1994, Electronic Imaging.

[96]  Gunther Wyszecki,et al.  Color Science: Concepts and Methods, Quantitative Data and Formulae, 2nd Edition , 2000 .

[97]  Arnold M. Lund The Influence of Video Image Size and Resolution on Viewing-Distance Preferences , 1993 .

[98]  Sugato Chakravarty,et al.  Methodology for the subjective assessment of the quality of television pictures , 1995 .

[99]  J. M. Foley,et al.  Human luminance pattern-vision mechanisms: masking experiments require a new model. , 1994, Journal of the Optical Society of America. A, Optics, image science, and vision.

[100]  Andrew B. Watson,et al.  Toward a perceptual video-quality metric , 1998, Electronic Imaging.

[101]  R. L. Pandey Vimal,et al.  Orientation tuning of the spatial-frequency-tuned mechanisms of the red-green channel. , 1997, Journal of the Optical Society of America. A, Optics, image science, and vision.

[102]  P K Ahnelt,et al.  The photoreceptor mosaic , 1998, Eye.

[103]  M P Eckstein,et al.  Visual signal detection in structured backgrounds. II. Effects of contrast gain control, background variations, and white noise. , 1997, Journal of the Optical Society of America. A, Optics, image science, and vision.

[104]  Z. L. Budrikis,et al.  Picture Quality Prediction Based on a Visual Model , 1982, IEEE Trans. Commun..

[105]  S A Klein,et al.  Minimizing and maximizing the joint space-spatial frequency uncertainty of Gabor-like functions: comment. , 1992, Journal of the Optical Society of America. A, Optics and image science.

[106]  RussLL L. Ds Vnlos,et al.  SPATIAL FREQUENCY SELECTIVITY OF CELLS IN MACAQUE VISUAL CORTEX , 2022 .

[107]  D. Heeger Half-squaring in responses of cat striate cells , 1992, Visual Neuroscience.

[108]  D. H. Kelly Spatiotemporal variation of chromatic and achromatic contrast thresholds. , 1983, Journal of the Optical Society of America.

[109]  Reginald L. Lagendijk,et al.  Spatiotemporal model of human vision for digital video compression , 1997, Electronic Imaging.

[110]  Stefan Winkler,et al.  A perceptual distortion metric for digital color images , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[111]  L. Kaufman,et al.  Handbook of perception and human performance , 1986 .

[112]  Thom Carney,et al.  Mindseye: a visual programming and modeling environment for imaging science , 1998, Electronic Imaging.

[113]  M. Miranda,et al.  [The eye as an optical instrument]. , 1978, Boletin de la Asociacion Medica de Puerto Rico.

[114]  H. Wilson,et al.  Orientation bandwidths of spatial mechanisms measured by masking. , 1984, Journal of the Optical Society of America. A, Optics and image science.

[115]  Huib de Ridder Minkowski-metrics as a combination rule for digital-image-coding impairments , 1992 .

[116]  M. Ghanbari,et al.  An objective measurement tool for MPEG video quality , 1998, Signal Process..

[117]  Stanley A. Klein,et al.  Seven models of masking , 1997, Electronic Imaging.

[118]  Christian J. Van Den Branden Lambrecht Color moving pictures quality metric , 1996, ICIP.

[119]  Huib de Ridder,et al.  Naturalness and image quality: chroma and hue variation in color images of natural scenes , 1995, Electronic Imaging.

[120]  D. Macleod,et al.  Spectral sensitivities of the human cones. , 1993, Journal of the Optical Society of America. A, Optics, image science, and vision.

[121]  David J. Sakrison,et al.  The effects of a visual fidelity criterion of the encoding of images , 1974, IEEE Trans. Inf. Theory.

[122]  K. D. De Valois,et al.  Orientation and spatial-frequency discrimination for luminance and chromatic gratings. , 1990, Journal of the Optical Society of America. A, Optics and image science.

[123]  D. Heeger Normalization of cell responses in cat striate cortex , 1992, Visual Neuroscience.

[124]  D H Kelly,et al.  Motion and vision. I. Stabilized images of stationary gratings. , 1979, Journal of the Optical Society of America.

[125]  Andrew B. Watson,et al.  Image quality and entropy masking , 1997, Electronic Imaging.

[126]  Charles A. Poynton,et al.  A technical introduction to digital video , 1996 .

[127]  K. Mullen,et al.  Color and luminance spatial tuning estimated by noise masking in the absence of off-frequency looking. , 1995, Journal of the Optical Society of America. A, Optics, image science, and vision.

[128]  M. B. Mandler,et al.  A three channel model of temporal frequency perception , 1984, Vision Research.

[129]  R. L. Valois,et al.  The orientation and direction selectivity of cells in macaque visual cortex , 1982, Vision Research.

[130]  W. Weibull A Statistical Distribution Function of Wide Applicability , 1951 .

[131]  C. Lambrecht Perceptual models and architectures for video coding applications , 1996 .

[132]  B. Wandell,et al.  Matching color images: the effects of axial chromatic aberration , 1994 .

[133]  Scott J. Daly,et al.  Visible differences predictor: an algorithm for the assessment of image fidelity , 1992, Electronic Imaging.

[134]  J. H. Hateren,et al.  Independent component filters of natural images compared with simple cells in primary visual cortex , 1998 .

[135]  Robert W. G. Hunt,et al.  The reproduction of colour , 1957 .

[136]  Jacob Nachmias,et al.  On the psychometric function for contrast detection , 1981, Vision Research.

[137]  E. Peli In search of a contrast metric: Matching the perceived contrast of gabor patches at different phases and bandwidths , 1997, Vision Research.