Perceptual-based quality assessment for audio-visual services: A survey

Accurate measurement of the perceived quality of audio-visual services at the end-user is becoming a crucial issue in digital applications due to the growing demand for compression and transmission of audio-visual services over communication networks. Content providers strive to offer the best quality of experience for customers linked to their different quality of service (QoS) solutions. Therefore, developing accurate, perceptual-based quality metrics is a key requirement in multimedia services. In this paper, we survey state-of-the-art signal-driven perceptual audio and video quality assessment methods independently, and investigate relevant issues in developing joint audio-visual quality metrics. Experiments with respect to subjective quality results have been conducted for analyzing and comparing the performance of the quality metrics. We consider emerging trends in audio-visual quality assessment, and propose feasible solutions for future work in perceptual-based audio-visual quality metrics.

[1]  Thilo Thiede,et al.  A New Perceptual Quality Measure for Bit-Rate Reduced Audio , 1996 .

[2]  Andrew B. Watson,et al.  Digital images and human vision , 1993 .

[3]  Guizhong Liu,et al.  A Multiple Visual Models Based Perceptive Analysis Framework for Multilevel Video Summarization , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[4]  Bernd Girod,et al.  What's wrong with mean-squared error? , 1993 .

[5]  James Hu,et al.  DVQ: A digital video quality metric based on human vision , 2001 .

[6]  John Jenkins,et al.  ASSESSING MULTIMEDIA QUALITY FROM THE USER'S PERSPECTIVE , 1999 .

[7]  Francis Rumsey,et al.  Development and Initial Validation of a Multichannel Audio Quality Expert System , 2005 .

[8]  Charles D. Creusere,et al.  An Objective Metric of Human Subjective Audio Quality Optimized for a Wide Range of Audio Fidelities , 2008, IEEE Transactions on Audio, Speech, and Language Processing.

[9]  Miska M. Hannuksela,et al.  Perceptual quality assessment based on visual attention analysis , 2009, ACM Multimedia.

[10]  Margaret H. Pinson,et al.  A new standardized method for objectively measuring video quality , 2004, IEEE Transactions on Broadcasting.

[11]  George Ghinea,et al.  QoS impact on user perception and understanding of multimedia video clips , 1998, MULTIMEDIA '98.

[12]  Z. L. Budrikis,et al.  Picture Quality Prediction Based on a Visual Model , 1982, IEEE Trans. Commun..

[13]  Alan Hanjalic,et al.  Affective video content representation and modeling , 2005, IEEE Transactions on Multimedia.

[14]  Stefan Winkler,et al.  Perceived Audiovisual Quality of Low-Bitrate Multimedia Content , 2006, IEEE Transactions on Multimedia.

[15]  Stephen D. Voran,et al.  Objective estimation of perceived speech quality. I. Development of the measuring normalizing block technique , 1999, IEEE Trans. Speech Audio Process..

[16]  Andrew Perkis,et al.  Spatial and temporal pooling of image quality metrics for perceptual video quality assessment on packet loss streams , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[17]  Susu Yao,et al.  Just noticeable distortion model and its applications in video coding , 2005, Signal Process. Image Commun..

[18]  Birger Kollmeier,et al.  Objective Modeling of Speech Quality with a Psychoacoustically Validated Auditory Model , 2000 .

[19]  Stephen Wolf,et al.  Video Quality Measurement Techniques , 2002 .

[20]  Scott D. Lipscomb.  Cross‐modal integration: Synchronization of auditory and visual components in simple and complex media , 1999 .

[21]  Yong Man Ro,et al.  Quality models for audiovisual streaming , 2006, Electronic Imaging.

[22]  Tao Liu,et al.  Saliency based objective quality assessment of decoded video affected by packet losses , 2008, 2008 15th IEEE International Conference on Image Processing.

[23]  Rita Cucchiara,et al.  Semantic Video Transcoding Using Classes of Relevance , 2003, Int. J. Image Graph..

[24]  Stefan Winkler,et al.  A no-reference perceptual blur metric , 2002, Proceedings. International Conference on Image Processing.

[25]  Toshiko Tominaga,et al.  Multimedia Quality Integration Function for Videophone Services , 2007, IEEE GLOBECOM 2007 - IEEE Global Telecommunications Conference.

[26]  Charles D. Creusere,et al.  Audio quality assessment using the mean structural similarity measure , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[27]  Ying Zhong,et al.  Influence of Task and Scene Content on Subjective Video Quality , 2004, ICIAR.

[28]  Birger Kollmeier,et al.  PEMO-Q—A New Method for Objective Audio Quality Assessment Using a Model of Auditory Perception , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[29]  渡辺馨.  Objective measurement method of audio quality in accordance with ITU-R Recommendation BS. 1387 , 2001 .

[30]  Sugato Chakravarty,et al.  Methodology for the subjective assessment of the quality of television pictures , 1995 .

[31]  C.D. Creusere,et al.  Scalable Perceptual Metric for Evaluating Audio Quality , 2005, Conference Record of the Thirty-Ninth Asilomar Conference onSignals, Systems and Computers, 2005..

[32]  Alan C. Bovik,et al.  . Efficient DCT-domain blind measurement and reduction of blocking artifacts , 2002, IEEE Trans. Circuits Syst. Video Technol..

[33]  D. S. Hands,et al.  A basic multimedia quality model , 2004, IEEE Transactions on Multimedia.

[34]  Charles D. Creusere,et al.  Evaluating low bitrate scalable audio quality using advanced version of PEAQ and energy equalization approach , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[35]  John G. Beerends,et al.  A Perceptual Audio Quality Measure Based on a Psychoacoustic Sound Representation , 1992 .

[36]  Stefan Winkler,et al.  Digital Video Quality: Vision Models and Metrics , 2005 .

[37]  C. L. M..  The Psychology of Attention , 1890, Nature.

[38]  A. Bovik,et al.  OBJECTIVE VIDEO QUALITY ASSESSMENT , 2003 .

[39]  C. Jones,et al.  Development of opinion-based audiovisual quality models for desktop video-teleconferencing , 1998, 1998 Sixth International Workshop on Quality of Service (IWQoS'98) (Cat. No.98EX136).

[40]  Thomas Sporer.  Objective Audio Signal Evaluation-Applied Psychoacoustics for Modeling the Perceived Quality of Digital Audio , 1997 .

[41]  J. Deutsch.  Perception and Communication , 1958, Nature.

[42]  Yong Man Ro,et al.  Multimedia quality evaluation across different modalities , 2005, IS&T/SPIE Electronic Imaging.

[43]  Karlheinz Brandenburg.  Evaluation of Quality for Audio Encoding at Low Bit Rates , 1987 .

[44]  Weisi Lin.  Computational Models for Just-Noticeable Difference , 2005 .

[45]  Stefan Winkler,et al.  Perceptual Video Quality Metrics — A Review , 2005 .

[46]  D. Broadbent.  Perception and communication , 1958 .

[47]  Methods for the subjective assessment of small impairments in audio systems , 2015 .

[48]  RECOMMENDATION ITU-R BS.1534-1 - Method for the subjective assessment of intermediate quality level of coding systems , 2003 .

[49]  Yong Man Ro,et al.  Graph-Based Perceptual Quality Model for Audiovisual Contents , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[50]  B. Paillard,et al.  PERCEVAL: Perceptual Evaluation of the Quality of Audio Signals , 1992 .

[51]  Yazrina Yahya,et al.  Assessing Multimedia Quality from the User , 1970 .

[52]  Stefan Winkler.  Video quality and beyond , 2007, 2007 15th European Signal Processing Conference.

[53]  Margaret H. Pinson,et al.  An objective method for combining multiple subjective data sets , 2003, Visual Communications and Image Processing.

[54]  R. Puglia,et al.  Audiovisual quality estimation for mobile streaming services , 2005, 2005 2nd International Symposium on Wireless Communication Systems.

[55]  Jayme Garcia Arnal Barbedo,et al.  A new cognitive model for objective assessment of audio quality , 2005 .

[56]  Markus Rupp,et al.  Reference-Free Video Quality Metric for Mobile Streaming Applications , 2005 .

[57]  Ralf Steinmetz,et al.  A Media Synchronization Survey: Reference Model, Specification, and Case Studies , 1996, IEEE J. Sel. Areas Commun..

[58]  Jean-Bernard Rault,et al.  A Perceptual Model Applied to Audio Bit-Rate Reduction , 1995 .

[59]  WATCH , 2004 .

[60]  Florian Wickelmaier,et al.  Perceptual Audio Evaluation - Theory, Method and Application , 2006 .

[61]  D W Massaro,et al.  Perception of asynchronous and conflicting visual and auditory speech. , 1996, The Journal of the Acoustical Society of America.

[62]  C. Koch,et al.  Computational modelling of visual attention , 2001, Nature Reviews Neuroscience.

[63]  Thomas Sporer,et al.  PEAQ - The ITU Standard for Objective Measurement of Perceived Audio Quality , 2000 .

[64]  Milind R. Naphade,et al.  Extracting semantics from audio-visual content: the final frontier in multimedia retrieval , 2002, IEEE Trans. Neural Networks.

[65]  David J. Sakrison,et al.  The effects of a visual fidelity criterion of the encoding of images , 1974, IEEE Trans. Inf. Theory.

[66]  Scott J. Daly,et al.  Visible differences predictor: an algorithm for the assessment of image fidelity , 1992, Electronic Imaging.

[67]  Michael Peter Hollier,et al.  An Experimental Investigation into Multi-Modal Synchronization Sensitivity for Perceptual Model Development , 1998 .

[68]  S. Jumisko-Pyykko,et al.  Produced Quality is Not Perceived Quality - A Qualitative Approach to Overall Audiovisual Quality , 2007, 2007 3DTV Conference.

[69]  Michael Meehan,et al.  Physiological measures of presence in stressful virtual environments , 2002, SIGGRAPH.

[70]  Weisi Lin,et al.  Modeling visual attention's modulatory aftereffects on visual sensitivity and quality evaluation , 2005, IEEE Transactions on Image Processing.

[71]  Christof Faller,et al.  MAXIMIZING AUDIOVISUAL QUALITY AT LOW BITRATES , 2005 .

[72]  Charles D. Creusere,et al.  Understanding perceptual distortion in MPEG scalable audio coding , 2005, IEEE Transactions on Speech and Audio Processing.

[73]  Zhou Wang,et al.  Video quality assessment based on structural distortion measurement , 2004, Signal Process. Image Commun..

[74]  John G. Beerends,et al.  The Influence of Video Quality on Perceived Audio Quality and Vice Versa , 1999 .

[75]  Fa-Long Luo,et al.  Mobile Multimedia Broadcasting Standards: Technology and Practice , 2008 .

[76]  J. Juola,et al.  Audiovisual synchrony and temporal order judgments: Effects of experimental method and stimulus type , 2008, Perception & psychophysics.

[77]  Chulhee Lee,et al.  Objective measurements of video quality using the wavelet transform , 2003 .

[78]  Michael Peter Hollier,et al.  Multi-modal Perception , 1999 .

[79]  Yujie Gao.  Audio Coding Standard Overview: MPEG4-AAC, HE-AAC, and HE-AAC V2 , 2009 .

[80]  Satu Jumisko-Pyykkö,et al.  Watch, Press, and Catch - Impact of Divided Attention on Requirements of Audiovisual Quality , 2007, HCI.

[81]  Michael R. Frater,et al.  Impact of audio on subjective assessment of video quality in videoconferencing applications , 2001, IEEE Trans. Circuits Syst. Video Technol..

[82]  Ulrich Dr.-Ing. Reiter.  Bimodal Audiovisual Perception in Interactive Application Systems of Moderate Complexity , 2009 .

[83]  W. Schweigert,et al.  Research Methods and Statistics in Psychology , 2023 .

[84]  Harley R. Myler,et al.  Gabor difference analysis of digital video quality , 2004, IEEE Transactions on Broadcasting.

[85]  Miska M. Hannuksela,et al.  An objective video quality metric based on spatiotemporal distortion , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[86]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[87]  Hong Ren Wu,et al.  Digital Video Image Quality and Perceptual Coding , 2005 .

[88]  T. Mexia,et al.  Author ' s personal copy , 2009 .

[89]  N. F. Dixon,et al.  The Detection of Auditory Visual Desynchrony , 1980, Perception.

[90]  Ralf Steinmetz,et al.  Human Perception of Jitter and Media Synchronization , 1996, IEEE J. Sel. Areas Commun..

[91]  Miska M. Hannuksela,et al.  Semantic audiovisual analysis for video summarization , 2009, IEEE EUROCON 2009.