Modern trends on quality of experience assessment and future work

Over the past 20 years, research on quality of experience (QoE) has been actively expanded even to cover aesthetic, emotional and psychological experiences. QoE has been an important research topic in determining the perceptual factors that are essential to users in keeping with the emergence of new display technologies. In this paper, we provide in-depth reviews of recent assessment studies in this field. Compared to previous reviews, our research examines the human factors observed over various recent displays and their associated assessment methods. In this study, we first provide a comprehensive QoE analysis on 2D display including image/video quality assessment (I/VQA), visual preference, and human visual system-related studies. Second, we analyze stereoscopic 3D (S3D) QoE research on the topics of I/VQA and visual discomfort from the human perception point of view on S3D display. Third, we investigate QoE in a head-mounted display-based virtual reality (VR) environment, and deal with VR sickness and 360 I/VQA with their individual approach. All of our reviews are analyzed through comparison of benchmark models. Furthermore, we layout QoE works on future display and modern deep-learning applications.

[1]  Ching-Jen Chen,et al.  Numerical simulation of flow in a screw-type blood pump , 2005, J. Vis..

[2]  Yong Man Ro,et al.  Visual comfort assessment metric based on salient object motion information in stereoscopic video , 2012, J. Electronic Imaging.

[3]  Kwanghoon Sohn,et al.  Visual fatigue modeling and analysis for stereoscopic video , 2012 .

[4]  Alan C. Bovik,et al.  3D Visual Discomfort Predictor: Analysis of Disparity and Neural Activity Statistics , 2015, IEEE Transactions on Image Processing.

[5]  Alan Conrad Bovik,et al.  Deep Visual Discomfort Predictor for Stereoscopic 3D Images , 2018, IEEE Transactions on Image Processing.

[6]  Alan C. Bovik,et al.  Video Quality Pooling Adaptive to Perceptual Distortion Severity , 2013, IEEE Transactions on Image Processing.

[7]  Alan C. Bovik,et al.  Motion Tuned Spatio-Temporal Quality Assessment of Natural Videos , 2010, IEEE Transactions on Image Processing.

[8]  Yong Man Ro,et al.  Predicting Visual Discomfort Using Object Size and Disparity Information in Stereoscopic Images , 2013, IEEE Transactions on Broadcasting.

[9]  Eric C. Larson,et al.  Most apparent distortion: full-reference image quality assessment and the role of strategy , 2010, J. Electronic Imaging.

[10]  Lai-Man Po,et al.  No-Reference Video Quality Assessment With 3D Shearlet Transform and Convolutional Neural Networks , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[11]  Feng Qi,et al.  Stereoscopic video quality assessment based on visual attention and just-noticeable difference models , 2015, Signal, Image and Video Processing.

[12]  Yong Man Ro,et al.  Deep Virtual Reality Image Quality Assessment With Human Perception Guider for Omnidirectional Image , 2020, IEEE Transactions on Circuits and Systems for Video Technology.

[13]  Alan C. Bovik,et al.  3D Visual Discomfort Prediction: Vergence, Foveation, and the Physiological Optics of Accommodation , 2014, IEEE Journal of Selected Topics in Signal Processing.

[14]  Zhibo Chen,et al.  Blind Stereoscopic Video Quality Assessment: From Depth Perception to Overall Experience , 2018, IEEE Transactions on Image Processing.

[15]  Sanghoon Lee,et al.  Blind Deep S3D Image Quality Evaluation via Local to Global Feature Aggregation , 2017, IEEE Transactions on Image Processing.

[16]  Alan C. Bovik,et al.  No-Reference Sharpness Assessment of Camera-Shaken Images by Analysis of Spectral Structure , 2014, IEEE Transactions on Image Processing.

[17]  Cagri Ozcinar,et al.  Visual Attention-Aware Omnidirectional Video Streaming Using Optimal Tiles for Virtual Reality , 2019, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[18]  Andrey S. Krylov,et al.  No-Reference Stereoscopic Image Quality Assessment Using Convolutional Neural Network for Adaptive Feature Extraction , 2018, IEEE Access.

[19]  R. Blake,et al.  What is Suppressed during Binocular Rivalry? , 1980, Perception.

[20]  Alan C. Bovik,et al.  3D Visual Activity Assessment Based on Natural Scene Statistics , 2014, IEEE Transactions on Image Processing.

[21]  Gordon Wetzstein,et al.  Towards a Machine-Learning Approach for Sickness Prediction in 360° Stereoscopic Videos , 2018, IEEE Transactions on Visualization and Computer Graphics.

[22]  Wijnand A. IJsselsteijn,et al.  Visual discomfort of 3D TV: Assessment methods and modeling , 2011, Displays.

[23]  Lei Zhang,et al.  Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index , 2013, IEEE Transactions on Image Processing.

[24]  Weisi Lin,et al.  The Analysis of Image Contrast: From Quality Assessment to Automatic Enhancement , 2016, IEEE Transactions on Cybernetics.

[25]  Marios S. Pattichis,et al.  Foveated video quality assessment , 2002, IEEE Trans. Multim..

[26]  Jiachen Yang,et al.  Stereoscopic video quality assessment based on 3D convolutional neural networks , 2018, Neurocomputing.

[27]  Kai Zeng,et al.  Quality Prediction of Asymmetrically Distorted Stereoscopic 3D Images , 2015, IEEE Transactions on Image Processing.

[28]  Zhou Wang,et al.  A Patch-Structure Representation Method for Quality Assessment of Contrast Changed Images , 2015, IEEE Signal Processing Letters.

[29]  C.-C. Jay Kuo,et al.  A Haar Wavelet Approach to Compressed Image Quality Measurement , 2000, J. Vis. Commun. Image Represent..

[30]  Ja-Ling Wu,et al.  Quality Assessment of Stereoscopic 3D Image Compression by Binocular Integration Behaviors , 2014, IEEE Transactions on Image Processing.

[31]  Yong Liu,et al.  Blind Image Quality Assessment Based on High Order Statistics Aggregation , 2016, IEEE Transactions on Image Processing.

[32]  Alan C. Bovik,et al.  Massive Online Crowdsourced Study of Subjective and Objective Picture Quality , 2015, IEEE Transactions on Image Processing.

[33]  Alan C. Bovik,et al.  Multimodal Interactive Continuous Scoring of Subjective 3D Video Quality of Experience , 2014, IEEE Transactions on Multimedia.

[34]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[35]  Damon M. Chandler,et al.  ViS3: an algorithm for video quality assessment via analysis of spatial and spatiotemporal slices , 2014, J. Electronic Imaging.

[36]  Kay Connelly,et al.  Toward total quality of experience: A QoE model in a communication ecosystem , 2012, IEEE Communications Magazine.

[37]  C.-C. Jay Kuo,et al.  Objective Video Quality Assessment Based on Perceptually Weighted Mean Squared Error , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[38]  Lina J. Karam,et al.  A No-Reference Objective Image Sharpness Metric Based on the Notion of Just Noticeable Blur (JNB) , 2009, IEEE Transactions on Image Processing.

[39]  F. Okano,et al.  Repeated vergence adaptation causes the decline of visual functions in watching stereoscopic television , 2005, Journal of Display Technology.

[40]  Ke Gu,et al.  Learning a No-Reference Quality Assessment Model of Enhanced Images With Big Data , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[41]  Alan Conrad Bovik,et al.  Enhancement of Visual Comfort and Sense of Presence on Stereoscopic 3D Images , 2017, IEEE Transactions on Image Processing.

[42]  Do-Kyoung Kwon,et al.  Full-reference quality assessment of stereopairs accounting for rivalry , 2013, Signal Process. Image Commun..

[43]  Alan C. Bovik,et al.  No-Reference Quality Assessment of Natural Stereopairs , 2013, IEEE Transactions on Image Processing.

[44]  Sanghoon Lee,et al.  Transition of Visual Attention Assessment in Stereoscopic Images With Evaluation of Subjective Visual Quality and Discomfort , 2015, IEEE Transactions on Multimedia.

[45]  James S. Wolffsohn,et al.  Target spatial frequency determines the response to conflicting defocus- and convergence-driven accommodative stimuli , 2006, Vision Research.

[46]  Yong Man Ro,et al.  Binocular Fusion Net: Deep Learning Visual Comfort Assessment for Stereoscopic 3D , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[47]  Alan C. Bovik,et al.  Saliency Prediction on Stereoscopic Videos , 2014, IEEE Transactions on Image Processing.

[48]  Mei Yu,et al.  No reference stereo video quality assessment based on motion feature in tensor decomposition domain , 2018, J. Vis. Commun. Image Represent..

[49]  Jongyoo Kim,et al.  Deep CNN-Based Blind Image Quality Predictor , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[50]  Sanghoon Lee,et al.  Visual Preference Assessment on Ultra-High-Definition Images , 2016, IEEE Transactions on Broadcasting.

[51]  Steven W. Zucker,et al.  Local Scale Control for Edge Detection and Blur Estimation , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  Ahmet M. Kondoz,et al.  A new reduced reference metric for color plus depth 3D video , 2014, J. Vis. Commun. Image Represent..

[53]  Rajiv Soundararajan,et al.  Study of Subjective and Objective Quality Assessment of Video , 2010, IEEE Transactions on Image Processing.

[54]  Sumio Yano,et al.  Visual fatigue caused by stereoscopic images and the search for the requirement to prevent them: A review , 2012, Displays.

[55]  Patrick Le Callet,et al.  Quality Assessment of Stereoscopic Images , 2008, EURASIP J. Image Video Process..

[56]  Sanghoon Lee,et al.  Blind Sharpness Prediction for Ultrahigh-Definition Video Based on Human Visual Resolution , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[57]  Lei Zhang,et al.  A Feature-Enriched Completely Blind Image Quality Evaluator , 2015, IEEE Transactions on Image Processing.

[58]  Lin Ma,et al.  Learning structure of stereoscopic image for no-reference quality assessment with convolutional neural network , 2016, Pattern Recognit..

[59]  Sanghoon Lee,et al.  Deep Visual Saliency on Stereoscopic Images , 2019, IEEE Transactions on Image Processing.

[60]  Lina J. Karam,et al.  A No-Reference Image Blur Metric Based on the Cumulative Probability of Blur Detection (CPBD) , 2011, IEEE Transactions on Image Processing.

[61]  Sanghoon Lee,et al.  Fully Deep Blind Image Quality Predictor , 2017, IEEE Journal of Selected Topics in Signal Processing.

[62]  Alan C. Bovik,et al.  Transfer Function Model of Physiological Mechanisms Underlying Temporal Visual Discomfort Experienced When Viewing Stereoscopic 3D Images , 2015, IEEE Transactions on Image Processing.

[63]  Alan C. Bovik,et al.  Stereoscopic 3D Visual Discomfort Prediction: A Dynamic Accommodation and Vergence Interaction Model , 2016, IEEE Transactions on Image Processing.

[64]  Yuukou Horita,et al.  Objective No-Reference Stereoscopic Image Quality Prediction Based on 2D Image Features and Relative Disparity , 2012, Adv. Multim..

[65]  Alan Conrad Bovik,et al.  Quality Assessment of Perceptual Crosstalk on Two-View Auto-Stereoscopic Displays , 2017, IEEE Transactions on Image Processing.

[66]  Mark Mon-Williams,et al.  Natural problems for stereoscopic depth perception in virtual environments , 1995, Vision Research.

[67]  Masaki Emoto,et al.  Research on Human Factors in Ultrahigh-Definition Television (UHDTV) to Determine its Specifications , 2008 .

[68]  Nikolay N. Ponomarenko,et al.  Image database TID2013: Peculiarities, results and perspectives , 2015, Signal Process. Image Commun..

[69]  Kwanghyun Lee,et al.  3D Perception Based Quality Pooling: Stereopsis, Binocular Rivalry, and Binocular Suppression , 2015, IEEE Journal of Selected Topics in Signal Processing.

[70]  David M. Hoffman,et al.  Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. , 2008, Journal of vision.

[71]  Sumio Yano,et al.  A study of visual fatigue and visual comfort for 3D HDTV/HDTV images , 2002 .

[72]  Sanghoon Lee,et al.  Visual Presence: Viewing Geometry Visual Information of UHD S3D Entertainment , 2016, IEEE Transactions on Image Processing.

[73]  Alan C. Bovik,et al.  A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms , 2006, IEEE Transactions on Image Processing.

[74]  Marios S. Pattichis,et al.  Foveated video compression with optimal rate control , 2001, IEEE Trans. Image Process..

[75]  Nikolay N. Ponomarenko,et al.  TID2008 – A database for evaluation of full-reference visual quality assessment metrics , 2004 .

[76]  Zulin Wang,et al.  Assessing Visual Quality of Omnidirectional Videos , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[77]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[78]  Zhou Wang,et al.  No-Reference Quality Assessment of Contrast-Distorted Images Based on Natural Scene Statistics , 2015, IEEE Signal Processing Letters.

[79]  Alan C. Bovik,et al.  Subjective evaluation of stereoscopic image quality , 2013, Signal Process. Image Commun..

[80]  Kwanghoon Sohn,et al.  Visual Fatigue Prediction for Stereoscopic Image , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[81]  Franco Oberti,et al.  A new sharpness metric based on local kurtosis, edge and energy information , 2004, Signal Process. Image Commun..

[82]  Martin Reisslein,et al.  Objective Video Quality Assessment Methods: A Classification, Review, and Performance Comparison , 2011, IEEE Transactions on Broadcasting.

[83]  Yong Man Ro,et al.  Predicting Visual Discomfort of Stereoscopic Images Using Human Attention Model , 2013, IEEE Transactions on Circuits and Systems for Video Technology.