A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Many objective video quality assessment (VQA) algorithms include a key step of temporal pooling of frame-level quality scores. However, less attention has been paid to studying the relative efficiencies of different pooling methods on noreference (blind) VQA. Here we conduct a large-scale comparative evaluation to assess the capabilities and limitations of multiple temporal pooling strategies on blind VQA of usergenerated videos. The study yields insights and general guidance regarding the application and selection of temporal pooling models. In addition, we also propose an ensemble pooling model built on top of high-performing temporal pooling models. Our experimental results demonstrate the relative efficacies of the evaluated temporal pooling models, using several popular VQA algorithms evaluated on two recent largescale natural video quality databases. Conclusively, we also provide an empirical recipe for applying temporal pooling of frame-based quality predictions.

[1]  Rajiv Soundararajan,et al.  Study of Subjective and Objective Quality Assessment of Video , 2010, IEEE Transactions on Image Processing.

[2]  Alan C. Bovik,et al.  Learning a Continuous-Time Streaming Video QoE Model , 2018, IEEE Transactions on Image Processing.

[3]  Alan C. Bovik,et al.  Temporal hysteresis model of time varying subjective video quality , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4]  Snjezana Rimac-Drlje,et al.  Influence of temporal pooling method on the objective video quality evaluation , 2009, 2009 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting.

[5]  Alan C. Bovik,et al.  No-Reference Image Quality Assessment in the Spatial Domain , 2012, IEEE Transactions on Image Processing.

[6]  Alan Conrad Bovik,et al.  Study of Temporal Effects on Subjective Video Quality of Experience , 2017, IEEE Transactions on Image Processing.

[7]  Alan C. Bovik,et al.  Spatiotemporal Feature Integration and Model Fusion for Full Reference Video Quality Assessment , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[8]  Alan C. Bovik,et al.  Video Quality Pooling Adaptive to Perceptual Distortion Severity , 2013, IEEE Transactions on Image Processing.

[9]  Alan C. Bovik,et al.  Motion Tuned Spatio-Temporal Quality Assessment of Natural Videos , 2010, IEEE Transactions on Image Processing.

[10]  Zhou Wang,et al.  Information Content Weighting for Perceptual Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[11]  Alan C. Bovik,et al.  BBAND INDEX: A NO-REFERENCE BANDING ARTIFACT PREDICTOR , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[12]  Martin Slanina,et al.  “To pool or not to pool”: A comparison of temporal pooling methods for HTTP adaptive video streaming , 2013, 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX).

[13]  Dietmar Saupe,et al.  The Konstanz natural video database (KoNViD-1k) , 2017, 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX).

[14]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[15]  Ming Jiang,et al.  Quality Assessment of In-the-Wild Videos , 2019, ACM Multimedia.

[16]  Alan C. Bovik,et al.  Making a “Completely Blind” Image Quality Analyzer , 2013, IEEE Signal Processing Letters.

[17]  Lei Zhang,et al.  Deep Convolutional Neural Models for Picture-Quality Prediction: Challenges and Solutions to Data-Driven Image Quality Assessment , 2017, IEEE Signal Processing Magazine.

[18]  Alan C. Bovik,et al.  Visual Importance Pooling for Image Quality Assessment , 2009, IEEE Journal of Selected Topics in Signal Processing.

[19]  Murdock,et al.  The serial position effect of free recall , 1962 .

[20]  Dubravko Culibrk,et al.  Evaluating the Role of Content in Subjective Video Quality Assessment , 2014, TheScientificWorldJournal.

[21]  Patrick Le Callet,et al.  Considering Temporal Variations of Spatial Visual Distortions in Video Quality Assessment , 2009, IEEE Journal of Selected Topics in Signal Processing.

[22]  Alan C. Bovik,et al.  Video quality assessment accounting for temporal visual masking of local flicker , 2018, Signal Process. Image Commun..

[23]  Alan C. Bovik,et al.  Theory of order statistic filters and their relationship to linear FIR filters , 1989, IEEE Trans. Acoust. Speech Signal Process..

[24]  Soo-Chang Pei,et al.  Image Quality Assessment Using Human Visual DOG Model Fused With Random Forest , 2015, IEEE Transactions on Image Processing.

[25]  Domonkos Varga,et al.  No-Reference Video Quality Assessment Based on the Temporal Pooling of Deep Features , 2019, Neural Processing Letters.

[26]  Eiji Kamioka,et al.  Modeling of Cumulative QoE in On-Demand Video Services: Role of Memory Effect and Degree of Interest , 2019, Future Internet.

[27]  Christophe Charrier,et al.  Blind Prediction of Natural Video Quality , 2014, IEEE Transactions on Image Processing.

[28]  Alan C. Bovik,et al.  UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content , 2020, IEEE Transactions on Image Processing.

[29]  Gustavo de Veciana,et al.  Modeling the Time—Varying Subjective Quality of HTTP Video Streams With Rate Adaptations , 2013, IEEE Transactions on Image Processing.

[30]  Ulrich Engelke,et al.  Visual Attention in Quality Assessment , 2011, IEEE Signal Processing Magazine.

[31]  David S. Doermann,et al.  No-reference video quality assessment via feature learning , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[32]  Alan C. Bovik,et al.  No-Reference Quality Assessment of Tone-Mapped HDR Pictures , 2017, IEEE Transactions on Image Processing.

[33]  David S. Doermann,et al.  Unsupervised feature learning framework for no-reference image quality assessment , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Danny De Vleeschauwer,et al.  Model for estimating QoE of video delivered using HTTP adaptive streaming , 2013, 2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013).

[35]  Alan Conrad Bovik,et al.  Large-Scale Study of Perceptual Video Quality , 2018, IEEE Transactions on Image Processing.

[36]  Alan C. Bovik,et al.  Perceptual quality prediction on authentically distorted images using a bag of features approach , 2016, Journal of vision.

[37]  Alan C. Bovik,et al.  A Completely Blind Video Integrity Oracle , 2016, IEEE Transactions on Image Processing.

[38]  Anil C. Kokaram,et al.  A Perceptual Quality Metric for Videos Distorted by Spatially Correlated Noise , 2016, ACM Multimedia.

[39]  Praful Gupta,et al.  SpEED-QA: Spatial Efficient Entropic Differencing for Image and Video Quality , 2017, IEEE Signal Processing Letters.

[40]  Ghassan Al-Regib,et al.  Perceptual video quality assessment: Spatiotemporal pooling strategies for different distortions and visual maps , 2016, 2016 IEEE 18th International Workshop on Multimedia Signal Processing (MMSP).

[41]  Damon M. Chandler,et al.  ViS3: an algorithm for video quality assessment via analysis of spatial and spatiotemporal slices , 2014, J. Electronic Imaging.

[42]  Lei Zhang,et al.  Blind Image Quality Assessment Using Joint Statistics of Gradient Magnitude and Laplacian Features , 2014, IEEE Transactions on Image Processing.