A Study of High Frame Rate Video Formats

High frame rates are acknowledged to increase the perceived quality of certain video content. However, the lack of high frame rate test content has previously restricted the scope of research in this area—especially in the context of immersive video formats. This problem has been addressed through the publication of a high frame rate video database BVI-HFR, which was captured natively at 120 fps. BVI-HFR spans a variety of scenes, motions, and colors, and is shown to be representative of BBC broadcast content. In this paper, temporal down-sampling is utilized to enable both subjective and objective comparisons across a range frame rates. A large-scale subjective experiment has demonstrated that high frame rates lead to increases in perceived quality, and that a degree of content dependence exists—notably related to camera motion. Various image and video quality metrics have been benchmarked on these subjective evaluations, and analysis shows that those which explicitly account for temporal distortions (e.g., FRQM) provide improved correlation with subjective opinions compared to generic quality metrics such as PSNR.

[1]  Fan Zhang,et al.  Support for reduced presentation durations in subjective video quality assessment , 2016, Signal Process. Image Commun..

[2]  Shao-Yi Chien,et al.  Algorithm and Architecture Design of Multirate Frame Rate Up-conversion for Ultra-HD LCD Systems , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[3]  Zhou Wang,et al.  Multiscale structural similarity for image quality assessment , 2003, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.

[4]  Denis G. Pelli,et al.  ECVP '07 Abstracts , 2007, Perception.

[5]  Fan Zhang,et al.  A study of subjective video quality at various frame rates , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[6]  Katy C. Noland,et al.  High Frame Rates and the Visibility of Motion Artifacts , 2017 .

[7]  Yao Wang,et al.  Q-STAR: A Perceptual Video Quality Model Considering Impact of Spatial, Temporal, and Amplitude Resolutions , 2012, IEEE Transactions on Image Processing.

[8]  Mike Armstrong,et al.  High Frame-Rate Television , 2009 .

[9]  Antonio Torralba,et al.  Statistics of natural image categories , 2003, Network.

[10]  Li Zhang,et al.  Multiview and 3D Video Compression Using Neighboring Block Based Disparity Vectors , 2016, IEEE Transactions on Multimedia.

[11]  Andrew B. Watson,et al.  High Frame Rates and Human Vision: A View through the Window of Visibility , 2013 .

[12]  Damon M. Chandler,et al.  A spatiotemporal most-apparent-distortion model for video quality assessment , 2011, 2011 18th IEEE International Conference on Image Processing.

[13]  Shiqi Wang,et al.  Perceptual quality assessment of high frame rate video , 2015, 2015 IEEE 17th International Workshop on Multimedia Signal Processing (MMSP).

[14]  Peter G. J. Barten,et al.  Contrast sensitivity of the human eye and its e ects on image quality , 1999 .

[15]  Robert S. Allison,et al.  Evidence that Viewers Prefer Higher Frame-Rate Film , 2015, ACM Trans. Appl. Percept..

[16]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[17]  Rajiv Soundararajan,et al.  Study of Subjective and Objective Quality Assessment of Video , 2010, IEEE Transactions on Image Processing.

[18]  Panos Nasiopoulos,et al.  Human Visual System-Based Saliency Detection for High Dynamic Range Content , 2016, IEEE Transactions on Multimedia.

[19]  Andrew B. Watson,et al.  Window of visibility: a psychophysical theory of fidelity in time-sampled visual motion displays , 1986 .

[20]  Zhan Ma,et al.  Modeling of Rate and Perceptual Quality of Compressed Video as Functions of Frame Rate and Quantization Stepsize and Its Applications , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[21]  Alexandre G. Ciancio,et al.  Quality of Experience in a Stereoscopic Multiview Environment , 2018, IEEE Transactions on Multimedia.

[22]  Yong Liu,et al.  Perceptual Quality Maximization for Video Calls With Packet Losses by Optimizing FEC, Frame Rate, and Quantization , 2018, IEEE Transactions on Multimedia.

[23]  Zhou Wang,et al.  Perceptual aliasing factors and the impact of frame rate on video quality , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[24]  David R. Bull,et al.  The visibility of motion artifacts and their effect on motion quality , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[25]  Katy C. Noland,et al.  The Application of Sampling Theory to Television Frame Rate Requirements , 2014 .

[26]  L. J. M. Rothkrantz,et al.  DETECTING STRESS USING EYE BLINKS AND BRAIN ACTIVITY FROM EEG SIGNALS , 2009 .

[27]  Margaret H. Pinson,et al.  A new standardized method for objectively measuring video quality , 2004, IEEE Transactions on Broadcasting.

[28]  Masaki Emoto,et al.  High-Frame-Rate Motion Picture Quality and Its Independence of Viewing Distance , 2014, Journal of Display Technology.

[29]  Keith Bugeja,et al.  Frame Rate vs Resolution: A Subjective Evaluation of Spatiotemporal Perceived Quality Under Varying Computational Budgets , 2018, Comput. Graph. Forum.

[30]  Peter J. Bex,et al.  MULTIPLE IMAGES APPEAR WHEN MOTION ENERGY DETECTION FAILS , 1995 .

[31]  Michael Vollmer,et al.  High speed and slow motion: the technology of modern high speed cameras , 2011 .

[32]  Chau Yuen,et al.  Enabling Adaptive High-Frame-Rate Video Streaming in Mobile Cloud Gaming Applications , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[33]  D. Chandler,et al.  Supplement to “ VSNR : A Visual Signal-to-Noise Ratio for Natural Images Based on Near-Threshold and Suprathreshold Vision ” , 2007 .

[34]  Ming Wang,et al.  TCP-Oriented Raptor Coding for High-Frame-Rate Video Transmission Over Wireless Networks , 2016, IEEE Journal on Selected Areas in Communications.

[35]  Hadi Hadizadeh,et al.  Full-Reference Objective Quality Assessment of Tone-Mapped Images , 2017, IEEE Transactions on Multimedia.

[36]  Douglas Lanman,et al.  Put on your 3D glasses now: the past, present, and future of virtual and augmented reality , 2014, SIGGRAPH '14.

[37]  Kai Kunze,et al.  In the Eye of the Beholder: The Impact of Frame Rate on Human Eye Blink , 2016, CHI Extended Abstracts.

[38]  Rod Selfridge,et al.  Visibility of Motion Blur and Strobing Artefacts in Video at 100 Frames per Second , 2016, CVMP 2016.

[39]  Martin S Banks,et al.  Temporal presentation protocols in stereoscopic displays: Flicker visibility, perceived motion, and perceived depth , 2011, Journal of the Society for Information Display.

[40]  F. Bossen,et al.  Common test conditions and software reference configurations , 2010 .

[41]  Masaki Emoto,et al.  P‐30: Temporal Sampling Parameters and Motion Portrayal of Television , 2009 .

[42]  Hans-Peter Seidel,et al.  Emulating displays with continuously varying frame rates , 2016, ACM Trans. Graph..

[43]  Ke Wang,et al.  On the Optimal Presentation Duration for Subjective Video Quality Assessment , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[44]  Shahram Shirani,et al.  Frame Rate Upconversion Using Optical Flow and Patch-Based Reconstruction , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[45]  Hyun Wook Park,et al.  A Region-Based Motion-Compensated Frame Interpolation Method Using a Variance-Distortion Curve , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[46]  Sugato Chakravarty,et al.  Methodology for the subjective assessment of the quality of television pictures , 1995 .

[47]  Zhan Ma,et al.  Perceptual Quality Assessment of Video Considering Both Frame Rate and Quantization Artifacts , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[48]  Zhan Ma,et al.  Modeling the impact of frame rate on perceptual quality of video , 2008, 2008 15th IEEE International Conference on Image Processing.

[49]  Fan Zhang,et al.  What's on TV: A large scale quantitative characterisation of modern broadcast video content , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[50]  Qin Huang,et al.  Perceptual Quality Driven Frame-Rate Selection (PQD-FRS) for High-Frame-Rate Video , 2016, IEEE Transactions on Broadcasting.

[51]  Robert S. Allison,et al.  Paper: Expert Viewers' Preferences for Higher Frame Rate 3D Film , 2016 .

[52]  Gary J. Sullivan,et al.  Overview of the High Efficiency Video Coding (HEVC) Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[53]  Fan Zhang,et al.  Investigating the impact of high frame rates on video compression , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[54]  Fan Zhang,et al.  A frame rate dependent video quality metric based on temporal wavelet decomposition and spatiotemporal pooling , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[55]  Yoshihiko Kuroki,et al.  Effects of motion image stimuli with normal and high frame rates on EEG power spectra: comparison with continuous motion image stimuli , 2014 .

[56]  Stefan Winkler,et al.  The Evolution of Video Quality Measurement: From PSNR to Hybrid Metrics , 2008, IEEE Transactions on Broadcasting.

[57]  Judith Redi,et al.  Perceptual Annoyance Models for Videos With Combinations of Spatial and Temporal Artifacts , 2016, IEEE Transactions on Multimedia.

[58]  Jean Lorenceau,et al.  Psychophysical Assessment of Perceptual Performance With Varying Display Frame Rates , 2016, Journal of Display Technology.

[59]  Stefan Winkler,et al.  Analysis of Public Image and Video Databases for Quality Assessment , 2012, IEEE Journal of Selected Topics in Signal Processing.

[60]  Alan C. Bovik,et al.  Image information and visual quality , 2006, IEEE Trans. Image Process..

[61]  Munchurl Kim,et al.  Efficient In-Loop Filtering Across Tile Boundaries for Multi-Core HEVC Hardware Decoders With 4 K/8 K-UHD Video Applications , 2015, IEEE Transactions on Multimedia.

[62]  Mick Wilson,et al.  Toward QoE-Assured 4K Video-on-Demand Delivery Through Mobile Edge Virtualization With Adaptive Prefetching , 2017, IEEE Transactions on Multimedia.

[63]  Chau Yuen,et al.  Modeling and Optimization of High Frame Rate Video Transmission Over Wireless Networks , 2016, IEEE Transactions on Wireless Communications.

[64]  Scott Daly,et al.  Engineering observations from spatiovelocity and spatiotemporal visual models , 1998, Electronic Imaging.

[65]  Scott Daly,et al.  A Psychophysical Study Exploring Judder Using Fundamental Signals and Complex Imagery , 2014 .

[66]  Yoshihiko Kuroki,et al.  A psychophysical study of improvements in motion‐image quality by using high frame rates , 2007 .