Cloud Gaming With Foveated Graphics

Cloud gaming enables playing high end games, originally designed for PC or game console setups, on low end devices, such as net-books and smartphones, by offloading graphics rendering to GPU powered cloud servers. However, transmitting the high end graphics requires a large amount of available network bandwidth, even though it is a compressed video stream. Foveated video encoding (FVE) reduces the bandwidth requirement by taking advantage of the non-uniform acuity of human visual system and by knowing where the user is looking. We have designed and implemented a system for cloud gaming with foveated graphics using a consumer grade real-time eye tracker and an open source cloud gaming platform. In this article, we describe the system and its evaluation through measurements with representative games from different genres to understand the effect of parameterization of the FVE scheme on bandwidth requirements and to understand its feasibility from the latency perspective. We also present results from a user study. The results suggest that it is possible to find a "sweet spot" for the encoding parameters so that the users hardly notice the presence of foveated encoding but at the same time the scheme yields most of the bandwidth savings achievable.

[1]  Kajal T. Claypool,et al.  Latency and player actions in online games , 2006, CACM.

[2]  Laurent Itti,et al.  Visual attention guided bit allocation in video compression , 2011, Image Vis. Comput..

[3]  Laurent Itti,et al.  Automatic foveation for video compression using a neurobiological model of visual attention , 2004, IEEE Transactions on Image Processing.

[4]  Lea Skorin-Kapov,et al.  Empirical QoE study of in-home streaming of online games , 2014, 2014 13th Annual Workshop on Network and Systems Support for Games.

[5]  Multimedia Systems Conference 2014, MMSys '14, Singapore, March 19-21, 2014 , 2014, MMSys.

[6]  Konrad Tollmar,et al.  QoE design tradeoffs for foveated content provision , 2017, 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX).

[7]  Patrick Le Callet,et al.  Do gaze disruptions indicate the perceived quality of non-uniformly coded natural scenes? , 2017, HVEI.

[8]  Mahmoud Reza Hashemi,et al.  A game attention model for efficient bit rate allocation in cloud gaming , 2014, Multimedia Systems.

[9]  B. Wandell.  Foundations of vision , 1995 .

[10]  Jim Mullin,et al.  New techniques for assessing audio and video quality in real-time interactive communications , 2002 .

[11]  Christine Guillemot,et al.  Perceptually-Friendly H.264/AVC Video Coding Based on Foveated Just-Noticeable-Distortion Model , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[12]  Cheng-Hsin Hsu,et al.  On the Quality of Service of Cloud Gaming Systems , 2014, IEEE Transactions on Multimedia.

[13]  Tiago H. Falk,et al.  Audio-Visual Multimedia Quality Assessment: A Comprehensive Survey , 2017, IEEE Access.

[14]  Gwendal Simon,et al.  The brewing storm in cloud gaming: A measurement study on cloud to end-user latency , 2012, 2012 11th Annual Workshop on Network and Systems Support for Games (NetGames).

[15]  Zhou Wang,et al.  Foveated Image and Video Coding , 2004 .

[16]  Romass Pauliks,et al.  A survey on some measurement methods for subjective video quality assessment , 2013, 2013 World Congress on Computer and Information Technology (WCCIT).

[17]  Miska M. Hannuksela,et al.  HEVC-compliant Tile-based Streaming of Panoramic Video for Virtual Reality Applications , 2016, ACM Multimedia.

[18]  Agostino Gibaldi,et al.  Evaluation of the Tobii EyeX Eye tracking controller and Matlab toolkit for research , 2016, Behavior Research Methods.

[19]  Matti Siekkinen,et al.  Foveated video streaming for cloud gaming , 2017, 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP).

[20]  Zhou Wang,et al.  Foveation scalable video coding with automatic fixation selection , 2003, IEEE Trans. Image Process..

[21]  Jorrit van den Berg,et al.  Using MPEG DASH SRD for zoomable and navigable video , 2016, MMSys.

[22]  Joohwan Kim,et al.  Latency Requirements for Foveated Rendering in Virtual Reality , 2017, ACM Trans. Appl. Percept..

[23]  P. Verghese,et al.  Accuracy of eye position for saccades and smooth pursuit , 2016, Journal of vision.

[24]  Tobias Hoßfeld,et al.  An Evaluation of QoE in Cloud Gaming Based on Subjective Tests , 2011, 2011 Fifth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing.

[25]  Daniel J. Wigdor,et al.  How Much Faster is Fast Enough?: User Perception of Latency & Latency Improvements in Direct and Indirect Touch , 2015, CHI.

[26]  Ryan Shea,et al.  Cloud gaming: architecture and performance , 2013, IEEE Network.

[27]  Wei Cai,et al.  Toward Gaming as a Service , 2014, IEEE Internet Computing.

[28]  Mahmoud Reza Hashemi,et al.  An object-based framework for cloud gaming using player's visual attention , 2015, 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[29]  Wei Cai,et al.  A Cognitive Platform for Mobile Cloud Gaming , 2013, 2013 IEEE 5th International Conference on Cloud Computing Technology and Science.

[30]  Ayub Bokani.  Empirical Evaluation of Real-Time Video Foveation , 2014, VideoNext '14.

[31]  Chun-Ying Huang,et al.  Measuring the latency of cloud gaming systems , 2011, ACM Multimedia.

[32]  Michele Rucci,et al.  Fixational eye movements and perception , 2016, Vision Research.

[33]  Gregory J. Zelinsky,et al.  Design and evaluation of a foveated video streaming service for commodity client devices , 2016, MMSys.