Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming

The emerging cloud gaming service has been growing rapidly, but not yet able to reach mobile customers due to many limitations, such as bandwidth and latency. We introduce a 3D image warping assisted real-time video coding method that can potentially meet all the requirements of mobile cloud gaming. The proposed video encoder selects a set of key frames in the video sequence, uses the 3D image warping algorithm to interpolate other non-key frames, and encodes the key frames and the residues frames with an H.264/AVC encoder. Our approach is novel in taking advantage of the run-time graphics rendering contexts (rendering viewpoint, pixel depth, camera motion, etc.) from the 3D game engine to enhance the performance of video encoding for the cloud gaming service. The experiments indicate that our proposed video encoder has the potential to beat the state-of-art x264 encoder in the scenario of real-time cloud gaming. For example, by implementing the proposed method in a 3D tank battle game, we experimentally show that more than 2 dB quality improvement is possible.

[1]  Luc Van Gool,et al.  Advanced three-dimensional television system technologies , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[2]  D. Gourlay,et al.  Remote walkthrough over mobile networks using 3-D image warping and streaming , 2004 .

[3]  Jr. Leonard McMillan,et al.  An Image-Based Approach to Three-Dimensional Computer Graphics , 1997 .

[4]  Klara Nahrstedt,et al.  Real-time parallel remote rendering for mobile devices using graphics processing units , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[5]  Mark Claypool,et al.  The effects of loss and latency on user performance in unreal tournament 2003® , 2004, NetGames '04.

[6]  William R. Mark,et al.  Post-Rendering 3D Image Warping: Visibility, Reconstruction, and Performance for Depth-Image Warping , 1999 .

[7]  Yao Wang,et al.  Video Processing and Communications , 2001 .

[8]  Ulug Bayazit.  Macroblock data classification and nonlinear bit count estimation for low delay H.263 rate control , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[9]  Lurng-Kuo Liu,et al.  Low-delay MPEG-2 video coding , 1998, Electronic Imaging.

[10]  Marc Levoy.  Polygon-assisted JPEG and MPEG compression of synthetic images , 1995, SIGGRAPH.

[11]  Byung Cheol Song,et al.  An intra-frame rate control algorithm for ultra low delay H.264/AVC coding , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[12]  Stephan Wong,et al.  An efficient realization of forward integer transform in H.264/AVC intra-frame encoder , 2010, 2010 International Conference on Embedded Computer Systems: Architectures, Modeling and Simulation.

[13]  Albrecht Rothermel,et al.  Architecture Analysis for Low-Delay Video Coding , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[14]  Oriana Riva,et al.  Challenges and Lessons in Developing Middleware on Smart Phones , 2008, Computer.

[15]  Simone Milani,et al.  A cognitive approach for effective coding and transmission of 3D video , 2011, ACM Trans. Multim. Comput. Commun. Appl..

[16]  Klara Nahrstedt,et al.  A high-quality low-delay remote rendering system for 3D video , 2010, ACM Multimedia.

[17]  Zhengguo Li,et al.  A Novel Rate Control Scheme for Low Delay Video Communication of H.264/AVC Standard , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[18]  Ajay Luthra,et al.  Overview of the H.264/AVC video coding standard , 2003, IEEE Trans. Circuits Syst. Video Technol..

[19]  Reinhard Klein,et al.  Augmented Compression for Server-Side Rendering , 2008, VMV.