A Survey of Interactive Remote Rendering Systems

Remote rendering means rendering 3D graphics on a computing device and displaying the results on another computing device connected through a network. The concept was originally developed for sharing computing resources remotely. It has been receiving increasing attention from researchers in both academia and industry in recent years due to the proliferation of cloud computing and mobile devices. In this article, we survey the interactive remote rendering systems proposed in the literature, analyze how to improve the state of the art, and summarize the related technologies. The readers of this article will understand the history of remote rendering systems and obtain some inspirations of the future research directions in this area.

[1]  Pietro Zanuttigh,et al.  Compression of depth information for 3D rendering , 2009, 2009 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[2]  Hans-Peter Seidel,et al.  Render2MPEG: A Perception‐based Framework Towards Integrating Rendering and Video Compression , 2008, Comput. Graph. Forum.

[3]  Ajay Luthra,et al.  Overview of the H.264/AVC video coding standard , 2003, IEEE Trans. Circuits Syst. Video Technol..

[4]  Peter H. N. de With,et al.  Multiview Depth-Image Compression Using an Extended H.264 Encoder , 2007, ACIVS.

[5]  Jason Nieh,et al.  THINC: a virtual display architecture for thin-client computing , 2005, SOSP '05.

[6]  Feng Qian,et al.  A close examination of performance and power characteristics of 4G LTE networks , 2012, MobiSys '12.

[7]  Ulrich Neumann,et al.  Web‐Based Remote Renderingwith IBRAC (Image‐Based Rendering Acceleration and Compression) , 2000, Comput. Graph. Forum.

[8]  Rajkumar Buyya,et al.  Cloud-Based Augmentation for Mobile Devices: Motivation, Taxonomies, and Open Challenges , 2013, IEEE Communications Surveys & Tutorials.

[9]  Wouter Pasman,et al.  Comparing Simplification and Image-Based Techniques for 3D Client-Server Rendering Systems , 2003, IEEE Trans. Vis. Comput. Graph..

[10]  Yo-Sung Ho,et al.  H.264-Based Depth Map Sequence Coding Using Motion Information of Corresponding Texture Video , 2006, PSIVT.

[11]  Wolfgang Straßer,et al.  Real time compression of triangle mesh connectivity , 1998, SIGGRAPH.

[12]  Andy Hopper,et al.  Virtual Network Computing , 1998, IEEE Internet Comput..

[13]  Filip De Turck,et al.  A hybrid thin-client protocol for multimedia streaming and interactive gaming applications , 2006, NOSSDAV '06.

[14]  George Drettakis,et al.  Flexible point-based rendering on mobile devices , 2004, IEEE Computer Graphics and Applications.

[15]  Cheng-Hsin Hsu,et al.  Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming , 2011, ACM Multimedia.

[16]  Hans-Christian Hege,et al.  Interactive exploration of large remote micro-CT scans , 2004, IEEE Visualization 2004.

[17]  Dieter Schmalstieg,et al.  A network architecture for remote rendering , 1998, Proceedings. 2nd International Workshop on Distributed Interactive Simulation and Real-Time Applications (Cat. No.98EX191).

[18]  Thomas C. Hudson,et al.  Multiple Image Warping for Remote Display of Rendered Images , 1999 .

[19]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[20]  Hans-Christian Hege,et al.  Distributed Visualization with OpenGL Vizserver: Practical Experiences , 2000 .

[21]  Klara Nahrstedt,et al.  A real-time remote rendering system for interactive mobile graphics , 2012, TOMCCAP.

[22]  Dani Lischinski,et al.  Streaming of Complex 3D Scenes for Remote Walkthroughs , 2001, Comput. Graph. Forum.

[23]  Daniel Cohen-Or,et al.  Deep compression for streaming texture intensive animations , 1999, SIGGRAPH.

[24]  Yücel Altunbasak,et al.  3TP: an application-Layer protocol for streaming 3-D models , 2005, IEEE Transactions on Multimedia.

[25]  Daniel Cohen-Or,et al.  Selective Pixel Transmission for Navigating in Remote Virtual Environments , 1997, Comput. Graph. Forum.

[26]  K. Bouatouch,et al.  Remote Rendering of Massively Textured 3 D Scenes Through Progressive Texture Maps , 2004 .

[27]  Thomas Ertl,et al.  Widening the remote visualization bottleneck , 2003, 3rd International Symposium on Image and Signal Processing and Analysis, 2003. ISPA 2003. Proceedings of the.

[28]  Klara Nahrstedt,et al.  Distortion over latency: Novel metric for measuring interactive performance in remote rendering systems , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[29]  Ioana M. Boier Martin.  Adaptive rendering of 3D models over net-works using multiple modalities , 2000 .

[30]  Marc Levoy,et al.  Protected interactive 3D graphics via remote rendering , 2004, ACM Trans. Graph..

[31]  Charles Woodward,et al.  Wireless 3D CAD Viewing on a PDA Device , 2002 .

[32]  S. B. Kang,et al.  Survey of image-based representations and compression techniques , 2003, IEEE Trans. Circuits Syst. Video Technol..

[33]  Daniel Cohen-Or,et al.  Streaming Scenes to MPEG-4 Video-Enabled Devices , 2003, IEEE Computer Graphics and Applications.

[34]  Andrea Sanna,et al.  A Streaming-Based Solution for Remote Visualization of 3D Graphics on Mobile Devices , 2007, IEEE Transactions on Visualization and Computer Graphics.

[35]  William Yurcik,et al.  Real-time 3D video compression for tele-immersive environments , 2006, Electronic Imaging.

[36]  Martin Reisslein,et al.  Objective Video Quality Assessment Methods: A Classification, Review, and Performance Comparison , 2011, IEEE Transactions on Broadcasting.

[37]  Sujit Dey,et al.  Cloud mobile gaming: modeling and measuring user experience in mobile wireless networks , 2012, MOCO.

[38]  Claudio Zunino,et al.  An accelerated remote graphics architecture for PDAS , 2003, Web3D '03.

[39]  Wolfgang Straßer,et al.  Tetrahedral mesh compression with the cut-border machine , 1999, Proceedings Visualization '99 (Cat. No.99CB37067).

[40]  Thomas Ertl,et al.  Combining local and remote visualization techniques for interactive volume rendering in medical applications , 2000, Proceedings Visualization 2000. VIS 2000 (Cat. No.00CH37145).

[41]  Marc Levoy,et al.  QSplat: a multiresolution point rendering system for large meshes , 2000, SIGGRAPH.

[42]  Emilio Camahort,et al.  Interactive three-dimensional rendering on mobile computer devices , 2005, ACE '05.

[43]  Renato Pajarola,et al.  Equalizer: A Scalable Parallel Rendering Framework , 2008, IEEE Transactions on Visualization and Computer Graphics.

[44]  Thomas A. Funkhouser,et al.  RING: a client-server system for multi-user virtual environments , 1995, I3D '95.

[45]  Vid Petrovic,et al.  The future of the CAVE , 2011 .

[46]  M. Orhon.  The X Window System , 2005 .

[47]  Wim Lamotte,et al.  On the Applicability of Remote Rendering of Networked Virtual Environments on Mobile Devices , 2006, 2006 International Conference on Systems and Networks Communications (ICSNC'06).

[48]  Thomas A. DeFanti,et al.  Two bit/pixel full color encoding , 1986, SIGGRAPH.

[49]  Luc Van Gool,et al.  Advanced three-dimensional television system technologies , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[50]  D. Gourlay,et al.  Remote walkthrough over mobile networks using 3-D image warping and streaming , 2004 .

[51]  Ulug Bayazit.  Macroblock data classification and nonlinear bit count estimation for low delay H.263 rate control , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[52]  Hugues Hoppe,et al.  Progressive meshes , 1996, SIGGRAPH.

[53]  Bo Wen,et al.  Realistic mesh compression based on geometry image , 2012, 2012 Picture Coding Symposium.

[54]  Marília Curado,et al.  An Overview of Quality of Experience Measurement Challenges for Video Applications in IP Networks , 2010, WWIC.

[55]  Murat Kunt,et al.  Recent results in high-compression image coding (Invited Papaer) , 1987 .

[56]  Jason Nieh,et al.  Measuring thin-client performance using slow-motion benchmarking , 2001, TOCS.

[57]  A. Abdel-Hamid,et al.  Prediction-based Prefetching for Remote Rendering Streaming in Mobile Virtual Environments , 2007, 2007 IEEE International Symposium on Signal Processing and Information Technology.

[58]  Bernd Fröhlich,et al.  An Image-Warping Architecture for VR: Low Latency versus Image Quality , 2009, 2009 IEEE Virtual Reality Conference.

[59]  Luc Van Gool,et al.  ATTEST: Advanced Three-dimensional Television System Technologies , 2002 .

[60]  Greg Humphreys,et al.  Chromium: a stream-processing framework for interactive rendering on clusters , 2002, SIGGRAPH.

[61]  Ioana M. Boier-Martin,et al.  An adaptive framework for 3D graphics over networks , 1999, Comput. Graph..

[62]  Shenchang Eric Chen,et al.  QuickTime VR: an image-based approach to virtual environment navigation , 1995, SIGGRAPH.

[63]  Zhengguo Li,et al.  A Novel Rate Control Scheme for Low Delay Video Communication of H.264/AVC Standard , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[64]  Sharon Crawford,et al.  Microsoft Windows NT Server 4.0 オフィシャルマニュアル , 1997 .

[65]  Stephan Wong,et al.  An efficient realization of forward integer transform in H.264/AVC intra-frame encoder , 2010, 2010 International Conference on Embedded Computer Systems: Architectures, Modeling and Simulation.

[66]  Albrecht Rothermel,et al.  Architecture Analysis for Low-Delay Video Coding , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[67]  Jason Nieh,et al.  On the performance of wide-area thin-client computing , 2006, TOCS.

[68]  E. Wes Bethel,et al.  Interactive, Internet Delivery of Visualization via Structured Prerendered Multiresolution Imagery , 2008, IEEE Transactions on Visualization and Computer Graphics.

[69]  Klara Nahrstedt,et al.  View-dependent real-time 3d video compression for mobile devices , 2008, ACM Multimedia.

[70]  Kwan-Liu Ma,et al.  Explorable images for visualizing volume data , 2010, 2010 IEEE Pacific Visualization Symposium (PacificVis).

[71]  Voicu Popescu,et al.  The WarpEngine: an architecture for the post-polygonal age , 2000, SIGGRAPH.

[72]  Michael Deering,et al.  Geometry compression , 1995, SIGGRAPH.

[73]  Hans-Peter Seidel,et al.  Scalable Remote Rendering with Depth and Motion‐flow Augmented Streaming , 2011, Comput. Graph. Forum.

[74]  Klara Nahrstedt,et al.  Real-time remote rendering of 3D video for mobile devices , 2009, MM '09.

[75]  Hai Tao,et al.  Compression and transmission of depth maps for image-based rendering , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[76]  Klaus Diepold,et al.  Depth map compression via compressed sensing , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[77]  Mark Claypool,et al.  Thin to win? Network performance analysis of the OnLive thin client game system , 2012, 2012 11th Annual Workshop on Network and Systems Support for Games (NetGames).

[78]  Cheng-Hsin Hsu,et al.  GamingAnywhere: an open cloud gaming system , 2013, MMSys.

[79]  Guangming Shi,et al.  Compress Compound Images in H.264/MPGE-4 AVC by Exploiting Spatial Correlation , 2010, IEEE Transactions on Image Processing.

[80]  C.-C. Jay Kuo,et al.  A dual graph approach to 3D triangular mesh compression , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[81]  Sathish S. Vadhiyar,et al.  An Adaptive Framework for Simulation and Online Remote Visualization of Critical Climate Applications in Resource-constrained Environments , 2010, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis.

[82]  E. Wes Bethel,et al.  Chromium Renderserver: Scalable and Open Remote Rendering Infrastructure , 2008, IEEE Transactions on Visualization and Computer Graphics.

[83]  Simone Milani,et al.  A cognitive approach for effective coding and transmission of 3D video , 2011, TOMCCAP.

[84]  Lifeng Sun,et al.  A multi-view video coding approach using Layered Depth Image , 2007, 2007 IEEE 9th Workshop on Multimedia Signal Processing.

[85]  Ming Li,et al.  Pseudo-Immersive Real-Time Display of 3D Scenes on Mobile Devices , 2011, 2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission.

[86]  Kadi Bouatouch,et al.  A Client-Server Approach to Image-Based Rendering on Mobile Terminals , 2005 .

[87]  Marc Levoy,et al.  Streaming QSplat: a viewer for networked visualization of large, dense models , 2001, I3D '01.

[88]  Klara Nahrstedt,et al.  Real-time parallel remote rendering for mobile devices using graphics processing units , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[89]  Mark Claypool,et al.  The effects of loss and latency on user performance in unreal tournament 2003® , 2004, NetGames '04.

[90]  William R. Mark,et al.  Post-Rendering 3D Image Warping: Visibility, Reconstruction, and Performance for Depth-Image Warping , 1999 .

[91]  I. Nave,et al.  Games@large graphics streaming architecture , 2008, 2008 IEEE International Symposium on Consumer Electronics.

[92]  Kwan-Liu Ma,et al.  A Parallel Visualization Pipeline for Terascale Earthquake Simulations , 2004, Proceedings of the ACM/IEEE SC2004 Conference.

[93]  Sujit Dey,et al.  Modeling and Characterizing User Experience in a Cloud Server Based Mobile Gaming Approach , 2009, GLOBECOM 2009 - 2009 IEEE Global Telecommunications Conference.

[94]  Bernd Hamann,et al.  A survey and performance analysis of software platforms for interactive cluster-based multi-screen rendering , 2003, IPT/EGVE.

[95]  Sang Uk Lee,et al.  Error Resilient 3-D Mesh Compression , 2006, IEEE Transactions on Multimedia.

[96]  M. Kunt,et al.  High compression image coding using an adaptive morphological subband decomposition , 1995, Proc. IEEE.

[97]  Tom Davis,et al.  Opengl programming guide: the official guide to learning opengl , 1993 .

[98]  Frédo Durand,et al.  A Survey of Visibility for Walkthrough Applications , 2003, IEEE Trans. Vis. Comput. Graph..

[99]  Marc Antonini,et al.  Multiresolution 3D mesh compression , 2002, Proceedings. International Conference on Image Processing.

[100]  Jin Li,et al.  Compression of the layered depth image , 2001, Proceedings DCC 2001. Data Compression Conference.

[101]  Harpreet S. Sawhney,et al.  A depth map representation for real-time transmission and view-based rendering of a dynamic 3D scene , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[102]  Kwan-Liu Ma,et al.  High Performance Visualization of Time-Varying Volume Data over a Wide-Area Network , 2000, ACM/IEEE SC 2000 Conference (SC'00).

[103]  Alessandro De Gloria,et al.  Platform for Distributed 3D Gaming , 2009, Int. J. Comput. Games Technol..

[104]  J. Duane Northcutt,et al.  The interactive performance of SLIM: a stateless, thin-client architecture , 1999, SOSP.

[105]  Kwan-Liu Ma.  A new approach to remote visualization of large volume data , 2010, COMG.

[106]  Chonho Lee,et al.  A survey of mobile cloud computing: architecture, applications, and approaches , 2013, Wirel. Commun. Mob. Comput..

[107]  Leila De Floriani,et al.  Multiresolution Mesh Representation: Models and Data Structures , 2002, Tutorials on Multiresolution in Geometric Modelling.

[108]  Shuhui Wang,et al.  A Unified LZ and Hybrid Coding for Compound Image Partial-Lossless Compression , 2009, 2009 2nd International Congress on Image and Signal Processing.

[109]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[110]  P. J. Narayanan,et al.  Compression of multiple depth maps for IBR , 2005, The Visual Computer.

[111]  Touradj Ebrahimi,et al.  The JPEG 2000 still image compression standard , 2001, IEEE Signal Process. Mag..

[112]  Jr. Leonard McMillan,et al.  An Image-Based Approach to Three-Dimensional Computer Graphics , 1997 .

[113]  Fausto Bernardini,et al.  Pervasive 3D Viewing for Product Data Management , 2003, IEEE Computer Graphics and Applications.

[114]  Craig Gotsman,et al.  Spectral compression of mesh geometry , 2000, EuroCG.

[115]  Michael W. Marcellin,et al.  An overview of JPEG-2000 , 2000, Proceedings DCC 2000. Data Compression Conference.

[116]  Tao Lin,et al.  United coding method for compound image compression , 2012, Multimedia Tools and Applications.

[117]  C.-C. Jay Kuo,et al.  Technologies for 3D mesh compression: A survey , 2005, J. Vis. Commun. Image Represent..

[118]  R. Schafer.  MPEG-4: a multimedia compression standard for interactive applications and services , 1998 .

[119]  Gordon Stoll,et al.  WireGL: a scalable graphics system for clusters , 2001, SIGGRAPH.

[120]  Chun-Fa Chang,et al.  Enhancing 3D Graphics on Mobile Devices by Image-Based Rendering , 2002, IEEE Pacific Rim Conference on Multimedia.

[121]  Marc Levoy.  Polygon-assisted JPEG and MPEG compression of synthetic images , 1995, SIGGRAPH.

[122]  Wenwu Zhu,et al.  Two decades of internet video streaming: A retrospective view , 2013, TOMCCAP.

[123]  Michael Zyda,et al.  Networked virtual environments - desgin and implementation , 1999 .

[124]  Jay Torborg,et al.  Talisman: commodity realtime 3D graphics for the PC , 1996, SIGGRAPH.

[125]  Lars C. Wolf,et al.  On the suitability of dead reckoning schemes for games , 2002, NetGames '02.

[126]  Yao Wang,et al.  Video Processing and Communications , 2001 .

[127]  Luc Renambot,et al.  SAGE: the Scalable Adaptive Graphics Environment , 2004 .

[128]  Murat Kunt,et al.  High compression image coding via directional filtering , 1985 .

[129]  L. Vasa,et al.  CODDYAC: Connectivity Driven Dynamic Mesh Compression , 2007, 2007 3DTV Conference.

[130]  Gary J. Sullivan,et al.  Overview of the Stereo and Multiview Video Coding Extensions of the H.264/MPEG-4 AVC Standard , 2011, Proceedings of the IEEE.

[131]  Joseph D. Touch,et al.  Defining High-Speed Protocols: Five Challenges and an Example that Survives the Challenges , 1995, IEEE J. Sel. Areas Commun..

[132]  Philip Ross.  Cloud Computing's Killer App: Gaming , 2009, IEEE Spectrum.

[133]  Peter Eisert,et al.  Low delay streaming of computer graphics , 2008, 2008 15th IEEE International Conference on Image Processing.

[134]  Sujit Dey,et al.  Adaptive Mobile Cloud Computing to Enable Rich Mobile Multimedia Applications , 2013, IEEE Transactions on Multimedia.

[135]  Pengwei Hao,et al.  Compound image compression for real-time computer screen image transmission , 2005, IEEE Transactions on Image Processing.

[136]  Aljoscha Smolic,et al.  The effects of multiview depth video compression on multiview rendering , 2009, Signal Process. Image Commun..

[137]  Greg Humphreys,et al.  Visual Computing in the Future: Computer Graphics as a Remote Service , 2003 .

[138]  Thomas Ertl,et al.  Remote line rendering for mobile devices , 2004, Proceedings Computer Graphics International, 2004..

[139]  Lurng-Kuo Liu,et al.  Low-delay MPEG-2 video coding , 1998, Electronic Imaging.

[140]  V. Radha,et al.  Enhanced layer based compound image compression , 2010, A2CWiC '10.

[141]  Dieter Schmalstieg.  The Remote Rendering Pipeline - Managing Geometry and Bandwidth in Distributed Virtual Environments , 1997 .

[142]  Klara Nahrstedt,et al.  A high-quality low-delay remote rendering system for 3D video , 2010, ACM Multimedia.

[143]  共立出版株式会社.  コンピュータ・サイエンス : ACM computing surveys , 1978 .

[144]  Ketan Mayer-Patel,et al.  Real-time multidepth stream compression , 2005, TOMCCAP.

[145]  Marius Preda,et al.  MPEG-4-based adaptive remote rendering for video games , 2011, Web3D '11.

[146]  Luigi Ciminiera,et al.  An Adaptive Control System to Deliver Interactive Virtual Environment Content to Handheld Devices , 2011, Mob. Networks Appl..

[147]  Rynson W. H. Lau,et al.  A hybrid motion prediction method for caching and prefetching in distributed virtual environments , 2001, VRST '01.

[148]  Byung Cheol Song,et al.  An intra-frame rate control algorithm for ultra low delay H.264/AVC coding , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[149]  Bing Han,et al.  Block-based method for real-time compound video compression , 2010, Defense + Commercial Sensing.

[150]  Wes Bethel.  Visapult: A Prototype Remote and Distributed Visualization Application and Framework , 2000, SIGGRAPH 2000.

[151]  Anna Gerber,et al.  Opengl Programming Guide The Official Guide To Learning Opengl Versions 3 0 And 3 1 , 2016 .

[152]  Qian Du,et al.  Hyperspectral Image Compression Using JPEG2000 and Principal Component Analysis , 2007, IEEE Geoscience and Remote Sensing Letters.

[153]  Stefan Eilemann,et al.  OpenGL multipipe SDK: a toolkit for scalable parallel rendering , 2005, VIS 05. IEEE Visualization, 2005..

[154]  Guillermo Sapiro,et al.  LOCO-I: a low complexity, context-based, lossless image compression algorithm , 1996, Proceedings of Data Compression Conference - DCC '96.

[155]  T. Tu,et al.  From Mesh Generation to Scientific Visualization: An End-to-End Approach to Parallel Supercomputing , 2006, ACM/IEEE SC 2006 Conference (SC'06).

[156]  Azzedine Boukerche,et al.  Remote rendering and streaming of progressive panoramas for mobile devices , 2006, MM '06.

[157]  Simon Stegmaier,et al.  A Generic Solution for Hardware-Accelerated Remote Visualization , 2002, VisSym.

[158]  Cheng-Hsin Hsu,et al.  On the Quality of Service of Cloud Gaming Systems , 2014, IEEE Transactions on Multimedia.

[159]  James P. Ahrens,et al.  Remote large data visualization in the paraview framework , 2006, EGPGV '06.

[160]  Jason Nieh,et al.  The Performance of Remote Display Mechanisms for Thin-Client Computing , 2002, USENIX Annual Technical Conference, General Track.