Real-time multidepth stream compression

The goal of tele-immersion has long been to enable people at remote locations to share a sense of presence. A tele-immersion system acquires the 3D representation of a collaborator's environment remotely and sends it over the network where it is rendered in the user's environment. Acquisition, reconstruction, transmission, and rendering all have to be done in real-time to create a sense of presence. With added commodity hardware resources, parallelism can increase the acquisition volume and reconstruction data quality while maintaining real-time performance. However, this is not as easy for rendering since all of the data need to be combined into a single display.In this article, we present an algorithm to compress data from such 3D environments in real-time to solve this imbalance. We present a compression algorithm which scales comparably to the acquisition and reconstruction, reduces network transmission bandwidth, and reduces the rendering requirement for real-time performance. This is achieved by exploiting the coherence in the 3D environment data and removing them in real-time. We have tested the algorithm using a static office data set as well as a dynamic scene, the results of which are presented in the article.

[1]  Anselmo Lastra,et al.  LDI tree: a hierarchical representation for image-based rendering , 1999, SIGGRAPH.

[2]  Pedro Larrañaga,et al.  An empirical comparison of four initialization methods for the K-Means algorithm , 1999, Pattern Recognit. Lett..

[3]  Reinhard Koch,et al.  Visual Modeling with a Hand-Held Camera , 2004, International Journal of Computer Vision.

[4]  Marc Levoy,et al.  QSplat: a multiresolution point rendering system for large meshes , 2000, SIGGRAPH.

[5]  William J. Dally,et al.  Point Sample Rendering , 1998, Rendering Techniques.

[6]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[7]  Andreas Kunz,et al.  blue-c: a spatially immersive display and 3D video portal for telepresence , 2003, ACM Trans. Graph..

[8]  Greg Welch,et al.  Toward a compelling sensation of telepresence: demonstrating a portal to a distant (static) office , 2000 .

[9]  Ketan Mayer-Patel,et al.  Real-time compression for dynamic 3D environments , 2003, MULTIMEDIA '03.

[10]  Oliver Schreer,et al.  An immersive 3D video-conferencing system using shared virtual team user environments , 2002, CVE '02.

[11]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[12]  Newton Lee,et al.  ACM Transactions on Multimedia Computing, Communications and Applications (ACM TOMCCAP) , 2007, CIE.

[13]  Ketan Mayer-Patel,et al.  Real-Time Terascale Implementation of Tele-immersion , 2003, International Conference on Computational Science.

[14]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[15]  H. Towles.  3D Tele-Immersion Over Internet2 , 2002 .

[16]  John G. Eyles,et al.  PixelFlow: high-speed rendering using image composition , 1992, SIGGRAPH.

[17]  Carolina Cruz-Neira,et al.  Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE , 2023 .

[18]  Greg Welch,et al.  Toward a compelling sensation of telepresence: demonstrating a portal to a distant (static) office , 2000, Proceedings Visualization 2000. VIS 2000 (Cat. No.00CH37145).

[19]  Thomas Malzbender,et al.  The Coliseum Immersive Teleconferencing System , 2002 .

[20]  Gordon Stoll,et al.  Lightning-2: a high-performance display subsystem for PC clusters , 2001, SIGGRAPH.

[21]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[22]  Markus H. Gross,et al.  3D video fragments: dynamic point samples for real-time free-viewpoint video , 2004, Comput. Graph..

[23]  Luc Van Gool,et al.  Blue-c: a spatially immersive display and 3D video portal for telepresence , 2003, IPT/EGVE.