Real-time streaming of point-based 3D video

Free-viewpoint video is a promising technology for next-generation virtual and augmented reality applications. Our goal is to enhance collaborative VR applications with 3D video-conferencing features. In this paper, we propose a 3D video streaming technique which can be deployed in telepresence environments. The streaming characteristics of real-time 3D video sequences are investigated under various system and networking conditions. We introduce several encoding techniques and analyze their behavior with respect to resolution, bandwidth and inter-frame jitter. Our 3D video pipeline uses point samples as basic primitives and is fully integrated with a communication framework handling acknowledgment information for reliable network transmissions and application control data. The 3D video reconstruction process dynamically adapts to processing and networking bottlenecks. Our results show that a reliable transmission of our pixel-based differential prediction encoding leads to the best performance in terms of bandwidth, but is also quite sensitive to packet losses. A redundantly encoded stream achieves better results in presence of burst losses and seamlessly adapts to varying network throughput.

[1]  Massachusett Framingham,et al.  The Common Object Request Broker: Architecture and Specification Version 3 , 2003 .

[2]  Ramesh C. Jain,et al.  Immersive video , 1996, Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium.

[3]  Hirokazu Kato,et al.  3-D live: real time interaction for mixed reality , 2002, CSCW '02.

[4]  Pedro V. Sander,et al.  Geometry videos: a new representation for 3D animations , 2003, SCA '03.

[5]  Mathieu Desbrun,et al.  Progressive encoding of complex isosurfaces , 2003, ACM Trans. Graph..

[6]  Marc Levoy,et al.  Streaming QSplat: a viewer for networked visualization of large, dense models , 2001, I3D '01.

[7]  Markus H. Gross,et al.  3D video fragments: dynamic point samples for real-time free-viewpoint video , 2004, Comput. Graph..

[8]  Jarek Rossignac,et al.  Edgebreaker: Connectivity Compression for Triangle Meshes , 1999, IEEE Trans. Vis. Comput. Graph..

[9]  E. O. Elliott.  Estimates of error rates for codes on burst-noise channels , 1963 .

[10]  Markus H. Gross,et al.  3D video recorder , 2002, 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings..

[11]  Jason Leigh,et al.  Reliable Blast UDP : predictable high performance bulk data transfer , 2002, Proceedings. IEEE International Conference on Cluster Computing.

[12]  G. Nigel Martin,et al.  * Range encoding: an algorithm for removing redundancy from a digitised message , 1979 .

[13]  Khalid Sayood,et al.  Introduction to Data Compression , 1996 .

[14]  Takeo Kanade,et al.  Spatio-Temporal View Interpolation , 2002, Rendering Techniques.

[15]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[16]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..

[17]  Marc Levoy,et al.  QSplat: a multiresolution point rendering system for large meshes , 2000, SIGGRAPH.

[18]  Samuel P. Morgan,et al.  SMART retransmission: performance with overload and random losses , 1997, Proceedings of INFOCOM '97.

[19]  David D. Clark,et al.  Architectural considerations for a new generation of protocols , 1990, SIGCOMM '90.

[20]  Wojciech Matusik,et al.  Polyhedral Visual Hulls for Real-Time Rendering , 2001, Rendering Techniques.

[21]  Henning Schulzrinne,et al.  RTP: A Transport Protocol for Real-Time Applications , 1996, RFC.

[22]  Leif Kobbelt,et al.  Efficient High Quality Rendering of Point Sampled Geometry , 2002, Rendering Techniques.

[23]  M. Gutierrez,et al.  A MPEG-4 virtual human animation engine for interactive web based applications , 2002, Proceedings. 11th IEEE International Workshop on Robot and Human Interactive Communication.

[24]  Laurent Kling.  1,2,3 A,B,C a,b,c , 2001 .

[25]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[26]  Wolfgang Straßer,et al.  Real time compression of triangle mesh connectivity , 1998, SIGGRAPH.

[27]  Luc Van Gool,et al.  Blue-c: a spatially immersive display and 3D video portal for telepresence , 2003, IPT/EGVE.