3D tele-immersion platform for interactive immersive experiences between remote users

Tele-immersion (TI) related technologies can change the way people interact and bridge the gap between the physical and digital worlds. However, while the technology itself advances, most developed platforms have complex setups and require large investments. In this work, a low-cost platform is introduced, integrating multiple TI-related advances. Focusing on ease of use and rapid deployment, a fast and fully automatic calibration method is proposed. The platform enables real-time 3D reconstruction of users and their placement into a pre-authored 3D environment. Moreover, interaction is achieved through the user's body posture, removing the need for additional equipment and enabling natural control while immersed. Developing a real-time TI platform requires the efficient integration of several multidisciplinary elements. An elegant, minimal solution to these challenges is proposed and validated in a prototype TI multiplayer game, SpaceWars.

[1]  Henry Fuchs,et al.  Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[2]  Petros Daras,et al.  Toward Real-Time and Efficient Compression of Human Time-Varying Meshes , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[3]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[4]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[5]  Daniel Berjón,et al.  Seamless, Static Multi‐Texturing of 3D Meshes , 2015, Comput. Graph. Forum.

[6]  Thomas S. Huang,et al.  Technologies for virtual reality/tele—immersion applications: issues of research in image display and global networking , 2001 .

[7]  Gregory K. Wallace,et al.  The JPEG still picture compression standard , 1991, CACM.

[8]  Petros Daras,et al.  Dynamic adaptive mesh streaming for real-time 3D teleimmersion , 2015, Web3D.

[9]  Petros Daras,et al.  Fast and smooth 3D reconstruction using multiple RGB-Depth sensors , 2014, 2014 IEEE Visual Communications and Image Processing Conference.

[10]  Adrien Bartoli,et al.  Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces , 2013, BMVC.

[11]  Ruzena Bajcsy,et al.  Collaborative dancing in tele-immersive environment , 2006, MM '06.

[12]  Klara Nahrstedt,et al.  "I'm the Jedi!" - A Case Study of User Experience in 3D Tele-immersive Gaming , 2010, 2010 IEEE International Symposium on Multimedia.

[13]  Petros Daras,et al.  Real-Time, Full 3-D Reconstruction of Moving Foreground Objects From Multiple Consumer Depth Cameras , 2013, IEEE Transactions on Multimedia.

[14]  Ruzena Bajcsy,et al.  TEEVE: the next generation architecture for tele-immersive environments , 2005, Seventh IEEE International Symposium on Multimedia (ISM'05).

[15]  D. Kendall.  A Survey of the Statistical Theory of Shape , 1989 .

[16]  Michael M. Kazhdan,et al.  Reconstruction of solid models from oriented point sets , 2005, SGP '05.

[17]  Petros Daras,et al.  Real-time, realistic full-body 3D reconstruction and texture mapping from multiple Kinects , 2013, IVMSP 2013.

[18]  Fang Yu,et al.  Space Connection: A New 3D Tele-immersion Platform for Web-Based Gesture-Collaborative Games and Services , 2015, 2015 IEEE/ACM 4th International Workshop on Games and Software Engineering.

[19]  Ruzena Bajcsy,et al.  High-Quality Visualization for Geographically Distributed 3-D Teleimmersive Applications , 2011, IEEE Transactions on Multimedia.

[20]  Klara Nahrstedt,et al.  MobileTI: a portable tele-immersive system , 2009, ACM Multimedia.

[21]  Aljoscha Smolic,et al.  3D video and free viewpoint video - From capture to display , 2011, Pattern Recognit..

[22]  Henry Fuchs,et al.  Real-time volumetric 3D capture of room-sized scenes for telepresence , 2012, 2012 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[23]  Klara Nahrstedt.  3D teleimmersion for remote injury assessment , 2012, SAM '12.

[24]  Petros Daras,et al.  A case study for tele-immersion communication applications: From 3D capturing to rendering , 2014, 2014 IEEE Visual Communications and Image Processing Conference.

[25]  Ruzena Bajcsy,et al.  Teleimmersive Environment for Remote Medical Collaboration , 2009, MMVR.