Multi-camera tele-immersion system with real-time model driven data compression

Vision-based full-body 3D reconstruction for tele-immersive applications generates large amount of data points, which have to be sent through the network in real time. In this paper, we introduce a skeleton-based compression method using motion estimation where kinematic parameters of the human body are extracted from the point cloud data in each frame. First we address the issues regarding the data capturing and transfer to a remote site for the tele-immersive collaboration. We compare the results of the existing compression methods and the proposed skeleton-based compression technique. We examine the robustness and efficiency of the algorithm through experimental results with our multi-camera tele-immersion system. The proposed skeleton-based method provides high and flexible compression ratios from 50:1 to 5000:1 with reasonable reconstruction quality (peak signal-to-noise ratio from 28 to 31 dB) while preserving real-time (10+ fps) processing.

[1]  Maureen K. Holden,et al.  Virtual Environments for Motor Rehabilitation: Review , 2005, Cyberpsychology Behav. Soc. Netw..

[2]  Jake K. Aggarwal,et al.  Human Motion Analysis: A Review , 1999, Comput. Vis. Image Underst..

[3]  Ruzena Bajcsy,et al.  The Effects of Fully Immersive Virtual Reality on the Learning of Physical Tasks , 2006 .

[4]  Ruzena Bajcsy,et al.  Skeleton-Based Data Compression for Multi-camera Tele-Immersion System , 2007, ISVC.

[5]  Takeo Kanade,et al.  Real-time 3-D pose estimation using a high-speed range sensor , 1993, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[6]  François X. Sillion,et al.  A Real-Time System for Full Body Interaction with Virtual Worlds , 2004, EGVE.

[7]  Jon Louis Bentley,et al.  Multidimensional binary search trees used for associative searching , 1975, CACM.

[8]  Dietmar Saupe,et al.  Compression of Point-Based 3D Models by Shape-Adaptive Wavelet Coding of Multi-Height Fields , 2004, PBG.

[9]  Luc Van Gool,et al.  Blue-c: a spatially immersive display and 3D video portal for telepresence , 2003, IPT/EGVE.

[10]  Richard A. Robb,et al.  Virtual reality in medicine: A personal perspective , 2002 .

[11]  William Yurcik,et al.  Real-time 3D video compression for tele-immersive environments , 2006, Electronic Imaging.

[12]  Trevor Darrell,et al.  Stereo tracking using ICP and normal flow constraint , 2002, Object recognition supported by user interaction for service robots.

[13]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[14]  Markus H. Gross,et al.  3D video fragments: dynamic point samples for real-time free-viewpoint video , 2004, Comput. Graph..

[15]  Craig Gotsman,et al.  Compression of soft-body animation sequences , 2004, Comput. Graph..

[16]  Markus H. Gross,et al.  Real-time streaming of point-based 3D video , 2004, IEEE Virtual Reality 2004.

[17]  Thomas Malzbender,et al.  The Coliseum Immersive Teleconferencing System , 2002 .

[18]  Rüdiger Dillmann,et al.  Sensor fusion for 3D human body tracking with an articulated 3D body model , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[19]  Ketan Mayer-Patel,et al.  Real-time multidepth stream compression , 2005, TOMCCAP.

[20]  Jim X. Chen,et al.  Rendering avatars in virtual reality: integrating a 3D model with 2D images , 2002, Comput. Sci. Eng..

[21]  Radu Horaud,et al.  Hand Motion from 3D Point Trajectories and a Smooth Surface Model , 2004, ECCV.

[22]  Trevor Darrell,et al.  3-D articulated pose tracking for untethered diectic reference , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[23]  Hans-Peter Seidel,et al.  Predictive point-cloud compression , 2005, SIGGRAPH '05.

[24]  Michael Isard,et al.  CONDENSATION—Conditional Density Propagation for Visual Tracking , 1998, International Journal of Computer Vision.

[25]  J. Lanier.  Virtually there. , 2001, Scientific American.

[26]  Jinghua Zhang,et al.  Octree-based animated geometry compression , 2004, Data Compression Conference, 2004. Proceedings. DCC 2004.

[27]  Joan L. Mitchell,et al.  JPEG: Still Image Data Compression Standard , 1992 .

[28]  Daniel Thalmann,et al.  Real-Time Animation of Realistic Virtual Humans , 1998, IEEE Computer Graphics and Applications.

[29]  Wenyi Zhao,et al.  Effects of camera alignment errors on stereoscopic depth estimates , 1996, Pattern Recognit..

[30]  Marc Levoy,et al.  Efficient variants of the ICP algorithm , 2001, Proceedings Third International Conference on 3-D Digital Imaging and Modeling.

[31]  Massimo Piccardi,et al.  Background subtraction techniques: a review , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[32]  Jarek Rossignac,et al.  Dynapack: space-time compression of the 3D animations of triangle meshes with fixed connectivity , 2003, SCA '03.

[33]  Ralf Sarlette,et al.  Simple and efficient compression of animation sequences , 2005, SCA '05.

[34]  Emily A Keshner,et al.  Virtual reality and physical rehabilitation: a new toy or a new research and rehabilitation tool? , 2004, Journal of NeuroEngineering and Rehabilitation.

[35]  Dariu Gavrila,et al.  The Visual Analysis of Human Movement: A Survey , 1999, Comput. Vis. Image Underst..

[36]  Kostas Daniilidis,et al.  Real time trinocular stereo for tele-immersion , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[37]  Yoshihiko Nomura,et al.  Error analysis and optimization of camera calibration , 1991, Proceedings IROS '91:IEEE/RSJ International Workshop on Intelligent Robots and Systems '91.

[38]  Jed Lengyel,et al.  Compression of time-dependent geometry , 1999, SI3D.

[39]  J. McComas,et al.  Current uses of virtual reality for children with disabilities. , 1998, Studies in health technology and informatics.

[40]  Marc Alexa,et al.  Representing Animations by Principal Components , 2000, Comput. Graph. Forum.

[41]  Takeo Kanade,et al.  Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[42]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  Okan Arikan.  Compression of motion capture databases , 2006, SIGGRAPH 2006.

[44]  Ruzena Bajcsy,et al.  A Framework for Constructing Real-time Immersive Environments for Training Physical Activities , 2006, J. Multim..

[45]  Lei Li,et al.  ERT-VR: an immersive virtual reality system for emergency rescue training , 2004, Virtual Reality.

[46]  R. Kenyon,et al.  Using Immersive Technology for Postural Research and Rehabilitation , 2004, Assistive technology : the official journal of RESNA.