Multicamera Real-Time 3D Modeling for Telepresence and Remote Collaboration

We present a multicamera real-time 3D modeling system that aims at enabling new immersive and interactive environments. This system, called Grimage, allows to retrieve in real-time a 3D mesh of the observed scene as well as the associated textures. This information enables a strong visual presence of the user into virtual worlds. The 3D shape information is also used to compute collisions and reaction forces with virtual objects, enforcing the mechanical presence of the user in the virtual world. The innovation is a fully integrated system with both immersive and interactive capabilities. It embeds a parallel version of the EPVH modeling algorithm inside a distributed vision pipeline. It also adopts the hierarchical component approach of the FlowVR middleware to enforce software modularity and enable distributed executions. Results show high refresh rates and low latencies obtained by taking advantage of the I/O and computing resources of PC clusters. The applications we have developed demonstrate the quality of the visual and mechanical presence with a single platform and with a dual platform that allows telecollaboration.

[1]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[2]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Takeo Kanade,et al.  Constructing virtual worlds using dense stereo , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[4]  David Harwood.  A statistical approach for real time robust background subtraction , 1999 .

[5]  Takeo Kanade,et al.  A real time system for robust 3D voxel reconstruction of human motions , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[6]  Kostas Daniilidis,et al.  Real time trinocular stereo for tele-immersion , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[7]  Jean Ponce,et al.  On computing exact visual hulls of solids bounded by smooth surfaces , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[8]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[9]  Thomas Malzbender,et al.  The Coliseum Immersive Teleconferencing System , 2002 .

[10]  L. Gharai,et al.  Large Scale Video Conferencing : A Digital Amphitheater , 2002 .

[11]  Oliver Schreer,et al.  An immersive 3D video-conferencing system using shared virtual team user environments , 2002, CVE '02.

[12]  Luc Van Gool,et al.  Blue-c: a spatially immersive display and 3D video portal for telepresence , 2003, IPT/EGVE.

[13]  Edmond Boyer,et al.  Exact polyhedral visual hulls , 2003, BMVC.

[14]  Bruno Raffin,et al.  A Distributed Approach for Real Time 3D Modeling , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[15]  Wojciech Matusik,et al.  3D TV , 2004, SIGGRAPH '04.

[16]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[17]  Jérémie Allard,et al.  Marker-less Real Time 3D Modeling for Virtual Reality , 2004 .

[18]  Marc Pollefeys,et al.  Synchronization and calibration of camera networks from silhouettes , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[19]  Zhengyou Zhang,et al.  Camera calibration with one-dimensional objects , 2002, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Fast polygonal approximation of digital curves , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[21]  Wojciech Matusik,et al.  3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes , 2004, ACM Trans. Graph..

[22]  Jérémie Allard,et al.  FlowVR: A Middleware for Large Scale Virtual Reality Applications , 2004, Euro-Par.

[23]  Wojciech Matusik,et al.  3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes , 2004, ACM Trans. Graph..

[24]  Jérémie Allard,et al.  Running large VR applications on a PC cluster: the flow VR experience , 2005, EGVE'05.

[25]  John F. Canny,et al.  MultiView: spatially faithful group video conferencing , 2005, CHI.

[26]  Jérémie Allard,et al.  A shader-based parallel rendering framework , 2005, VIS 05. IEEE Visualization, 2005..

[27]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[28]  Jérémie Allard,et al.  The GrImage Platform: A Mixed Reality Environment for Interactions , 2006, Fourth IEEE International Conference on Computer Vision Systems (ICVS'06).

[29]  Edmond Boyer.  On Using Silhouettes for Camera Calibration , 2006, ACCV.

[30]  Jérémie Allard,et al.  Grimage: markerless 3D interactions , 2007, SIGGRAPH '07.

[31]  Stephane Cotin,et al.  EP4A: Software and Computer Based Simulator Research: Development and Outlook SOFA—An Open Source Framework for Medical Simulation , 2007, MMVR.

[32]  Adrian Hilton,et al.  Wand-based Multiple Camera Studio Calibration , 2007 .

[33]  Bruno Raffin,et al.  A hierarchical component model for large parallel interactive applications , 2012, The Journal of Supercomputing.

[34]  Fred Daneshgaran,et al.  Special Issue on Iterative Decoding and Cross-Layering Techniques for Multimedia Broadcasting and Communications per la rivista International Journal of Digital Multimedia Broadcasting , 2008 .

[35]  Bruno Raffin,et al.  High Performance Interactive Computing with FlowVR , 2008 .

[36]  Bruno Raffin,et al.  Grimage: 3D modeling for remote collaboration and telepresence , 2008, VRST '08.

[37]  Ruzena Bajcsy,et al.  Immersive 3D Environment for Remote Collaboration and Training of Physical Activities , 2008, 2008 IEEE Virtual Reality Conference.

[38]  Wojciech Matusik,et al.  Articulated mesh animation from multi-view silhouettes , 2008, ACM Trans. Graph..

[39]  François Faure,et al.  Interactive Physical Simulation on Multicore Architectures , 2009, EGPGV@Eurographics.

[40]  Edmond Boyer,et al.  Efficient Polyhedral Modeling from Silhouettes , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Hans-Peter Seidel,et al.  Motion capture using joint skeleton tracking and surface estimation , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.