Combining approximate geometry with view-dependent texture mapping - a hybrid approach to 3D video teleconferencing

We present a hybrid system using computer vision and graphics methods that effectively combines fast automatic 3D model extraction with view-dependent texture mapping for the purpose of real-time person-to-person 3D video teleconferencing. In our approach the problem of creating a dense disparity map is bypassed. We compute only a sparse geometric model, which is textured from multiple cameras using an image-based rendering approach. Using a commodity graphics card and three personal computers arranged in a cluster, we achieve a performance of 3 to 4 frames per second. We demonstrate the ability to provide a 3D tele-immersive environment with our system.

[1]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[2]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[3]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Kostas Daniilidis,et al.  Real time trinocular stereo for tele-immersion , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[5]  Aldo Laurentini,et al.  How Far 3D Shapes Can Be Understood from 2D Silhouettes , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  P. Anandan,et al.  Direct recovery of shape from multiple views: a parallax based approach , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[7]  Sean Hayes,et al.  View synthesis by edge transfer with application to the generation of immersive video objects , 1998, VRST '98.

[8]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[9]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[10]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[11]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[12]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[13]  Reinhard Koch,et al.  Plenoptic Modeling and Rendering from Image Sequences Taken by Hand-Held Camera , 1999, DAGM-Symposium.

[14]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[15]  Harry Shum,et al.  Rendering with concentric mosaics , 1999, SIGGRAPH.

[16]  William Buxton,et al.  Turning your video monitor into a virtual window , 1995, IEEE Pacific Rim Conference on Communications, Computers, and Signal Processing. Proceedings.

[17]  Aldo Laurentini.  How Many 2D Silhouettes Does It Take to Reconstruct a 3D Object? , 1997, Comput. Vis. Image Underst..

[18]  Christian P. Spagno,et al.  Technical System for Collaborative Work , 2002, EGVE.

[19]  Oliver Schreer,et al.  Real-Time Disparity Analysis for Immersive 3-D Teleconferencing by Hybrid Recursive Matching and Census Transform , 2001, ICCV 2001.

[20]  William W. Gaver,et al.  A Virtual Window on media space , 1995, CHI '95.

[21]  W. Brent Seales,et al.  Geometrically correct imagery for teleconferencing , 1999, MULTIMEDIA '99.

[22]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[23]  Kostas Daniilidis,et al.  View-independent scene acquisition for tele-presence , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[24]  Takeo Kanade,et al.  Appearance-based virtual view generation of temporally-varying events from multi-camera images in the 3D room , 1999, Second International Conference on 3-D Digital Imaging and Modeling (Cat. No.PR00062).

[25]  Ruigang Yang,et al.  Model-based head pose tracking with stereovision , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[26]  Greg Welch,et al.  The office of the future: a unified approach to image-based modeling and spatially immersive displays , 1998, SIGGRAPH.

[27]  O. Faugeras.  Three-dimensional computer vision: a geometric viewpoint , 1993 .