Online model reconstruction for interactive virtual environments

We present a system for generating real-time 3D reconstructions of the user and other real objects in an immersive virtual environment (IVE) for visualization and interaction. For example, when parts of the user's body are in his field of view, our system allows him to see a visually faithful graphical representation of himself, an avatar. In addition, the user can grab real objects, and then see and interact with those objects in the IVE. Our system bypasses an explicit 3D modeling stage, and does not use additional tracking sensors or prior object knowledge, nor do we generate dense 3D representations of objects using computer vision techniques. We use a set of outside-looking-in cameras and a novel visual hull technique that leverages the tremendous recent advances in graphics hardware performance and capabilities. We accelerate the visual hull computation by using projected textures to rapidly determine which volume samples lie within the visual hull. The samples are combined to form the object reconstruction from any given viewpoint. Our system produces results at interactive rates, and because it harnesses ever-improving graphics hardware, the rates and quality should continue to improve. We further examine realtime generated models as active participants in simulations (with lighting) in IVEs, and give results using synthetic and real data. Additional

[1]  James F. Blinn,et al.  Blue screen matting , 1996, SIGGRAPH.

[2]  Fabrice Neyret,et al.  Interactive Volumetric Textures , 1998, Rendering Techniques.

[3]  Jake K. Aggarwal,et al.  Volume/surface octrees for the representation of three-dimensional objects , 1986, Comput. Vis. Graph. Image Process..

[4]  Kevin J. Dalton,et al.  3D shape reconstruction using volume intersection techniques , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[5]  Mel Slater,et al.  Body Centred Interaction in Immersive Virtual Environments , 1994 .

[6]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Takeo Kanade,et al.  DigitEyes: vision-based hand tracking for human-computer interaction , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[8]  Hideo Saito,et al.  Modeling, Combining, and Rendering Dynamic Real-World Events From Image Sequences , 1998 .

[9]  Mary C. Whitton,et al.  Walking > walking-in-place > flying, in virtual environments , 1999, SIGGRAPH.

[10]  Ruzena Bajcsy,et al.  Fish-scales: representing fuzzy manifolds , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[11]  Takeo Kanade Peter,et al.  Virtualized Reality: Digitizing a 3D Time-Varying Event As Isand in Real Time , 1999 .

[12]  Greg Welch,et al.  The HiBall Tracker: high-performance wide-area tracking for virtual and augmented environments , 1999, VRST '99.

[13]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[14]  Richard Szeliski,et al.  Rapid octree construction from image sequences , 1993 .

[15]  Takeo Kanade,et al.  Virtualized Reality : Digitizing a 3D Time-Varying Event As Is and in Real Time , 1999 .

[16]  Michael Potmesil.  Generating octree models of 3D objects from their silhouettes in a sequence of images , 1987, Comput. Vis. Graph. Image Process..

[17]  Paul Debevec,et al.  13th Eurographics Workshop on Rendering , 2002 .

[18]  Ramesh C. Jain,et al.  Reality modeling and visualization from multiple video sequences , 1996, IEEE Computer Graphics and Applications.