Virtualized reality: concepts and early results

The visual medium evolved from early paintings to the realistic paintings of the classical era to photographs. The medium of moving imagery started with motion pictures. Television and video recording advanced it to show action "live" or capture and playback later. In all of the above media, the view of the scene is determined at the transcription time, independent of the viewer. We have been developing a new visual medium called virtualized reality. It delays the selection of the viewing angle until view time, using techniques from computer vision and computer graphics. The visual event is captured using many cameras that cover the action from all sides. The 3D structure of the event, aligned with the pixels of the image, is computed for a few selected directions using a stereo technique. Triangulation and texture mapping enable the placement of a "soft-camera" to reconstruct the event from any new viewpoint. With a stereo-viewing system, virtualized reality allows a viewer to move freely in the scene, independent of the transcription angles used to record the scene. We describe the hardware and software setup in our "studio" to make virtualized reality movies. Examples are provided to demonstrate the effectiveness of the system.

[1]  Jon A. Webb,et al.  Design, implementation, and performance of a scalable multi-camera interactive video capture system , 1995, Proceedings of Conference on Computer Architectures for Machine Perception.

[2]  Harry Shum,et al.  An integral approach to free-formed object modeling , 1995, Proceedings of IEEE International Conference on Computer Vision.

[3]  Larry S. Davis,et al.  Computer Architectures for Machine Perception , 1993 .

[4]  Marc Levoy,et al.  Zippered polygon meshes from range images , 1994, SIGGRAPH.

[5]  Richard Szeliski,et al.  Recovering 3D Shape and Motion from Image Streams Using Nonlinear Least Squares , 1994, J. Vis. Commun. Image Represent..

[6]  Denis Laurendeau,et al.  Multi-resolution surface modeling from multiple range views , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  Olivier D. Faugeras,et al.  3-D scene representation as a collection of images , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[8]  Takeo Kanade,et al.  A Multiple-Baseline Stereo , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Tony DeRose,et al.  Piecewise smooth surface reconstruction , 1994, SIGGRAPH.

[10]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[11]  Takeo Kanade,et al.  Virtual Space Teleconferencing Using a Sea of Cameras , 1994 .

[12]  Yuichi Ohta,et al.  Passive depth acquisition for 3D image displays , 1994 .