The plenoptic video

This paper presents a system for capturing and rendering a dynamic image-based representation called the plenoptic video. It is a simplified light field for dynamic environments, where user viewpoints are constrained to the camera plane of a linear array of video cameras. Important issues such as multiple camera calibration, real-time compression, decompression and rendering are addressed. The system consists of a camera array of eight Sony CCX-Z11 CCD cameras and eight Pentium 4 1.8-GHz computers connected together through a 100 Base-T local area network. It is possible to perform software-assisted real-time MPEG-2 compression at a resolution of (720/spl times/480). Using selective transmission, we are able to stream continuously plenoptic video with (256/spl times/256) resolution at a rate of 15 f/s over the network. For rendering from raw data on the hard disk, real-time rendering can be achieved with a resolution of (720/spl times/480) and a rate of 15 f/s. A new compression algorithm using both temporal and spatial predictions is also proposed for the efficient compression of the plenoptic videos. Experimental results demonstrate the usefulness of the proposed parallel processing based system in capturing and rendering high-quality dynamic image-based representations using off-the-shelf equipment, and its potential applications in visualization and immersive television systems.

[1]  Harry Shum,et al.  On the Compression of Image Based Rendering Scene: A Comparison among Block, Reference and Wavelet Coders , 2001, Int. J. Image Graph..

[2]  Harry Shum,et al.  Am object-based approach to plenoptic videos , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[3]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[4]  J-R. Olm.  Encoding and reconstruction of multiview video objects , 1999 .

[5]  Harry Shum,et al.  The compression of simplified dynamic light fields , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[6]  Harry Shum,et al.  The plenoptic videos: capturing, rendering and compression , 2004, 2004 IEEE International Symposium on Circuits and Systems (IEEE Cat. No.04CH37512).

[7]  Leonard McMillan,et al.  A Real-Time Distributed Light Field Camera , 2002, Rendering Techniques.

[8]  Barry G. Haskell,et al.  Basics of stereoscopic video, new compression results with MPEG-2 and a proposal for MPEG-4 , 1997, Signal Process. Image Commun..

[9]  P. Debevec,et al.  Image-based modeling, rendering, and lighting , 2002, IEEE Computer Graphics and Applications.

[10]  Harry Shum,et al.  Pop-up light field: An interactive image-based modeling and rendering system , 2004, TOGS.

[11]  Takeshi Naemura,et al.  Pixel independent random access image sensor for real time image-based rendering system , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[12]  Harry Shum,et al.  On the data compression and transmission aspects of panoramic video , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[13]  M. Lukacs,et al.  Predictive coding of multi-viewpoint image sets , 1986, ICASSP '86. IEEE International Conference on Acoustics, Speech, and Signal Processing.

[14]  Sotiris Malassiotis,et al.  Object-based coding of stereoscopic and 3D image sequences , 1999, IEEE Signal Process. Mag..

[15]  Takeshi Naemura,et al.  Real-Time Video-Based Modeling and Rendering of 3D Scenes , 2002, IEEE Computer Graphics and Applications.

[16]  Daniel G. Aliaga,et al.  Plenoptic stitching: a scalable method for reconstructing 3D interactive walk throughs , 2001, SIGGRAPH.

[17]  S. B. Kang,et al.  Survey of image-based representations and compression techniques , 2003, IEEE Trans. Circuits Syst. Video Technol..

[18]  Richard Szeliski,et al.  Creating full view panoramic image mosaics and environment maps , 1997, SIGGRAPH.

[19]  Takeshi Naemura,et al.  Compression and representation of 3-D images , 1999 .

[20]  Marcus A. Magnor,et al.  Model-based coding of multiviewpoint imagery , 2000, Visual Communications and Image Processing.

[21]  Xin Tong,et al.  Coding of multi-view images for immersive viewing , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).

[22]  Mark A. Horowitz,et al.  Light field video camera , 2000, IS&T/SPIE Electronic Imaging.

[23]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[24]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[25]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[26]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[27]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[28]  Harry Shum,et al.  Virtual reality using the concentric mosaic: construction, rendering and data compression , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[29]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[30]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[31]  Peter Eisert,et al.  Model-aided coding of multi-viewpoint image data , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[32]  Marcus A. Magnor,et al.  Data compression for light-field rendering , 2000, IEEE Trans. Circuits Syst. Video Technol..

[33]  Marcus A. Magnor,et al.  Hardware-Accelerated Dynamic Light Field Rendering , 2002, VMV.

[34]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[35]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[36]  Harry Shum,et al.  Rendering with concentric mosaics , 1999, SIGGRAPH.

[37]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[38]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..