The Design and Construction of a Movable Image-Based Rendering System and Its Application to Multiview Conferencing

Image-based rendering (IBR) is an promising technology for rendering photo-realistic views of scenes from a collection of densely sampled images or videos. It provides a framework for developing revolutionary virtual reality and immersive viewing systems. While there has been considerable progress recently in the capturing, storage and transmission of image-based representations, most multiple camera systems are designed to be stationary and hence their ability to cope with moving objects and dynamic environment is somewhat limited. This paper studies the design and construction of a movable image-based rendering system based on a class of dynamic representations called plenoptic videos, its associated video processing algorithms and an application to multiview audio-visual conferencing. It is constructed by mounting a linear array of 8 video cameras on an electrically controllable wheel chair and its motion is controllable manually or remotely through wireless LAN by means of additional hardware circuitry. We also developed a real-time object tracking algorithm and utilize the motion information computed to adjust continuously the azimuth or rotation angle of the movable IBR system in order to cope with a given moving object in a large environment. Due to imperfection in tracking and mechanical vibration encountered in movable systems, the videos may appear very shaky and a new video stabilization technique is proposed to overcome this problem. The usefulness of the system is illustrated by means of a multiview conferencing application using a multiview TV display. Through this pilot study, we hope to disseminate useful experience for the design and construction of movable IBR systems with improved viewing freedom and ability to cope with moving object in a large environment.

[1]  Harry Shum,et al.  Am object-based approach to plenoptic videos , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[2]  Harry Shum,et al.  Full-frame video stabilization , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[3]  Tony F. Chan,et al.  Active contours without edges , 2001, IEEE Trans. Image Process..

[4]  M. Landy,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[5]  Dorin Comaniciu,et al.  Kernel-Based Object Tracking , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[7]  Harry Shum,et al.  Pop-up light field: An interactive image-based modeling and rendering system , 2004, TOGS.

[8]  J. Sethian.  Level set methods : evolving interfaces in geometry, fluid mechanics, computer vision, and materials science , 1996 .

[9]  G. Sapiro,et al.  Geometric partial differential equations and image analysis [Book Reviews] , 2001, IEEE Transactions on Medical Imaging.

[10]  Harry Shum,et al.  Object-Based Coding for Plenoptic Videos , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[11]  Michael Gleicher,et al.  Content-preserving warps for 3D video stabilization , 2009, ACM Trans. Graph..

[12]  Heung-Yeung Shum,et al.  Image-Based Rendering and Synthesis , 2007, IEEE Signal Processing Magazine.

[13]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[14]  Harry Shum,et al.  On object-based compression for a class of dynamic image-based representations , 2005, IEEE International Conference on Image Processing 2005.

[15]  Harry Shum,et al.  Rendering with concentric mosaics , 1999, SIGGRAPH.

[16]  Richard Szeliski,et al.  Creating full view panoramic image mosaics and environment maps , 1997, SIGGRAPH.

[17]  Thia Kirubarajan,et al.  Estimation with Applications to Tracking and Navigation: Theory, Algorithms and Software , 2001 .

[18]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[19]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[20]  King-To Ng,et al.  A virtual reality system using the concentric mosaic: construction, rendering, and data compression , 2005, IEEE Transactions on Multimedia.

[21]  Harry Shum,et al.  The compression of simplified dynamic light fields , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[22]  Toshiaki Fujii,et al.  Ray space coding for 3D visual communication , 1996 .

[23]  Shenchang Eric Chen,et al.  QuickTime VR: an image-based approach to virtual environment navigation , 1995, SIGGRAPH.

[24]  Kun Zhou,et al.  Precomputed shadow fields for dynamic scenes , 2005, SIGGRAPH 2005.

[25]  Harry Shum,et al.  Data compression and transmission aspects of panoramic videos , 2005, IEEE Trans. Circuits Syst. Video Technol..

[26]  Harry Shum,et al.  The plenoptic video , 2005, IEEE Trans. Circuits Syst. Video Technol..

[27]  Harry Shum,et al.  Image-based rendering , 2006, Found. Trends Comput. Graph. Vis..

[28]  J.-Y. Bouguet,et al.  Pyramidal implementation of the lucas kanade feature tracker , 1999 .

[29]  Shing-Chow Chan,et al.  A Recursive Frequency Estimator Using Linear Prediction and a Kalman-Filter-Based Iterative Algorithm , 2008, IEEE Transactions on Circuits and Systems II: Express Briefs.

[30]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[31]  Michael Gleicher,et al.  Content-preserving warps for 3D video stabilization , 2009, ACM Trans. Graph..

[32]  Shmuel Peleg,et al.  Panoramic mosaics by manifold projection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[33]  Harry Shum,et al.  An Object-Based Approach to Image/Video-Based Synthesis and Processing for 3-D and Multiview Televisions , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[34]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[35]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[37]  S. Osher,et al.  Geometric Level Set Methods in Imaging, Vision, and Graphics , 2011, Springer New York.

[38]  Jian Sun,et al.  Lazy snapping , 2004, SIGGRAPH 2004.

[39]  M. Halle,et al.  3-D Displays and Signal Processing , 2007, IEEE Signal Processing Magazine.

[40]  Shing-Chow Chan,et al.  An object-based compression system for a class of dynamic image-based representations , 2005, Visual Communications and Image Processing.

[41]  Shing-Chow Chan,et al.  Object tracking for a class of dynamic image-based representations , 2005, Visual Communications and Image Processing.

[42]  James A. Sethian,et al.  Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry, Fluid , 2012 .

[43]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[44]  Shing-Chow Chan,et al.  Data compression and transmission aspects of panoramic videos , 2005 .

[45]  Harry Shum,et al.  Object tracking and matting for a class of dynamic image-based representations , 2005, IEEE Conference on Advanced Video and Signal Based Surveillance, 2005..

[46]  Harry Shum,et al.  The plenoptic videos: capturing, rendering and compression , 2004, 2004 IEEE International Symposium on Circuits and Systems (IEEE Cat. No.04CH37512).