A movable image-based rendering system and its application to multiview audio-visual conferencing

Image-based rendering (IBR) is an emerging technology for rendering photo-realistic views of scenes from a collection of densely sampled images or videos. It provides a framework for developing revolutionary virtual reality and immersive viewing systems. This paper studies the design of a movable image-based rendering system based on a class of dynamic representations called plenoptic videos. It is constructed by mounting a linear array of 8 video cameras on an electrically controllable wheel chair with its motion being controllable manually or remotely through wireless LAN by means of additional hardware circuitry. We also developed a real-time object tracking algorithm and utilize the motion information computed to adjust continuously the azimuth or rotation angle of the movable IBR system in order to cope with a given moving object. Due to the motion of the wheel chair, videos may appear shaky and video stabilization technique is proposed to overcome this problem. The system can be used in a multiview audio-visual conferencing via a multiview TV display. Through this pilot study, we hope to develop a framework for designing movable IBR systems with improved viewing freedom and ability to cope with moving object in large environment.

[1]  J.-Y. Bouguet,et al.  Pyramidal implementation of the lucas kanade feature tracker , 1999 .

[2]  Harry Shum,et al.  Object tracking and matting for a class of dynamic image-based representations , 2005, IEEE Conference on Advanced Video and Signal Based Surveillance, 2005..

[3]  Harry Shum,et al.  An Object-Based Approach to Image/Video-Based Synthesis and Processing for 3-D and Multiview Televisions , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[4]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[5]  Harry Shum,et al.  The plenoptic video , 2005, IEEE Trans. Circuits Syst. Video Technol..

[6]  Harry Shum,et al.  Image-based rendering , 2006, Found. Trends Comput. Graph. Vis..

[7]  Shmuel Peleg,et al.  Panoramic mosaics by manifold projection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[9]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Harry Shum,et al.  The plenoptic videos: capturing, rendering and compression , 2004, 2004 IEEE International Symposium on Circuits and Systems (IEEE Cat. No.04CH37512).

[11]  Heung-Yeung Shum,et al.  Image-Based Rendering and Synthesis , 2007, IEEE Signal Processing Magazine.

[12]  M. Landy,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[13]  Harry Shum,et al.  Object-Based Coding for Plenoptic Videos , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[14]  Shing-Chow Chan,et al.  A Recursive Frequency Estimator Using Linear Prediction and a Kalman-Filter-Based Iterative Algorithm , 2008, IEEE Transactions on Circuits and Systems II: Express Briefs.

[15]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[16]  Harry Shum,et al.  On object-based compression for a class of dynamic image-based representations , 2005, IEEE International Conference on Image Processing 2005.

[17]  Harry Shum,et al.  Rendering with concentric mosaics , 1999, SIGGRAPH.

[18]  Jian Sun,et al.  Lazy snapping , 2004, SIGGRAPH 2004.

[19]  Andrea Fusiello.  Image-based Rendering * , 2003 .

[20]  Zhi-Feng Gan,et al.  Title An Object-Based Approach to Plenoptic Videos , 2011 .