Video augmentation by image-based rendering under the perspective camera model

In this paper video streams are augmented by novel views of an object generated by an image-based rendering method. Reference views of the object and video images are captured by general uncalibrated perspective cameras. Our algorithm can be divided into three parts: 1) recovering structure information for reference views and video images; 2) interactive embedding of the model object into video images; and 3) rendering views of the object according to the motion of the video camera. Multiple model views are used to render views as the video camera changes its viewpoint. Generated views of the model are overlayed on the corresponding video images. Experimental results for real video and model images are given. Finally, a discussion on the limitations of the proposed method and subjects for future research are provided.

[1]  Ulrich Neumann,et al.  Dynamic registration correction in video-based augmented reality systems , 1995, IEEE Computer Graphics and Applications.

[2]  Amnon Shashua,et al.  Novel view synthesis in tensor space , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[3]  Reinhard Koch,et al.  Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[4]  Anders Heyden,et al.  Minimal Conditions on Intrinsic Parameters for Euclidean Reconstruction , 1998, ACCV.

[5]  Gilles Simon,et al.  Mixing synthesis and video images of outdoor environments: application to the bridges of Paris , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[6]  Olivier D. Faugeras,et al.  3-D scene representation as a collection of images , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[7]  Olivier D. Faugeras,et al.  Oriented Projective Geometry for Computer Vision , 1996, ECCV.

[8]  Michael Werman,et al.  Trilinearity of three perspective views and its associated tensor , 1995, Proceedings of IEEE International Conference on Computer Vision.

[9]  Kiriakos N. Kutulakos,et al.  Affine object representations for calibration-free augmented reality , 1996, Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium.

[10]  Steven M. Seitz,et al.  Toward image-based scene representation using view morphing , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[11]  Sing Bing Kang,et al.  Survey of image-based rendering techniques , 1998, Electronic Imaging.