High speed view interpolation for tele-teaching and tele-conferencing

This paper presents an algorithm to generate an interpolated view between two camera viewpoints in a fast and automatic way (6-7 fps on a PentIV @ 2.6 GHz, Geforce FX AGP 4). Nothing more than a desktop PC and a set of low end consumer grade cameras are needed to simulate the video stream of any intermediate camera. Parallel use of the GPU ('plane sweep' algorithm) and the CPU ('min-cut/max-flow' regularisation algorithm) is made to calculate the depth values. The final interpolations for any intermediate camera position are obtained by a projectively correct blended warp of the input images on a 3D mesh. Limited extrapolation is also feasible. The goal is to develop more advanced tele-teaching and videoconferencing environments, and this without the need of many cameras. Camera movements can be simulated and the best view can be selected whether this is recorded by a real camera or not. Compared to putting a human editor in control, the cost decreases dramatically, without losing all the added value of video stream editing.

[1]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[2]  Heinrich Niemann,et al.  Dense disparity maps in real-time with an application to augmented reality , 2002, Sixth IEEE Workshop on Applications of Computer Vision, 2002. (WACV 2002). Proceedings..

[3]  Paul Debevec,et al.  Modeling and Rendering Architecture from Photographs , 1996, SIGGRAPH 1996.

[4]  Vladimir Kolmogorov,et al.  An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision , 2004, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[6]  Ruigang Yang,et al.  Multi-resolution real-time stereo on commodity graphics hardware , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[7]  Steven M. Seitz,et al.  Toward image-based scene representation using view morphing , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[8]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[9]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[10]  Vladimir Kolmogorov,et al.  An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Andrew Blake,et al.  Gaze manipulation for one-to-one teleconferencing , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[12]  Til Aach,et al.  Illumination-invariant change detection , 2000, 4th IEEE Southwest Symposium on Image Analysis and Interpretation.

[13]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.