Hierarchical coarse to fine depth estimation for realistic view interpolation

This paper presents a novel approach for view synthesis and image interpolation. The algorithm is build up in a hierarchical way, and this on different structural levels instead of using a classic image pyramid. First coarse matching is done on a 'shape basis' only. A background-foreground segmentation yields a fairly accurate contour for every incoming video stream. Inter-relating these contours is a 1D problem and as such very fast. This step is then used to compute small position dependent bounding-boxes in 3D space which enclose the underlying object. The next step is a more expensive window based matching, within the volume of these bounding-boxes. This is limited to a number of regions around 'promising' feature points. Global regularisation is obtained by a graph cut. Speed results here from limiting the number of feature points. In a third step the interpolation is 'pre-rendered' and simultaneously evaluated on a per pixel basis. This is done by computing a Birchfield dissimilarity measure on the GPU. Per pixel parallelised operations keep computational cost low. Finally the bad interpolated parts are 'patched'. This per pixel correction yields the final interpolated view at the finest level. Here we also deal explicitly with opacity at the borders of the foreground object.

[1]  Vladimir Kolmogorov,et al.  An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[4]  Jonathan M. Garibaldi,et al.  Real-Time Correlation-Based Stereo Vision with Reduced Border Errors , 2002, International Journal of Computer Vision.

[5]  Luc Van Gool,et al.  Extended view interpolation by parallel use of the GPU and the CPU , 2005 .

[6]  Václav Hlavác,et al.  Rendering real-world objects using view interpolation , 1995, Proceedings of IEEE International Conference on Computer Vision.

[7]  Steven M. Seitz,et al.  Physically-valid view synthesis by image interpolation , 1995, Proceedings IEEE Workshop on Representation of Visual Scenes (In Conjunction with ICCV'95).

[8]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[9]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[10]  Ruigang Yang,et al.  Multi-resolution real-time stereo on commodity graphics hardware , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[11]  S. Birchfiled.  A Pixel Dissimilarity Measure That Is Insensitive to Image Sampling , 1998 .

[12]  Larry H. Matthies,et al.  Enhanced real-time stereo using bilateral filtering , 2004, Proceedings. 2nd International Symposium on 3D Data Processing, Visualization and Transmission, 2004. 3DPVT 2004..

[13]  Andrew Blake,et al.  Gaze manipulation for one-to-one teleconferencing , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[15]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[16]  Hans-Peter Seidel,et al.  On‐the‐Fly Processing of Generalized Lumigraphs , 2001, Comput. Graph. Forum.

[17]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[18]  Luc Van Gool,et al.  Cinematographic rules applied to a camera network , 2004 .

[19]  Til Aach,et al.  Illumination-invariant change detection , 2000, 4th IEEE Southwest Symposium on Image Analysis and Interpretation.

[20]  Andrew Blake,et al.  The SPS algorithm: patching figural continuity and transparency by Split-Patch Search , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[21]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[22]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.