Stereo for Image-Based Rendering using Image Over-Segmentation

In this paper, we propose a stereo method specifically designed for image-based rendering. For effective image-based rendering, the interpolated views need only be visually plausible. The implication is that the extracted depths do not need to be correct, as long as the recovered views appear to be correct. Our stereo algorithm relies on over-segmenting the source images. Computing match values over entire segments rather than single pixels provides robustness to noise and intensity bias. Color-based segmentation also helps to more precisely delineate object boundaries, which is important for reducing boundary artifacts in synthesized views. The depths of the segments for each image are computed using loopy belief propagation within a Markov Random Field framework. Neighboring MRFs are used for occlusion reasoning and ensuring that neighboring depth maps are consistent. We tested our stereo algorithm on several stereo pairs from the Middlebury data set, and show rendering results based on two of these data sets. We also show results for video-based rendering.

[1]  Takeo Kanade,et al.  Stereo by Intra- and Inter-Scanline Search Using Dynamic Programming , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  조규상,et al.  Stereo 영상의 Matching Algorithm , 1987 .

[3]  Jitendra Malik,et al.  Scale-Space and Edge Detection Using Anisotropic Diffusion , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  John E. Howland,et al.  Computer graphics , 1990, IEEE Potentials.

[5]  Takeo Kanade,et al.  A multiple-baseline stereo , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[6]  Edward H. Adelson,et al.  Representing moving images with layers , 1994, IEEE Trans. Image Process..

[7]  Takeo Kanade,et al.  A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Aaron F. Bobick,et al.  Disparity-Space Images and Large Occlusion Stereo , 1994, ECCV.

[9]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[10]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[11]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[12]  Robert T. Collins,et al.  A space-sweep approach to true multi-image matching , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[13]  Linda G. Shapiro,et al.  View-base Rendering: Visualizing Real Objects from Scanned Range and Color Data , 1997, Rendering Techniques.

[14]  Photorealistic scene reconstruction by voxel coloring , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[15]  Cordelia Schmid,et al.  Automatic line matching across views , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[16]  K. Chan,et al.  Energy Minimization and Relaxation Labeling , 1997 .

[17]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[18]  Richard Szeliski,et al.  A layered approach to stereo reconstruction , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[19]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[20]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[21]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[22]  Sing Bing Kang,et al.  Survey of image-based rendering techniques , 1998, Electronic Imaging.

[23]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[24]  Jeremy S. De Bonet,et al.  Poxels: Probabilistic Voxelized Volume Reconstruction , 1999 .

[25]  Reinhard Koch,et al.  Plenoptic Modeling and Rendering from Image Sequences Taken by Hand-Held Camera , 1999, DAGM-Symposium.

[26]  Richard Szeliski,et al.  Prediction error as a quality metric for motion and stereo , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[27]  Richard Szeliski,et al.  The geometry-image representation tradeoff for rendering , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[28]  Joon Hee Han,et al.  Contour Matching Using Epipolar Geometry , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Takeo Kanade,et al.  A Cooperative Algorithm for Stereo Matching and Occlusion Detection , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  Leonard McMillan,et al.  Dynamically reparameterized light fields , 2000, SIGGRAPH.

[31]  David Salesin,et al.  A Bayesian approach to digital matting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[32]  A Global Matching Framework for Stereo Computation , 2001, ICCV.

[33]  Richard Szeliski,et al.  Handling occlusions in dense multi-view stereo , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[34]  A Probabilistic Framework for Space Carving , 2001, ICCV.

[35]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[36]  Harry Shum,et al.  Simultaneous separation and depth recovery of specular reflections , 2002, eccv 2002.

[37]  Vladimir Kolmogorov,et al.  Multi-camera Scene Reconstruction via Graph Cuts , 2002, ECCV.

[38]  Andrew W. Fitzgibbon,et al.  Bayesian Estimation of Layers from Multiple Images , 2002, ECCV.

[39]  Stephen Lin,et al.  Diffuse-Specular Separation and Depth Recovery from Image Sequences , 2002, ECCV.

[40]  Richard Szeliski,et al.  On the Motion and Appearance of Specularities in Image Sequences , 2002, ECCV.

[41]  Nanning Zheng,et al.  Stereo Matching Using Belief Propagation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Vladimir Kolmogorov,et al.  Visual correspondence using energy minimization and mutual information , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[43]  Richard Szeliski,et al.  Stereo matching with reflections and translucency , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[44]  Andrea Fusiello.  Image-based Rendering * , 2003 .

[45]  Masatoshi Okutomi,et al.  A Simple Stereo Algorithm to Recover Precise Object Boundaries and Smooth Surfaces , 2004, International Journal of Computer Vision.

[46]  Han Wang,et al.  Minimization of MRF Energy with Relaxation Labeling , 2004, Journal of Mathematical Imaging and Vision.

[47]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[48]  George Drettakis,et al.  Volumetric reconstruction and interactive rendering of trees from photographs , 2004, ACM Trans. Graph..

[49]  Richard Szeliski,et al.  Stereo Matching with Transparency and Matting , 1999, International Journal of Computer Vision.

[50]  Steven M. Seitz,et al.  Photorealistic Scene Reconstruction by Voxel Coloring , 1997, International Journal of Computer Vision.

[51]  Richard Szeliski,et al.  Boundary Matting for View Synthesis , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[52]  George Drettakis,et al.  Volumetric reconstruction and interactive rendering of trees from photographs , 2004, SIGGRAPH 2004.

[53]  Margrit Gelautz,et al.  A layered stereo algorithm using image segmentation and global visibility constraints , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[54]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, ACM Trans. Graph..

[55]  Jian Sun,et al.  Symmetric stereo matching for occlusion handling , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[56]  Takeo Kanade,et al.  Three-dimensional scene flow , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Harry Shum,et al.  Image-based rendering , 2006, Found. Trends Comput. Graph. Vis..

[58]  R. J. Koshel,et al.  The Layered Approach , 2007 .