Multi-view image-based rendering and modeling

While both work with images, Computer Graphics and Computer Vision are different fields. Computer graphics starts with the creation of geometric models and produces linage sequences. Computer vision starts with images or image sequences and produces interpretations including geometric models. Lately, there has been a meeting in the middle, the goal being to create photorealistic images with the help of accurate models recovered from projections of real objects. This convergence has produced a new subfield called Image-Based Rendering and Modeling (IBRM). In this research, the geometric aspects of IBRM are studied. The case of two views is studied first. New algorithms are developed that automatically align the input images, match them and reconstruct 3-D surfaces in Euclidean space. The matching algorithm is designed to cope with complex shapes such as human faces. The reconstruction algorithms are then generalized to the multi-view case, based on a stratified framework. At the root of the stratification is a novel projective reconstruction algorithm that produces a non-metric structure of the scene. An image-based rendering system is implemented, that directly uses this non-metric structure to synthesize images from novel viewpoints. On top of that, a novel algorithm is developed to upgrade the non-metric structures into metric ones. Intrinsic and extrinsic camera parameters are obtained at the same time. One potential application, showcased in this thesis, is to blend computer graphics objects into real images with correct perspective and occlusion. The proposed theory and the associated algorithms lay down the groundwork for future development in multi-view image based rendering and modeling.

[1]  Qian Chen,et al.  A volumetric stereo matching method: application to image-based modeling , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[2]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[3]  Takeo Kanade,et al.  Virtual ized reality: constructing time-varying virtual worlds from real world events , 1997 .

[4]  Jake K. Aggarwal,et al.  Structure from stereo-a review , 1989, IEEE Trans. Syst. Man Cybern..

[5]  J. Oliensis,et al.  Multiframe structure from motion in perspective , 1995, Proceedings IEEE Workshop on Representation of Visual Scenes (In Conjunction with ICCV'95).

[6]  Mi-Suen Lee,et al.  Tensor voting for salient feature inference in computer vision , 1998 .

[7]  Long Quan,et al.  Relative 3D Reconstruction Using Multiple Uncalibrated Images , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Mi-Suen Lee,et al.  Inferring segmented surface description from stereo data , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[9]  Takeo Kanade,et al.  A Paraperspective Factorization Method for Shape and Motion Recovery , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[11]  Paul A. Beardsley,et al.  3D Model Acquisition from Extended Image Sequences , 1996, ECCV.

[12]  O. Faugeras.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[13]  Rangachar Kasturi,et al.  Machine vision , 1995 .

[14]  Zhengyou Zhang,et al.  A New Multistage Approach to Motion and Structure Estimation: From Essential Parameters to Euclidean , 1996 .

[15]  Davi Geiger,et al.  Occlusions, Discontinuities, and Epipolar Lines in Stereo , 1998, ECCV.

[16]  John Oliensis,et al.  A Critique of Structure-from-Motion Algorithms , 2000, Comput. Vis. Image Underst..

[17]  Thierry Viéville,et al.  Canonic Representations for the Geometries of Multiple Projective Views , 1994, ECCV.

[18]  Henrique S. Malvar,et al.  Making Faces , 2019, Topoi.

[19]  Gérard G. Medioni,et al.  3-D Surface Description from Binocular Stereo , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Stéphane Christy,et al.  Euclidean Shape and Motion from Multiple Perspective Views by Affine Iterations , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Ramesh C. Jain,et al.  Reality modeling and visualization from multiple video sequences , 1996, IEEE Computer Graphics and Applications.

[22]  Ingemar J. Cox,et al.  A maximum-flow formulation of the N-camera stereo correspondence problem , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[23]  Peter F. Sturm,et al.  A Factorization Based Algorithm for Multi-Image Projective Structure and Motion , 1996, ECCV.

[24]  J. Alison Noble,et al.  Finding Corners , 1988, Alvey Vision Conference.

[25]  Gérard G. Medioni,et al.  A semi-automatic system to infer complex 3-D shapes from photographs , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[26]  O. Faugeras,et al.  3-D Reconstruction of Urban Scenes from Sequences of Images , 1995 .

[27]  Mi-Suen Lee,et al.  A Computational Framework for Segmentation and Grouping , 2000 .

[28]  Rajiv Gupta,et al.  Computing matched-epipolar projections , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Ruzena Bajcsy,et al.  Fish-scales: representing fuzzy manifolds , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[30]  Bill Triggs,et al.  Autocalibration and the absolute quadric , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[31]  Narendra Ahuja,et al.  Dense shape and motion from region correspondences by factorization , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[32]  Alessandro Verri,et al.  Against Quantitative Optical Flow , 1987 .

[33]  William E. Lorensen,et al.  Marching cubes: a high resolution 3D surface construction algorithm , 1996 .

[34]  Paul Debevec,et al.  Modeling and Rendering Architecture from Photographs , 1996, SIGGRAPH 1996.

[35]  Pascal Fua,et al.  Reconstructing complex surfaces from multiple stereo views , 1995, Proceedings of IEEE International Conference on Computer Vision.

[36]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[37]  B. Julesz,et al.  A disparity gradient limit for binocular fusion. , 1980, Science.

[38]  Koji Koyamada,et al.  Automatic Isosurface Propagation Using an Extrema Graph and Sorted Boundary Cell Lists , 1995, IEEE Trans. Vis. Comput. Graph..

[39]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[40]  Luc Van Gool,et al.  A stratified approach to metric self-calibration , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[41]  Gérard G. Medioni,et al.  Inference of Integrated Surface, Curve, and Junction Descriptions From Sparse 3D Data , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  David Salesin,et al.  Synthesizing realistic facial expressions from photographs , 1998, SIGGRAPH.

[43]  Michael Werman,et al.  Trilinearity of three perspective views and its associated tensor , 1995, Proceedings of IEEE International Conference on Computer Vision.

[44]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[45]  Wilfried Brauer,et al.  Intensity- and Gradient-Based Stereo Matching Using Hierarchical Gaussian Basis Functions , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  Ingemar J. Cox,et al.  A Maximum Likelihood Stereo Algorithm , 1996, Comput. Vis. Image Underst..

[47]  Richard Szeliski,et al.  Creating full view panoramic image mosaics and environment maps , 1997, SIGGRAPH.

[48]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[49]  Olivier D. Faugeras,et al.  A Comparison of Projective Reconstruction Methods for Pairs of Views , 1997, Comput. Vis. Image Underst..

[50]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[51]  John Oliensis,et al.  An Experimental Study of Projective Structure From Motion , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  Takeo Kanade,et al.  Virtualized reality: constructing time-varying virtual worlds from real world events , 1997, Proceedings. Visualization '97 (Cat. No. 97CB36155).

[53]  Amnon Shashua,et al.  Novel view synthesis in tensor space , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[54]  S. P. Mudur,et al.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[55]  Richard I. Hartley,et al.  In Defense of the Eight-Point Algorithm , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[56]  S. Bougnoux,et al.  From projective to Euclidean space under any practical situation, a criticism of self-calibration , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[57]  Ramakant Nevatia,et al.  Recovering LSHGCs and SHGCs from stereo , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[58]  Shenchang Eric Chen,et al.  QuickTime VR: an image-based approach to virtual environment navigation , 1995, SIGGRAPH.

[59]  Narendra Ahuja,et al.  Surfaces from Stereo: Integrating Feature Matching, Disparity Estimation, and Contour Detection , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[60]  Kiriakos N. Kutulakos,et al.  Calibration-Free Augmented Reality , 1998, IEEE Trans. Vis. Comput. Graph..

[61]  Qian Chen,et al.  Image synthesis from a sparse set of views , 1997, Proceedings. Visualization '97 (Cat. No. 97CB36155).

[62]  R. Hartley.  Triangulation, Computer Vision and Image Understanding , 1997 .

[63]  Anders Heyden,et al.  Recursive structure and motion from image sequences using shape and depth spaces , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[64]  Olivier D. Faugeras,et al.  What can be seen in three dimensions with an uncalibrated stereo rig , 1992, ECCV.

[65]  O. D. Faugeras,et al.  Camera Self-Calibration: Theory and Experiments , 1992, ECCV.

[66]  Andrew Zisserman,et al.  Robust parameterization and computation of the trifocal tensor , 1997, Image Vis. Comput..

[67]  P R Wolf,et al.  Elements of Photogrammetry , 1983 .

[68]  Richard Szeliski,et al.  Video mosaics for virtual environments , 1996, IEEE Computer Graphics and Applications.

[69]  Olivier D. Faugeras,et al.  On the geometry and algebra of the point and line correspondences between N images , 1995, Proceedings of IEEE International Conference on Computer Vision.

[70]  Qian Chen,et al.  Efficient iterative solution to M-view projective reconstruction problem , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[71]  Alex Pentland,et al.  Recursive Estimation of Motion, Structure, and Focal Length , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[72]  Long Quan,et al.  Relative 3D Reconstruction Using Multiple Uncalibrated Images , 1995, Int. J. Robotics Res..

[73]  David W. Jacobs,et al.  Linear fitting with missing data: applications to structure-from-motion and to characterizing intensity images , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[74]  Rachid Deriche,et al.  Robust Recovery of the Epipolar Geometry for an Uncalibrated Stereo Rig , 1994, ECCV.

[75]  Mi-Suen Lee,et al.  Synthesizing Novel Views from Unregistered 2‐D Images , 1997, Comput. Graph. Forum.

[76]  Amnon Shashua,et al.  Projective depth: A geometric invariant for 3D reconstruction from two perspective/orthographic views and for visual recognition , 1993, 1993 (4th) International Conference on Computer Vision.

[77]  Rajiv Gupta,et al.  Stereo from uncalibrated cameras , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[78]  Richard I. Hartley,et al.  Linear self-calibration of a rotating and zooming camera , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[79]  H. C. Longuet-Higgins,et al.  A computer algorithm for reconstructing a scene from two projections , 1981, Nature.

[80]  Reinhard Koch,et al.  Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[81]  B. Triggs,et al.  Projective Geometry for Image Analysis , 1996 .

[82]  Bill Triggs,et al.  Matching constraints and the joint image , 1995, Proceedings of IEEE International Conference on Computer Vision.

[83]  Takeo Kanade,et al.  Stereo by Intra- and Inter-Scanline Search Using Dynamic Programming , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[84]  Martial Hebert,et al.  Weakly-calibrated stereo perception for rover navigation , 1995, Proceedings of IEEE International Conference on Computer Vision.

[85]  Leila De Floriani,et al.  An on-line algorithm for constrained Delaunay triangulation , 1992, CVGIP Graph. Model. Image Process..