Visual Modeling with a Hand-Held Camera

In this paper a complete system to build visual models from camera images is presented. The system can deal with uncalibrated image sequences acquired with a hand-held camera. Based on tracked or matched features the relations between multiple views are computed. From this both the structure of the scene and the motion of the camera are retrieved. The ambiguity on the reconstruction is restricted from projective to metric through self-calibration. A flexible multi-view stereo matching scheme is used to obtain a dense estimation of the surface geometry. From the computed data different types of visual models are constructed. Besides the traditional geometry- and image-based approaches, a combined approach with view-dependent geometry and texture is presented. As an application fusion of real and virtual scenes is also shown.

[1]  Soren W. Henriksen,et al.  Manual of photogrammetry , 1980 .

[2]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[3]  T. J. Lauterborn.  American Society Of Photogrammetry , 1984 .

[4]  William H. Press,et al.  Numerical recipes in C. The art of scientific computing , 1987 .

[5]  Peter J. Rousseeuw,et al.  Robust regression and outlier detection , 1987 .

[6]  Jorge Stolfi,et al.  Oriented projective geometry , 1987, SCG '87.

[7]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[8]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[9]  Richard I. Hartley,et al.  Chirality , 2004, International Journal of Computer Vision.

[10]  Takeo Kanade,et al.  A multiple-baseline stereo , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[11]  William E. Lorensen,et al.  Decimation of triangle meshes , 1992, SIGGRAPH.

[12]  Olivier D. Faugeras,et al.  What can be seen in three dimensions with an uncalibrated stereo rig , 1992, ECCV.

[13]  Rajiv Gupta,et al.  Stereo from uncalibrated cameras , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[14]  O. D. Faugeras,et al.  Camera Self-Calibration: Theory and Experiments , 1992, ECCV.

[15]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[16]  Andrew Zisserman,et al.  Applications of Invariance in Computer Vision , 1993, Lecture Notes in Computer Science.

[17]  Richard I. Hartley,et al.  Euclidean Reconstruction from Uncalibrated Views , 1993, Applications of Invariance in Computer Vision.

[18]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Reg G. Willson.  Modeling and calibration of automated zoom lenses , 1994, Other Conferences.

[20]  Marc Levoy,et al.  Zippered polygon meshes from range images , 1994, SIGGRAPH.

[21]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[22]  Rachid Deriche,et al.  A Robust Technique for Matching two Uncalibrated Images Through the Recovery of the Unknown Epipolar Geometry , 1995, Artif. Intell..

[23]  Denis Laurendeau,et al.  A General Surface Approach to the Integration of a Set of Range Views , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[25]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[26]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[27]  William E. Lorensen,et al.  Marching cubes: a high resolution 3D surface construction algorithm , 1996 .

[28]  Olivier D. Faugeras,et al.  Oriented Projective Geometry for Computer Vision , 1996, ECCV.

[29]  Ingemar J. Cox,et al.  A Maximum Likelihood Stereo Algorithm , 1996, Comput. Vis. Image Underst..

[30]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[31]  A. Heyden,et al.  Euclidean reconstruction from constant intrinsic parameters , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[32]  Richard I. Hartley,et al.  In Defense of the Eight-Point Algorithm , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  Bill Triggs,et al.  Autocalibration and the absolute quadric , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[34]  Michael Werman,et al.  Multiresolution Textures from Image Sequences , 1997, IEEE Computer Graphics and Applications.

[35]  Lutz Falkenhagen.  Hierarchical Block-Based Disparity Estimation Considering Neighbourhood Constraints , 1997 .

[36]  Cordelia Schmid,et al.  Local Grayvalue Invariants for Image Retrieval , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  R. Hartley.  Triangulation, Computer Vision and Image Understanding , 1997 .

[38]  Peter F. Sturm.  Vision 3D non calibrée : contributions à la reconstruction projective et étude des mouvements critiques pour l'auto-calibrage. (Uncalibrated 3D Vision: Contributions to Projective Reconstruction and Study of the Critical Motions for Self-Calibration) , 1997 .

[39]  Anders Heyden,et al.  Euclidean reconstruction from image sequences with varying and unknown focal length and principal point , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[40]  Reinhard Koch,et al.  Automatische Oberflächenmodellierung starrer dreidimensionaler Objekte aus stereoskopischen Rundum-Ansichten , 1997 .

[41]  Peter F. Sturm,et al.  Critical motion sequences for monocular self-calibration and uncalibrated Euclidean reconstruction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[42]  Highlight and Re ection-Independent Multiresolution Textures from Image Sequences , 1997 .

[43]  Reinhard Koch,et al.  Multi Viewpoint Stereo from Uncalibrated Video Sequences , 1998, ECCV.

[44]  Andrew W. Fitzgibbon,et al.  Maintaining multiple motion model hypotheses over many views to recover matching and structure , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[45]  Harpreet S. Sawhney,et al.  Robust Video Mosaicing through Topology Inference and Local to Global Alignment , 1998, ECCV.

[46]  Reinhard Koch,et al.  Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[47]  Andrew W. Fitzgibbon,et al.  Automatic Camera Recovery for Closed or Open Image Sequences , 1998, ECCV.

[48]  Katsushi Ikeuchi,et al.  Consensus surfaces for modeling 3D objects from multiple range images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[49]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[50]  Reinhard Koch,et al.  Calibration of hand-held camera sequences for plenoptic modeling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[51]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.

[52]  Fredrik Kahl.  Critical motions and ambiguous Euclidean reconstructions in auto-calibration , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[53]  Peter F. Sturm,et al.  Critical motion sequences for the self-calibration of cameras and stereo systems with variable focal length , 1999, Image Vis. Comput..

[54]  M. Pollefeys.  Self-calibration and metric 3d reconstruction from uncalibrated image sequences , 1999 .

[55]  Luc Van Gool,et al.  Stratified Self-Calibration with the Modulus Constraint , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[56]  Reinhard Koch,et al.  Plenoptic Modeling and Rendering from Image Sequences Taken by Hand-Held Camera , 1999, DAGM-Symposium.

[57]  Reinhard Koch,et al.  A simple and efficient rectification method for general motion , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[58]  O. Faugeras,et al.  The Geometry of Multiple Images , 1999 .

[59]  Reinhard Koch,et al.  A Geometric Approach to Lightfield Calibration , 1999, CAIP.

[60]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[61]  Peter Meer,et al.  A general method for Errors-in-Variables problems in computer vision , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[62]  Maarten Vergauwen,et al.  Augmented Reality Using Uncalibrated Video Sequences , 2000, SMILE.

[63]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[64]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[65]  Olivier D. Faugeras,et al.  The geometry of multiple images - the laws that govern the formation of multiple images of a scene and some of their applications , 2001 .

[66]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[67]  Jiri Matas,et al.  Randomized RANSAC with T(d, d) test , 2002, BMVC.

[68]  Luc Van Gool,et al.  Surviving Dominant Planes in Uncalibrated Structure and Motion Recovery , 2002, ECCV.

[69]  Reinhard Koch,et al.  Self-Calibration and Metric Reconstruction Inspite of Varying and Unknown Intrinsic Camera Parameters , 1999, International Journal of Computer Vision.

[70]  Maarten Vergauwen,et al.  A Hierarchical Symmetric Stereo Algorithm Using Dynamic Programming , 2002, International Journal of Computer Vision.

[71]  Paul A. Beardsley,et al.  Sequential Updating of Projective and Affine Structure from Motion , 1997, International Journal of Computer Vision.

[72]  Jiri Matas,et al.  Randomized RANSAC with Td, d test , 2004, Image Vis. Comput..

[73]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[74]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.

[75]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[76]  Marc Pollefeys,et al.  Multiple view geometry , 2005 .

[77]  Alexandru Tupan,et al.  Triangulation , 1997, Comput. Vis. Image Underst..