3D Photography from Photographs and Video Clips

This paper addresses the problem of acquiring realistic visual models of the shape and appearance of complex three-dimensional (3D) scenes from collec-tions of images, a process dubbed 3D photography. We focus on three instances of this problem: (1) the image-based construction of projective visual hulls of complex surfaces from weakly-calibrated photographs; (2) the automated matching and registration of photographs of textured surfaces using affine-invariant patches and their geometric relationships; and (3) an approach to projective motion analysis and self-calibration explicitly accounting for natural camera constraints such as zero skew and capable of handling large numbers of images in an efficient and uniform manner. We also briefly discuss some relat-ed applications of oriented differential projective geometry to computer vision problems, including the determination of the ordering of rim segments in pro-jective visual hull computation, and a purely projective proof of Koenderink's famous characterization of the local shape of visual contours.

[1]  Bruce G. Baumgart,et al.  Geometric modeling for computer vision. , 1974 .

[2]  Hans P. Moravec.  Towards Automatic Visual Obstacle Avoidance , 1977, IJCAI.

[3]  R. Kelly,et al.  The Gestalt Photomapping System , 1977 .

[4]  S. Ullman,et al.  The interpretation of visual motion , 1977 .

[5]  Donald Bernard Gennery,et al.  Modelling the environment of an exploring vehicle by means of stereo vision , 1980 .

[6]  Axel Ruhe,et al.  Algorithms for separable nonlinear least squares problems , 1980 .

[7]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[8]  W E Grimson,et al.  A computer implementation of a theory of human stereo vision. , 1981, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[9]  Joseph Y.-T. Leung,et al.  Efficient algorithms for interval graphs and circular-arc graphs , 1982, Networks.

[10]  H. K. Nishihara,et al.  PRISM: A Practical Mealtime Imaging Stereo Matcher , 1984, Other Conferences.

[11]  J J Koenderink,et al.  What Does the Occluding Contour Tell Us about Solid Shape? , 1984, Perception.

[12]  Takeo Kanade,et al.  Stereo by Intra- and Inter-Scanline Search Using Dynamic Programming , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  J. H. Rieger.  Three-dimensional motion from fixed points of a deforming profile curve. , 1986, Optics letters.

[14]  Alex Pentland,et al.  Perceptual Organization and the Representation of Natural Form , 1986, Artif. Intell..

[15]  Yoshiaki Shirai,et al.  Three-Dimensional Computer Vision , 1987, Symbolic Computation.

[16]  R. Bajcsy,et al.  Three dimensional object representation revisited , 1987 .

[17]  Demetri Terzopoulos,et al.  Constraints on Deformable Models: Recovering 3D Shape and Nonrigid Motion , 1988, Artif. Intell..

[18]  Terrance E. Boult,et al.  Error Of Fit Measures For Recovering Parametric Solids , 1988, [1988 Proceedings] Second International Conference on Computer Vision.

[19]  C. Connolly,et al.  3D scene reconstruction from multiple intensity images , 1989, [1989] Proceedings. Workshop on Interpretation of 3D Scenes.

[20]  Narendra Ahuja,et al.  Octree Generation from Object Silhouettes in Perspective Views , 1990, Comput. Vis. Graph. Image Process..

[21]  Dimitris N. Metaxas,et al.  Dynamic 3D models with local and global deformations: deformable superquadrics , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[22]  Gabriel Taubin,et al.  Estimation of Planar Curves, Surfaces, and Nonplanar Space Curves Defined by Implicit Equations with Applications to Edge and Range Image Segmentation , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  T. Boult,et al.  Factorization-based segmentation of motions , 1991, Proceedings of the IEEE Workshop on Visual Motion.

[24]  C. Tomasi.  Detection and Tracking of Point Features , 1991 .

[25]  J J Koenderink,et al.  Affine structure from motion. , 1991, Journal of the Optical Society of America. A, Optics and image science.

[26]  Dimitris N. Metaxas,et al.  Dynamic 3D Models with Local and Global Deformations: Deformable Superquadrics , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  Olivier D. Faugeras,et al.  What can be seen in three dimensions with an uncalibrated stereo rig , 1992, ECCV.

[28]  Olivier D. Faugeras,et al.  Using Extremal Boundaries for 3-D Object Modeling , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Rajiv Gupta,et al.  Stereo from uncalibrated cameras , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[30]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Narendra Ahuja,et al.  Matching Two Perspective Views , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Daphna Weinshall,et al.  Linear and incremental acquisition of invariant shape models from image sequences , 1993, 1993 (4th) International Conference on Computer Vision.

[33]  Allen R. Hanson,et al.  Site Model Acquisition under the UMass RADIUS Project , 1994 .

[34]  Jean Ponce,et al.  Using Geometric Distance Fits for 3-D Object Modeling and Recognition , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[35]  David J. Kriegman,et al.  Parameterized Families of Polynomials for Bounded Algebraic Curve and Surface Fitting , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Marc Levoy,et al.  Zippered polygon meshes from range images , 1994, SIGGRAPH.

[37]  David B. Cooper,et al.  Describing Complicated Objects by Implicit Polynomials , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Olivier D. Faugeras,et al.  3-D scene representation as a collection of images , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[39]  Amnon Shashua,et al.  Algebraic Functions For Recognition , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Takeo Kanade,et al.  A multi-body factorization method for motion analysis , 1995, Proceedings of IEEE International Conference on Computer Vision.

[41]  Richard I. Hartley,et al.  In defence of the 8-point algorithm , 1995, Proceedings of IEEE International Conference on Computer Vision.

[42]  Steven M. Seitz,et al.  Physically-valid view synthesis by image interpolation , 1995, Proceedings IEEE Workshop on Representation of Visual Scenes (In Conjunction with ICCV'95).

[43]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[44]  Aldo Laurentini,et al.  How Far 3D Shapes Can Be Understood from 2D Silhouettes , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  W. Niem,et al.  Automatic Modelling of 3D Natural Objects from Multiple Views , 1995 .

[46]  Roberto Cipolla,et al.  Motion from the frontier of curved surfaces , 1995, Proceedings of IEEE International Conference on Computer Vision.

[47]  Rachid Deriche,et al.  A Robust Technique for Matching two Uncalibrated Images Through the Recovery of the Unknown Epipolar Geometry , 1995, Artif. Intell..

[48]  Shenchang Eric Chen,et al.  QuickTime VR: an image-based approach to virtual environment navigation , 1995, SIGGRAPH.

[49]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[50]  Michael T. Heath,et al.  Scientific Computing: An Introductory Survey , 1996 .

[51]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[52]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[53]  P. Anandan,et al.  A unified approach to moving object detection in 2D and 3D scenes , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[54]  Edmond Boyer,et al.  Object Models from Contour Sequences , 1996, ECCV.

[55]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[56]  Peter F. Sturm,et al.  A Factorization Based Algorithm for Multi-Image Projective Structure and Motion , 1996, ECCV.

[57]  Bill Triggs,et al.  Factorization methods for projective structure and motion , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[58]  Amnon Shashua,et al.  Novel view synthesis in tensor space , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[59]  Bill Triggs,et al.  Autocalibration and the absolute quadric , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[60]  Linda G. Shapiro,et al.  View-base Rendering: Visualizing Real Objects from Scanned Range and Color Data , 1997, Rendering Techniques.

[61]  Takeo Kanade,et al.  A Paraperspective Factorization Method for Shape and Motion Recovery , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[62]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[63]  Tony Lindeberg,et al.  Shape-adapted smoothing in estimation of 3-D shape cues from affine deformations of local 2-D brightness structure , 1997, Image Vis. Comput..

[64]  Martial Hebert,et al.  Unconstrained registration of large 3D point sets for complex model building , 1998, Proceedings. 1998 IEEE/RSJ International Conference on Intelligent Robots and Systems. Innovations in Theory, Practice and Applications (Cat. No.98CH36190).

[65]  Mei Han,et al.  Interactive construction of 3D models from panoramic mosaics , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[66]  Gavin S. P. Miller,et al.  Lazy Decompression of Surface Light Fields for Precomputed Global Illumination , 1998, Rendering Techniques.

[67]  Kiriakos N. Kutulakos,et al.  Calibration-Free Augmented Reality , 1998, IEEE Trans. Vis. Comput. Graph..

[68]  Andrew Zisserman,et al.  Wide baseline stereo matching , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[69]  Andrew W. Fitzgibbon,et al.  Automatic 3D model acquisition and generation of new images from video sequences , 1998, 9th European Signal Processing Conference (EUSIPCO 1998).

[70]  Reinhard Koch,et al.  Flexible 3D acquisition with a monocular camera , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[71]  Anders Heyden,et al.  Minimal Conditions on Intrinsic Parameters for Euclidean Reconstruction , 1998, ACCV.

[72]  Jean Ponce,et al.  Automatic model construction, pose estimation, and object recognition from photographs using triangular splines , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[73]  Reinhard Koch,et al.  Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[74]  Jitendra Malik,et al.  Tracking people with twists and exponential maps , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[75]  Takeo Kanade,et al.  A unified factorization algorithm for points, line segments and planes with uncertainty models , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[76]  Andrew E. Johnson,et al.  Surface matching for object recognition in complex three-dimensional scenes , 1998, Image Vis. Comput..

[77]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.

[78]  Qian Chen,et al.  Efficient iterative solution to M-view projective reconstruction problem , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[79]  M. Pollefeys.  Self-calibration and metric 3d reconstruction from uncalibrated image sequences , 1999 .

[80]  Katsushi Ikeuchi,et al.  Appearance compression and synthesis based on 3D model for mixed reality , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[81]  O. Faugeras,et al.  The Geometry of Multiple Images , 1999 .

[82]  Takeo Kanade,et al.  Uncertainty Modeling for Optimal Structure from Motion , 1999, Workshop on Vision Algorithms.

[83]  J. Oliensis,et al.  Fast and accurate self-calibration , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[84]  Andrew W. Fitzgibbon,et al.  Parallax geometry of smooth surfaces in multiple views , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[85]  Stefan Carlsson,et al.  Wide Baseline Point Matching Using Affine Invariants Computed from Intensity Profiles , 2000, ECCV.

[86]  David Salesin,et al.  Surface light fields for 3D photography , 2000, SIGGRAPH.

[87]  Mei Han,et al.  Creating 3D models with uncalibrated cameras , 2000, Proceedings Fifth IEEE Workshop on Applications of Computer Vision.

[88]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[89]  Martial Hebert,et al.  Iterative projective reconstruction from multiple views , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[90]  David Edward DiFranco,et al.  Recovery of 3D articulated motion from 2D correspondences , 2000 .

[91]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[92]  Jitendra Malik,et al.  Matching Shapes , 2001, ICCV.

[93]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[94]  Martial Hebert,et al.  Provably-convergent iterative methods for projective structure from motion , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[95]  Jean Ponce,et al.  On computing exact visual hulls of solids bounded by smooth surfaces , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[96]  David J. Kriegman,et al.  Beyond Lambert: reconstructing surfaces with arbitrary BRDFs , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[97]  Paul A. Beardsley,et al.  Image-based 3D photography using opacity hulls , 2002, ACM Trans. Graph..

[98]  Cordelia Schmid,et al.  An Affine Invariant Interest Point Detector , 2002, ECCV.

[99]  Zhengyou Zhang.  The International Symposium on CREST Ikeuchi Project 3 2 Plane Perspectivity : Mapping Between A Rectangle in Space And Its Image , 2003 .

[100]  Cordelia Schmid,et al.  3D object modeling and recognition using affine-invariant patches and multi-view spatial constraints , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[101]  S. Shankar Sastry,et al.  Euclidean Reconstruction and Reprojection Up to Subgroups , 2004, International Journal of Computer Vision.

[102]  Reinhard Koch,et al.  Self-Calibration and Metric Reconstruction Inspite of Varying and Unknown Intrinsic Camera Parameters , 1999, International Journal of Computer Vision.

[103]  Olivier D. Faugeras,et al.  A theory of self-calibration of a moving camera , 1992, International Journal of Computer Vision.

[104]  Nicholas Ayache,et al.  Efficient registration of stereo images by matching graph descriptions of edge segments , 1987, International Journal of Computer Vision.

[105]  Andrew W. Fitzgibbon,et al.  The Problem of Degeneracy in Structure and Motion Recovery from Uncalibrated Image Sequences , 1999, International Journal of Computer Vision.

[106]  Olivier D. Faugeras,et al.  The fundamental matrix: Theory, algorithms, and stability analysis , 2004, International Journal of Computer Vision.

[107]  Minas E. Spetsakis,et al.  Structure from motion using line correspondences , 1990, International Journal of Computer Vision.

[108]  Andrew Blake,et al.  Surface shape from the deformation of apparent contours , 1992, International Journal of Computer Vision.

[109]  Tony Lindeberg,et al.  Feature Detection with Automatic Scale Selection , 1998, International Journal of Computer Vision.

[110]  Narendra Ahuja,et al.  Structure and Motion Estimation from Dynamic Silhouettes under Perspective Projection , 1999, International Journal of Computer Vision.

[111]  Yakup Genc,et al.  Image-Based Rendering Using Parameterized Image Varieties , 2004, International Journal of Computer Vision.

[112]  Richard I. Hartley,et al.  Lines and Points in Three Views and the Trifocal Tensor , 1997, International Journal of Computer Vision.

[113]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[114]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.