Modeling and Rendering Architecture from Photographs

Imagine visiting your favorite place, taking a few pictures, and then turning those pictures into a photorealisic three-dimensional computer model. The work presented in this thesis combines techniques from computer vision and computer graphics to make this possible. The applications range from architectural planning and archaeological reconstructions to virtual environments and cinematic special effects. This thesis presents an approach for modeling and rendering existing architectural scenes from sparse sets of still photographs. The modeling approach, which combines both geometry-based and image-based techniques, has two components. The first component is an interactive photogrammetric modeling method which facilitates the recovery of the basic geometry of the photographed scene. The photogrammetric modeling approach is effective, convenient, and robust because it exploits the constraints that are characteristic of architectural scenes. The second component is a model-based stereo algorithm, which recovers how the real scene deviates from the basic model. By making use of the model, this new technique robustly recovers accurate depth from widely-spaced image pairs. Consequently, this approach can model large architectural environments with far fewer photographs than current image-based modeling approaches. For producing renderings, this thesis presents view-dependent texture mapping, a method of compositing multiple views of a scene that better simulates geometric detail on basic models. This approach can be used to recover models for use in either geometry-based or image-based rendering systems. This work presents results that demonstrate the approach's ability to create realistic renderings of architectural scenes from viewpoints far from the original photographs. This thesis concludes with a presentation of how these modeling and rendering techniques were used to create the interactive art installation Rouen Revisited, presented at the SIGGRAPH '96 art show.

[1]  Lance Williams,et al.  Casting curved shadows on curved surfaces , 1978, SIGGRAPH.

[2]  D Marr,et al.  A computational theory of human stereo vision. , 1979, Proceedings of the Royal Society of London. Series B, Biological sciences.

[3]  S. Ullman,et al.  The interpretation of visual motion , 1977 .

[4]  Thomas O. Binford,et al.  Depth from Edge and Intensity Based Stereo , 1981, IJCAI.

[5]  H. C. Longuet-Higgins,et al.  A computer algorithm for reconstructing a scene from two projections , 1981, Nature.

[6]  H. K. Nishihara,et al.  Practical Real-Time Imaging Stereo Matcher , 1984 .

[7]  J P Frisby,et al.  PMF: A Stereo Correspondence Algorithm Using a Disparity Gradient Limit , 1985, Perception.

[8]  Paul S. Heckbert,et al.  Survey of Texture Mapping , 1986, IEEE Computer Graphics and Applications.

[9]  Andrew Blake,et al.  Visual Reconstruction , 1987, Deep Learning for EEG-Based Brain–Computer Interfaces.

[10]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[11]  Yoshiaki Shirai,et al.  Three-Dimensional Computer Vision , 1987, Symbolic Computation.

[12]  Jitendra Malik,et al.  Scale-Space and Edge Detection Using Anisotropic Diffusion , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Joachim Pissarro,et al.  Monet's cathedral : Rouen, 1892-1894 , 1990 .

[14]  Carlo H. Séquin,et al.  Visibility preprocessing for interactive walkthroughs , 1991, SIGGRAPH.

[15]  David J. Fleet,et al.  Phase-based disparity measurement , 1991, CVGIP Image Underst..

[16]  Wen-Hsiang Tsai,et al.  Camera Calibration by Vanishing Lines for 3-D Computer Vision , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Jitendra Malik,et al.  A Computational Framework for Determining Stereo Correspondence from a Set of Linear Spatial Filters , 1991, ECCV.

[18]  Denis Laurendeau,et al.  Multi-resolution surface modeling from multiple range views , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[19]  O. D. Faugeras,et al.  Camera Self-Calibration: Theory and Experiments , 1992, ECCV.

[20]  Mark Segal,et al.  Fast shadows and lighting effects using texture mapping , 1992, SIGGRAPH.

[21]  Carlo H. Séquin,et al.  Adaptive display algorithm for interactive frame rates during visualization of complex virtual environments , 1993, SIGGRAPH.

[22]  Richard Szeliski,et al.  Robust Shape Recovery from Occluding Contours Using a Linear Smoother , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[24]  Harpreet S. Sawhney.  Simplifying motion and structure analysis using planar parallax and image warping , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[25]  C. J. Taylor,et al.  Minimization on the Lie Group SO(3) and Related Manifolds , 1994 .

[26]  Seth J. Teller,et al.  Partitioning and ordering large radiosity computations , 1994, SIGGRAPH.

[27]  Tony DeRose,et al.  Piecewise smooth surface reconstruction , 1994, SIGGRAPH.

[28]  Richard Szeliski,et al.  Image mosaicing for tele-reality applications , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[29]  Ramin Zabih,et al.  Non-parametric Local Transforms for Computing Visual Correspondence , 1994, ECCV.

[30]  Marc Levoy,et al.  Zippered polygon meshes from range images , 1994, SIGGRAPH.

[31]  Olivier D. Faugeras,et al.  3-D scene representation as a collection of images , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[32]  Mark James,et al.  Synthetic topiary , 1994, SIGGRAPH.

[33]  Robin Liggett,et al.  An environment for real-time urban simulation , 1995, I3D '95.

[34]  V. Michael Bove,et al.  Semiautomatic 3-D model extraction from uncalibrated 2-D camera views , 1995 .

[35]  William A. Barrett,et al.  Intelligent scissors for image composition , 1995, SIGGRAPH.

[36]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[37]  J. Bergen,et al.  Pyramid-based texture analysis/synthesis , 1995, Proceedings., International Conference on Image Processing.

[38]  Harry Shum,et al.  An integral approach to free-formed object modeling , 1995, Proceedings of IEEE International Conference on Computer Vision.

[39]  O. Faugeras,et al.  3-D Reconstruction of Urban Scenes from Sequences of Images , 1995 .

[40]  David J. Kriegman,et al.  Structure and Motion from Line Segments in Multiple Images , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  Alex Pentland,et al.  Recursive Estimation of Motion, Structure, and Focal Length , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Shree K. Nayar,et al.  Telecentric Optics for Computational Vision , 1996, ECCV.

[43]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[44]  Barbara J. Meier.  Painterly rendering for animation , 1996, SIGGRAPH.

[45]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[46]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[47]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[48]  Richard Szeliski,et al.  Stereo matching with non-linear diffusion , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[49]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[50]  Daniel Scharstein,et al.  Stereo vision for view synthesis , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[51]  Katsushi Ikeuchi,et al.  Object shape and reflectance modeling from observation , 1997, SIGGRAPH.

[52]  Linda G. Shapiro,et al.  View-base Rendering: Visualizing Real Objects from Scanned Range and Color Data , 1997, Rendering Techniques.

[53]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[54]  Hongbin Zha,et al.  Regularization-based 3D object modeling from multiple range images , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[55]  Paul E. Debevec,et al.  Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 1998, SIGGRAPH '08.