Panoramic Image Mosaics

This paper presents some techniques for constructing panoramic image mosaics from sequences of images. Our mosaic representation associates a transformation matrix with each input image, rather than explicitly projecting all of the images onto a common surface (e.g., a cylinder). In particular, to construct a full view panorama, we introduce a rotational mosaic representation that associates a rotation matrix (and optionally a focal length) with each input image. A patch-based alignment algorithm is developed to quickly align two images given motion models. Techniques for estimating and refining camera focal lengths are also presented. In order to reduce accumulated registration errors, we apply global alignment (block adjustment) to the whole sequence of images, which results in an optimally registered image mosaic. To compensate for small amounts of motion parallax introduced by translations of the camera and other unmodeled distortions, we develop a local alignment (deghosting) technique which warps each image based on the results of pairwise local image registrations. By combining both global and local alignment, we significantly improve the quality of our image mosaics, thereby enabling the creation of full view panoramic mosaics with hand-held cameras. We also present an inverse texture mapping algorithm for efficiently extracting environment maps from our panoramic image mosaics. By mapping the mosaic onto an arbitrary texture-mapped polyhedron surrounding the origin, we can explore the virtual environment using standard 3D graphics viewers and hardware without requiring special-purpose players.

[1]  Kenneth Turkowski,et al.  Creating image-based VR using a self-calibrating fisheye lens , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[2]  Katsushi Ikeuchi,et al.  Object shape and reflectance modeling from observation , 1997, SIGGRAPH.

[3]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[4]  Ming-Chao Chiang,et al.  Efficient image warping and super-resolution , 1996, Proceedings Third IEEE Workshop on Applications of Computer Vision. WACV'96.

[5]  Shree K. Nayar,et al.  Catadioptric omnidirectional camera , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[6]  Richard Szeliski,et al.  Direct methods for visual scene reconstruction , 1995, Proceedings IEEE Workshop on Representation of Visual Scenes (In Conjunction with ICCV'95).

[7]  Kristin J. Dana,et al.  Real-time scene stabilization and mosaic construction , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[8]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[9]  P. Anandan,et al.  Hierarchical Model-Based Motion Estimation , 1992, ECCV.

[10]  Shenchang Eric Chen,et al.  QuickTime VR: an image-based approach to virtual environment navigation , 1995, SIGGRAPH.

[11]  Pascal Fua,et al.  Using 3-Dimensional Meshes To Combine Image-Based and Geometry-Based Constraints , 1994, ECCV.

[12]  Ned Greene,et al.  Environment Mapping and Other Applications of World Projections , 1986, IEEE Computer Graphics and Applications.

[13]  Rakesh Kumar,et al.  Shape Recovery from Multiple Views: A Parallax Based Approach , 1994 .

[14]  Sing Bing Kang,et al.  Survey of image-based rendering techniques , 1998, Electronic Imaging.

[15]  Richard Szeliski,et al.  A layered video object coding system using sprite and affine motion model , 1997, IEEE Trans. Circuits Syst. Video Technol..

[16]  Richard Szeliski,et al.  Creating full view panoramic image mosaics and texture-mapped models , 1997, International Conference on Computer Graphics and Interactive Techniques.

[17]  Harpreet S. Sawhney,et al.  True multi-image alignment and its application to mosaicing and lens distortion correction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[18]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[19]  Donald P. Greenberg,et al.  Improved Computational Methods for Ray Tracing , 1984, TOGS.

[20]  E. Dubois,et al.  Digital picture processing , 1985, Proceedings of the IEEE.

[21]  L. Quam.  Hierarchical warp stereo , 1987 .

[22]  Qi Tian,et al.  Algorithms for subpixel registration , 1986 .

[23]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[24]  Richard I. Hartley.  Self-Calibration from Multiple Views with a Rotating Camera , 1994, ECCV.

[25]  Richard Szeliski,et al.  Video mosaics for virtual environments , 1996, IEEE Computer Graphics and Applications.

[26]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[27]  Michal Irani,et al.  Representation of scenes from collections of images , 1995, Proceedings IEEE Workshop on Representation of Visual Scenes (In Conjunction with ICCV'95).

[28]  Harpreet S. Sawhney.  Simplifying motion and structure analysis using planar parallax and image warping , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[29]  P. Anandan,et al.  Video compression using mosaic representations , 1995, Signal Process. Image Commun..

[30]  George Wolberg,et al.  Digital image warping , 1990 .

[31]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[32]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[33]  P. Anandan,et al.  Mosaic based representations of video sequences and their applications , 1995, Proceedings of IEEE International Conference on Computer Vision.

[34]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[35]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[36]  H. Malde.  Panoramic photographs. , 1983, American scientist.

[37]  P. Danielsson.  Euclidean distance mapping , 1980 .

[38]  Paul S. Heckbert,et al.  Fundamentals of Texture Mapping and Image Warping , 1989 .

[39]  Richard Szeliski,et al.  Image mosaicing for tele-reality applications , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[40]  Sing Bing Kang,et al.  Characterization of errors in compositing panoramic images , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[41]  Harpreet S. Sawhney,et al.  Compact Representations of Videos Through Dominant and Multiple Motion Estimation , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Lance Williams,et al.  Pyramidal parametrics , 1983, SIGGRAPH.

[43]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[44]  Didier Le Gall,et al.  MPEG: a video compression standard for multimedia applications , 1991, CACM.

[45]  Shmuel Peleg,et al.  Panoramic mosaics by manifold projection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[46]  Gideon P. Stein.  Accurate internal camera calibration using rotation, with analysis of sources of error , 1995, Proceedings of IEEE International Conference on Computer Vision.