PALM: portable sensor-augmented vision system for large-scene modeling

We propose PALM-a portable sensor-augmented vision system for large-scene modeling. The system solves the problem of recovering large structures in arbitrary scenes from video streams taken by a sensor-augmented camera. Central to the solution method is the use of multiple constraints derived from GPS measurements, camera orientation sensor readings, and image features. The knowledge of camera orientation enhances computational efficiency by making a linear formulation of perspective ray constraints possible. The overall shape is constructed by merging smaller shape segments. Shape merging errors are minimized using the concept of shape hierarchy, which is realized through a "landmarking" technique. The features of the system include its use of a small number of images and feature points, its portability, and its low cost interface for synchronizing sensor measurements with the video stream. Example reconstructions of a football stadium and two large buildings are presented and these results are compared with the ground truth.

[1]  William H. Press,et al.  Numerical Recipes in C, 2nd Edition , 1992 .

[2]  Richard I. Hartley,et al.  In defence of the 8-point algorithm , 1995, Proceedings of IEEE International Conference on Computer Vision.

[3]  Richard Hartley.  Camera Calibration Using Line Correspondences , 2001 .

[4]  Amnon Shashua,et al.  Novel view synthesis in tensor space , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[5]  Seth J. Teller,et al.  Acquisition of a large pose-mosaic dataset , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[6]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[7]  A. Gualtierotti.  H. L. Van Trees, Detection, Estimation, and Modulation Theory, , 1976 .

[8]  Rajiv Gupta,et al.  Stereo from uncalibrated cameras , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[9]  Anders Heyden,et al.  Euclidean reconstruction from image sequences with varying and unknown focal length and principal point , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[10]  Alex Pentland,et al.  Recursive Estimation of Motion, Structure, and Focal Length , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  H. C. Longuet-Higgins,et al.  A computer algorithm for reconstructing a scene from two projections , 1981, Nature.

[12]  Wolfgang Spohn,et al.  The Representation of , 1986 .

[13]  Long Quan,et al.  Relative 3D Reconstruction Using Multiple Uncalibrated Images , 1995, Int. J. Robotics Res..

[14]  Reinhard Koch,et al.  Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[15]  Takeo Kanade,et al.  A unified factorization algorithm for points, line segments and planes with uncertainty models , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[16]  A. W. M. van den Enden,et al.  Discrete Time Signal Processing , 1989 .

[17]  Takeo Kanade,et al.  A Paraperspective Factorization Method for Shape and Motion Recovery , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Narendra Ahuja,et al.  Segmentation and Factorization-Based Motion and Structure Estimation for Long Image Sequences , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using unit quaternions , 1987 .

[20]  Takeo Kanade,et al.  A multi-body factorization method for motion analysis , 1995, Proceedings of IEEE International Conference on Computer Vision.

[21]  Harry Shum,et al.  Motion estimation with quadtree splines , 1995, Proceedings of IEEE International Conference on Computer Vision.

[22]  Takeo Kanade,et al.  A sequential factorization method for recovering shape and motion from image streams , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  O. Faugeras.  Stratification of three-dimensional vision: projective, affine, and metric representations , 1995 .

[24]  Roger Mohr,et al.  Euclidean constraints for uncalibrated reconstruction , 1993, 1993 (4th) International Conference on Computer Vision.

[25]  Takeo Kanade,et al.  Optical flow estimation using wavelet motion model , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[26]  Richard I. Hartley,et al.  Projective Reconstruction and Invariants from Multiple Images , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  Steven A. Shafer,et al.  Dense Structure from a Dense Optical Flow Sequence , 1998, Comput. Vis. Image Underst..

[28]  David LaRose.  A Fast, Affordable System for Augmented Reality , 1998 .

[29]  P. Anandan,et al.  Hierarchical Model-Based Motion Estimation , 1992, ECCV.

[30]  Olivier D. Faugeras,et al.  3-D Reconstruction of Urban Scenes from Image Sequences , 1998, Comput. Vis. Image Underst..

[31]  S. Teller.  Automated urban model acquisition : Project rationale and status , 1999 .

[32]  Daniel D. Morris,et al.  Factorization methods for structure from motion , 1998, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[33]  Mei Han,et al.  Interactive construction of 3D models from panoramic mosaics , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[34]  Bill Triggs,et al.  Autocalibration and the absolute quadric , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[35]  Takeo Kanade,et al.  A locally adaptive window for signal matching , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[36]  Paul A. Beardsley,et al.  3D Model Acquisition from Extended Image Sequences , 1996, ECCV.

[37]  Takeo Kanade,et al.  Development of a video-rate stereo machine , 1995, Proceedings 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human Robot Interaction and Cooperative Robots.

[38]  Thomas L. Marzetta,et al.  Detection, Estimation, and Modulation Theory , 1976 .

[39]  M. Melamed.  Detection , 2021, SETI: Astronomy as a Contact Sport.

[40]  Rama Chellappa,et al.  3-D Motion Estimation Using a Sequence of Noisy Stereo Images: Models, Estimation, and Uniqueness Results , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[42]  O. Faugeras,et al.  Camera Self-Calibration from Video Sequences: the Kruppa Equations Revisited , 1996 .

[43]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[44]  O. D. Faugeras,et al.  Camera Self-Calibration: Theory and Experiments , 1992, ECCV.

[45]  S. B. Kang,et al.  Recovering 3 D Shape and Motion from Image Streams using Non-Linear Least Squares , 1993 .

[46]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[47]  Rama Chellappa,et al.  Estimation of Object Motion Parameters from Noisy Images , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Bill Triggs,et al.  Factorization methods for projective structure and motion , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[49]  T. Evgeniou.  Image Based Rendering Using Algebraic Techniques , 1996 .

[50]  Paul Debevec,et al.  Modeling and Rendering Architecture from Photographs , 1996, SIGGRAPH 1996.

[51]  Olivier D. Faugeras,et al.  What can two images tell us about a third one? , 1994, ECCV.

[52]  Amnon Shashua,et al.  Projective Structure from Uncalibrated Images: Structure From Motion and Recognition , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[53]  R. Chellappa,et al.  Recursive 3-D motion estimation from a monocular image sequence , 1990 .

[54]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[55]  Tomaso Poggio,et al.  Computational vision and regularization theory , 1985, Nature.

[56]  M. Hebert,et al.  The Representation, Recognition, and Locating of 3-D Objects , 1986 .

[57]  José M. F. Moura,et al.  Factorization as a rank 1 problem , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[58]  Stéphane Christy,et al.  Euclidean Shape and Motion from Multiple Perspective Views by Affine Iterations , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[59]  Seth Teller,et al.  Automatic Extraction of Textured Vertical Facades from Pose Imagery , 1998 .

[60]  Olivier D. Faugeras,et al.  Maintaining representations of the environment of a mobile robot , 1988, IEEE Trans. Robotics Autom..