Alignment of continuous video onto 3D point clouds

We propose a general framework for aligning continuous (oblique) video onto 3D sensor data. We align a point cloud computed from the video onto the point cloud directly obtained from a 3D sensor. This is in contrast to existing techniques where the 2D images are aligned to a 3D model derived from the 3D sensor data. Using point clouds enables the alignment for scenes full of objects that are difficult to model, for example, trees. To compute 3D point clouds from video, motion stereo is used along with a state-of-the-art algorithm for camera pose estimation. Our experiments with real data demonstrate the advantages of the proposed registration algorithm for texturing models in large-scale semi-urban environments. The capability to align video before a 3D model is built from the 3D sensor data opens up new possibilities for 3D modeling. We introduce a novel modeling-through-registration approach that fuses 3D information from both the 3D sensor and the video. Initial experiments with real data illustrate the potential of the proposed approach.

[1]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[2]  B. Anderson,et al.  Optimal Filtering , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[3]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using unit quaternions , 1987 .

[4]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[5]  Jake K. Aggarwal,et al.  Matching Aerial Images to 3-D Terrain Maps , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  Gérard G. Medioni,et al.  Object modeling by registration of multiple range images , 1991, Proceedings. 1991 IEEE International Conference on Robotics and Automation.

[7]  David G. Lowe,et al.  Fitting Parameterized Three-Dimensional Models to Images , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Gérard G. Medioni,et al.  Object modelling by registration of multiple range images , 1992, Image Vis. Comput..

[9]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  O. Faugeras.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[11]  Takeo Kanade,et al.  Real-time 3-D pose estimation using a high-speed range sensor , 1993, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[12]  Marc Rioux,et al.  Three-dimensional registration using range and intensity information , 1994, Other Conferences.

[13]  P. Anandan,et al.  Direct recovery of shape from multiple views: a parallax based approach , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[14]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[15]  P. Anandan,et al.  Parallax Geometry of Pairs of Points for 3D Scene Analysis , 1996, ECCV.

[16]  Carlo Tomasi,et al.  Comparison of approaches to egomotion computation , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  Katsuhiko Sakaue,et al.  Registration and integration of multiple range images for 3-D model construction , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[18]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.

[19]  Trevor Darrell,et al.  3D pose tracking with linear depth and brightness constraints , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[20]  Kia Ng,et al.  Automated reconstruction of 3D models from real environments , 1999 .

[21]  Yi-Ping Hung,et al.  RANSAC-Based DARCES: A New Approach to Fast Automatic Registration of Partially Overlapping Range Images , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  David Nistér,et al.  Reconstruction from Uncalibrated Sequences with a Hierarchy of Trifocal Tensors , 2000, ECCV.

[23]  Supun Samarasekera,et al.  Pose estimation, model refinement, and enhanced visualization using video , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[24]  Marc Levoy,et al.  Efficient variants of the ICP algorithm , 2001, Proceedings Third International Conference on 3-D Digital Imaging and Modeling.

[25]  Hai Tao,et al.  A global matching framework for stereo computation , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[26]  Claus Brenner,et al.  Towards Fully Automated 3D City Model Generation , 2001 .

[27]  Andrew Zisserman,et al.  Multiple view geometry in computer visiond , 2001 .

[28]  Patrick J. Flynn,et al.  A Survey Of Free-Form Object Representation and Recognition Techniques , 2001, Comput. Vis. Image Underst..

[29]  Ioannis Stamos,et al.  Automatic registration of 2-D with 3-D imagery in urban environments , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[30]  Ioannis Stamos,et al.  Geometry and Texture Recovery of Scenes of Large Scale , 2002, Comput. Vis. Image Underst..

[31]  Supun Samarasekera,et al.  Video Flashlights: Real Time Rendering of Multiple Videosfor Immersive Model Visualization , 2002, Rendering Techniques.

[32]  Xavier Pennec,et al.  Multi-scale EM-ICP: A Fast and Robust Approach for Surface Registration , 2002, ECCV.

[33]  Christian Früh,et al.  Constructing 3D City Models by Merging Aerial and Ground Views , 2003, IEEE Computer Graphics and Applications.

[34]  Christian Früh,et al.  Reconstructuring 3D City Models by Merging Ground-Based and Airborne Views , 2003, VLBV.

[35]  Suya You,et al.  Augmented virtual environments (AVE): dynamic fusion of imagery and 3D models , 2003, IEEE Virtual Reality, 2003. Proceedings..

[36]  Larry H. Matthies,et al.  Kalman filter-based algorithms for estimating depth from image sequences , 1989, International Journal of Computer Vision.

[37]  David Nistér,et al.  An efficient solution to the five-point relative pose problem , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Paul A. Beardsley,et al.  Sequential Updating of Projective and Affine Structure from Motion , 1997, International Journal of Computer Vision.

[39]  Andrew W. Fitzgibbon,et al.  The Problem of Degeneracy in Structure and Motion Recovery from Uncalibrated Image Sequences , 1999, International Journal of Computer Vision.

[40]  Paul A. Viola,et al.  Alignment by Maximization of Mutual Information , 1997, International Journal of Computer Vision.

[41]  Zhengyou Zhang,et al.  Iterative point matching for registration of free-form curves and surfaces , 1994, International Journal of Computer Vision.