Automatic registration of panoramic image sequence and mobile laser scanning data using semantic features

Abstract Inaccurate exterior orientation parameters (EoPs) between sensors obtained by pre-calibration leads to failure of registration between panoramic image sequence and mobile laser scanning data. To address this challenge, this paper proposes an automatic registration method based on semantic features extracted from panoramic images and point clouds. Firstly, accurate rotation parameters between the panoramic camera and the laser scanner are estimated using GPS and IMU aided structure from motion (SfM). The initial EoPs of panoramic images are obtained at the same time. Secondly, vehicles in panoramic images are extracted by the Faster-RCNN as candidate primitives to be matched with potential corresponding primitives in point clouds according to the initial EoPs. Finally, translation between the panoramic camera and the laser scanner is refined by maximizing the overlapping area of corresponding primitive pairs based on the Particle Swarm Optimization (PSO), resulting in a finer registration between panoramic image sequences and point clouds. Two challenging urban scenes were experimented to assess the proposed method, and the final registration errors of these two scenes were both less than three pixels, which demonstrates a high level of automation, robustness and accuracy.

[1]  Sanjiv K. Bhatia.  Adaptive K-Means Clustering , 2004, FLAIRS Conference.

[2]  Bruno Vallet,et al.  HYBRID ONLINE MOBILE LASER SCANNER CALIBRATION THROUGH IMAGE ALIGNMENT BY MUTUAL INFORMATION , 2016 .

[3]  Jianya Gong,et al.  Line-Based Registration of Panoramic Images and LiDAR Point Clouds for Mobile Mapping , 2017, Sensors.

[4]  Bisheng Yang,et al.  Automated Extraction of Building Outlines From Airborne Laser Scanning Point Clouds , 2013, IEEE Geoscience and Remote Sensing Letters.

[5]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Manuel Menezes de Oliveira Neto,et al.  Real-time line detection through an improved Hough transform voting scheme , 2008, Pattern Recognit..

[7]  Sebastian Thrun,et al.  Automatic Online Calibration of Cameras and Lasers , 2013, Robotics: Science and Systems.

[8]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[9]  Kiyun Yu,et al.  Co-Registration of Aerial Photos, ALS Data and Digital Maps Using Linear Features , 2006 .

[10]  Kiyun Yu,et al.  Registration of aerial imagery and aerial LiDAR data using centroids of plane roof surfaces as control information , 2006 .

[11]  Sagi Filin,et al.  Segmentation of terrestrial laser scanning data using geometry and image information , 2013 .

[12]  Ayman Habib,et al.  Surface Matching and Change Detection Using a Modified Hough Transformation for Robust Parameter Estimation , 2001 .

[13]  Bisheng Yang,et al.  Automatic registration of UAV-borne sequent images and LiDAR data , 2015 .

[14]  Dieter Fritsch,et al.  AN AUTOMATIC PROCEDURE FOR COMBINING DIGITAL IMAGES AND LASER SCANNER DATA , 2012 .

[15]  Jean-Michel Morel,et al.  ASIFT: A New Framework for Fully Affine Invariant Image Comparison , 2009, SIAM J. Imaging Sci..

[16]  Daniel P. Huttenlocher,et al.  Efficient Graph-Based Image Segmentation , 2004, International Journal of Computer Vision.

[17]  Ioannis Stamos,et al.  Automatic 3D to 2D registration for the photorealistic rendering of urban scenes , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[18]  A. Habib,et al.  Photogrammetric and Lidar Data Registration Using Linear Features , 2005 .

[19]  Wolfram Burgard,et al.  G2o: A general framework for graph optimization , 2011, 2011 IEEE International Conference on Robotics and Automation.

[20]  Bisheng Yang,et al.  Hierarchical extraction of urban objects from mobile laser scanning data , 2015 .

[21]  Vibhav Vineet,et al.  Conditional Random Fields as Recurrent Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[22]  E. G. Parmehr,et al.  Automatic registration of optical imagery with 3D LiDAR data using statistical similarity , 2014 .

[23]  Didier Stricker,et al.  Structure from Motion using full spherical panoramic cameras , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[24]  Claus Brenner,et al.  Skyline matching based camera orientation from images and mobile mapping point clouds , 2014 .

[25]  Maurice Clerc,et al.  The particle swarm - explosion, stability, and convergence in a multidimensional complex space , 2002, IEEE Trans. Evol. Comput..

[26]  Bisheng Yang,et al.  A shape-based segmentation method for mobile laser scanning point clouds , 2013 .

[27]  Roberto Scopigno,et al.  Fully Automatic Registration of Image Sets on Approximate Geometry , 2012, International Journal of Computer Vision.

[28]  Frank P. Ferrie,et al.  Automatic registration of mobile LiDAR and spherical panoramas , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[29]  Matthew Brand,et al.  Geolocalization using skylines from omni-images , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[30]  Stéphane Christy,et al.  Iterative Pose Computation from Line Correspondences , 1999, Comput. Vis. Image Underst..

[31]  Beatriz Marcotegui,et al.  Point cloud segmentation towards urban ground modeling , 2009, 2009 Joint Urban Remote Sensing Event.

[32]  Claus Brenner,et al.  Coarse orientation of terrestrial laser scans in urban environments , 2008 .

[33]  Russell C. Hardie,et al.  Automatic Registration of Optical Aerial Imagery to a LiDAR Point Cloud for Generation of City Models , 2015 .

[34]  Dani Lischinski,et al.  A Closed-Form Solution to Natural Image Matting , 2008 .

[35]  Roman M. Palenichka,et al.  Automatic Extraction of Control Points for the Registration of Optical Satellite and LiDAR Images , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[36]  Lu Wang,et al.  A robust approach for automatic registration of aerial images with untextured aerial LiDAR data , 2009, CVPR.

[37]  Yipu Zhao,et al.  2D-image to 3D-range registration in urban environments via scene categorization and combination of similarity measurements , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[38]  W. Cohen,et al.  Using Landsat-derived disturbance and recovery history and lidar to map forest biomass dynamics , 2014 .