Dense point cloud generation of urban scenes from nadir RGB images in a remote sensing system

A near real-time airborne 3D scanning system has been successfully implemented at Fraunhofer IOSB. This remote sensing system consists of a small aircraft and a ground control station. The aircraft is equipped with the following components: a Digital Acquisition System (DAQ), Inertial Navigation and Global Positioning Systems (INS/GPS), an Airborne Laser Scanner (ALS), and four industrial cameras. Two of these cameras (RGB and near-infrared ones) are nadir oriented, while the other two RGB cameras have an oblique orientation. The acquired LiDAR point clouds, images, and corresponding metadata are sent from the aircraft to the ground control station for further post-processing procedures, such as radiometric correction, boresight correction, and point cloud generation from images. In this paper, the procedure regarding point cloud generation of urban scenes, with images from the nadir RGB camera, is described in detail. To produce dense point clouds three main steps are necessary: generation of disparity maps, creation of depth maps, and calculation of world coordinates (X, Y, and Z). To create disparity maps, two adjacent images (stereopair) were rectified. Afterwards, the PatchMatch Stereo (PMS) algorithm for 3D reconstruction was executed, since it is easy to implement and provides good results according to the Middlebury Computer Vision dataset. Some steps were parallelized to optimize execution speed. Since depth is inversely proportional to disparity, depth maps were calculated from disparity maps. The height of scene elements Z was obtained by subtracting their depth to the camera height. To calculate the remaining world coordinates X and Y, the back-projection equation and the camera intrinsic and extrinsic parameters were used. To validate the PMS algorithm, its resulting point cloud was compared with a LiDAR point cloud and a PhotoScan point cloud. The root mean square errors of both comparisons showed similar values.

[1]  Andrew Zisserman,et al.  Computation of the Fundamental Matrix F , 2004 .

[2]  Dimitri Bulatov Temporal Selection of Images for a Fast Algorithm for Depth-map Extraction in Multi-baseline Configurations , 2015, VISAPP.

[3]  Andreas Lenz,et al.  Automatic in-flight boresight calibration considering topography for hyperspectral pushbroom sensors , 2014, 2014 IEEE Geoscience and Remote Sensing Symposium.

[4]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[5]  H. Hirschmüller Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information , 2005, CVPR.

[6]  Amal Mahdi Ali DETERMINATION OF THE GEOID HEIGHT (GEOID UNDULATION) BY USING MODERN SURVEYING TECHNOLOGIES , 2016 .

[7]  Daniel G. Aliaga,et al.  Building reconstruction using manhattan-world grammars , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Michael J. Black,et al.  Towards Probabilistic Volumetric Reconstruction Using Ray Potentials , 2015, 2015 International Conference on 3D Vision.

[9]  D. Bulatov Metric reconstruction from video sequences , 2008, Pattern Recognition and Image Analysis.

[10]  Wolfgang Middelmann,et al.  Concept for an airborne real-time ISR system with multi-sensor 3D data acquisition , 2016, Security + Defence.

[11]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Carsten Rother,et al.  PatchMatch Stereo - Stereo Matching with Slanted Support Windows , 2011, BMVC.

[13]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Andreas Lenz,et al.  Towards Fast 3D Reconstruction of Urban Areas from Aerial Nadir Images for a Near Real-Time Remote Sensing System , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[15]  R. Fletcher A modified Marquardt subroutine for non-linear least squares , 1971 .

[16]  Andrew Zisserman,et al.  MLESAC: A New Robust Estimator with Application to Estimating Image Geometry , 2000, Comput. Vis. Image Underst..

[17]  Wolfgang Middelmann,et al.  3D building reconstruction in a remote sensing workflow , 2017, Remote Sensing.

[18]  Jochen Meidow,et al.  VOXEL-BASED QUALITY EVALUATION OF PHOTOGRAMMETRIC BUILDING ACQUISITIONS , 2005 .

[19]  Konrad Schindler,et al.  A model-based method for building reconstruction , 2003, First IEEE International Workshop on Higher-Level Knowledge in 3D Modeling and Motion Analysis, 2003. HLK 2003..