Registration of Images To Lidar and GIS Data Without Establishing Explicit Correspondences

Recovering the camera orientation is a fundamental problem in photogrammetry for precision 3D recovery, orthophoto generation, and image registration. In this paper, we achieve this goal by fusing the image information with information extracted from different modalities, including lidar and GIS. In contrast to other approaches, which require feature correspondences, our approach exploits edges across the modalities without the necessity to explicitly establish correspondences. In the proposed approach, extracted edges from different modalities are not required to have analytical forms. This flexibility is achieved by minimizing a new cost function using a Bayesian approach, which takes the Euclidean distances between the projected edges extracted from the other data source and the edges extracted from the reference image as its random variable. The proposed formulation minimizes the overall distances between the sets of edges iteratively, such that the end product results in the correct camera parameters for the reference image as well as matching features across the modalities. The initial solution can be obtained from GPS/IMU data. The formulation is shown to successfully handle noise and missing observations in edges. Point matching methods may fail for oblique images, especially high oblique images. We eliminate the requirement for exact point-to-point matching. The feasibility of the method is experimented with nadir and oblique images. Introduction Registration is the process of aligning two or more datasets acquired for the same site with different coordinate systems to a single coordinate system. Considering the ever increasing amount of multi-modal datasets with different geometric, radiometric, temporal, and thematic resolutions, it is important to register these datasets to a single coordinate system by registration (Sester et al., 1998), which in turn provides us with the ability to exploit the advantages offered by each of the different modalities (Schenk and Csathó, 2002). The registration between such datasets is traditionally divided into four steps, namely: feature extraction, matching, transformation, and resampling (Zitová and Flusser, 2003). In this paper, we discuss a nontraditional method for registering aerial images to GIS (Geographical Information Systems) and lidar (Light Detection And Ranging) data. This is a continuation of the paper on view invariant shape recognition (Yilmaz and Barsai, 2008). This previous paper studied freeform matching using Fourier descriptors. Registration of aerial images is an important task in photogrammetry, as it is a pre-requisite for 3D surface recovery and orthophoto generation, as well as performing higher level inference tasks such as object recognition. An image is traditionally registered to a reference system using a set of Ground Control Points (GCPs) and corresponding pixels in the image. With the advancement in GPS/ IMU (Global Positioning System/Inertial Measurement Unit) technology, exterior orientation parameters (EOPs) are obtained from the instrument measurements and aerial images are registered on the fly, also called direct orientation. A major drawback of direct orientation is the computation of EOP without considering the interior orientation parameters (IOPs) that continually changes due to environmental and mechanical factors (Schenk, 1999). However, the EOP derived from direct orientation can be used as initial approximations for precise mapping applications. For indirect orientation, it is possible to use information extracted from other datasets such as lidar and GIS. While very accurate GIS and lidar datasets are publicly available from USGS (United States Geological Survey) and state agencies for any given area, they are usually not used for registration due to the fact that establishing correspondences between an image and GIS/lidar is a challenging problem and is an ongoing research (Heipke, 1997) (Shan and Toth, 2009) (Sourimant et al., 2011) (Bartie et al., 2011). In recent years, researchers have exploited the features extracted from the lidar data, such as lines and planes, to register them with the images (Habib et al., 2007) (Habib et al., 2005) (Jaw and Wu, 2006) (Schenk and Csathó, 2002) (Nagarajan and Schenk, 2016). Extracting linear and planar features from lidar data, however, is not always intuitive due to surface patterns, noise, and lidar point density. In contrast to registration of lidar with images, the registration of GIS data with images has received less attention except only a few attempts such as (Sester et al., 1998, Shan, 2000) and (Chawathe, 2007). GIS features and the features extracted from a different sensory data may not be the same due to different sampling of the real world (Schenk and Csathó, 2007). The number of vertices that constitute a shape can also make features to look different. This paper demonstrates a novel method to register an image with the GIS and lidar data without establishing correspondences by eliminating the feature matching step. Image registration methods can broadly be categorized based on (a) the dataset representation, (b) the model for establishing correspondences and (c) the choice of mathematical models (Nagarajan, 2010). In the case when Gabor Barsai is with Ferris State University, School of Engineering & Computing Technology, 1009 Campus Drive, JOH 408, Big Rapids, MI 49307 (barsi.2@buckeyemail.osu.edu). Alper Yilmaz is with The Ohio State University, Photogrammetric Computer Vision Laboratory, Bolz Hall, 2036 Neil Ave. Mall, Columbus, OH, 43210. Panu Srestasathiern is with the Geo-Informatics and Space Technology Development Agency, 120 The Government Complex, Building B, 6th and 7th Floor, Chaeng Wattana Road, Lak Si, Bangkok 10210, Thailand. Sudhager Nagarajan is with Florida Atlantic University, Civil, Environmental and Geomatics Engineering, Building EG-36, Room 222, 777 Blades Road, Boca Raton, FL, 33431. Photogrammetric Engineering & Remote Sensing Vol. 83, No. 10, October 2017, pp. 39–xxx. 0099-1112/17/39–xxx © 2017 American Society for Photogrammetry and Remote Sensing doi: 10.14358/PERS.83.10.xxx PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING October 2017 39 registration is performed between different modalities, it is important to relate information extracted from the complimentary domains where one modality provides information that is not available in the other one. Due to the overwhelming volume of research that has been performed on registration, this paper discusses only the previous research that is directly relevant to the methodology described in the remainder of the text. The interested reader can find overviews of registration from a variety of perspectives in (Campbell and Flynn, 2001; Huber and Hebert, 2003) and (Li et al., 2008). Here, we will concentrate on a feature-based approach that matches free-form edges without the need for explicit, feature to feature correspondences across different modalities. Most photogrammetric applications depend on the use of distinct points (Habib et al., 2002), which can be extracted manually, or automatically using Moravec point operator (Moravec, 1977), Fo ̈rstner point operator (Förstner and Gu ̈lch, 1987), Harris corner detector (Harris and Stephens, 1988) or Scale Invariant Feature Transform (SIFT) (Lowe, 2004). While point based methods, which depend on features extracted by “experienced personnel”, produce high \ precision results in a traditional setting, their accuracy, however, drops significantly when the process is automated. This performance degradation is due to challenges stemming from several factors including: partial occlusions, viewpoint changes, varying illumination, cluttered backgrounds, and complimentary information acquired from different sensor modalities (Remondino et al., 2008). In order to develop more accurate techniques, researchers have more recently resorted from points to higher level features, such as lines (Habib et al., 2003) (Schenk, 2004), surfaces (Jaw and Wu, 2006), curves (Hartley and Zisserman, 2000), manholes (Drewniok and Rohr, 1997) and road junctions (Pedersen, 1996). These features provide higher level of redundancy and decrease the effect of errors associated with noisy point locations and imperfect matches (Wang et al., 2008). All these methods and other similar approaches typically require user interaction to create local geometric models (Läbe and Ellenbeck, 1996) (Doucette et al., 2008), and to find and match elongated features, such as roads (Habib et al., 2003). Automated photogrammetric methods are based on individual point matching, but point matching fails for oblique images (Tuytelaars and Mikolajczyk, 2008) due to higher distorsions. In addition, GPS-IMU accuracy is dependent on the instrument itself, as shown in (Leberl et al., 2010). Add to that the IOPs may change during the use of the camera: air pressure change, shaking of the camera could change over the time it takes to obtain the images, making individual point matching very dependent on the quality of the individual points. For point-based methods applied on data from different modalities, the ICP (Iterative Closest Point) algorithm (Besl and McKay, 1992) is considered to be one of the most popular methods for many years with applications in registering targets to lidar scans (Barnea and Filin, 2008) (Wang and Brenner, 2008). The ICP algorithm finds the best correspondences between two point sets by iteratively determining the translation and rotation parameters of a 2D or a 3D rigid body transformation by minimizing the distance between the two data sets. The ICP algorithm, however, assumes that one point set is a subset of the other set, which is not the case when, for instance, an image is registered to 3D points from a lidar dataset. In addition, photogrammetric points may even not have any conjugate points among the lidar points. Despite its common application, ICP is used when the tr