Exploring the Potential of Conditional Adversarial Networks for Optical and SAR Image Matching

Tasks such as the monitoring of natural disasters or the detection of change highly benefit from complementary information about an area or a specific object of interest. The required information is provided by fusing high accurate coregistered and georeferenced datasets. Aligned high-resolution optical and synthetic aperture radar (SAR) data additionally enable an absolute geolocation accuracy improvement of the optical images by extracting accurate and reliable ground control points (GCPs) from the SAR images. In this paper, we investigate the applicability of a deep learning based matching concept for the generation of precise and accurate GCPs from SAR satellite images by matching optical and SAR images. To this end, conditional generative adversarial networks (cGANs) are trained to generate SAR-like image patches from optical images. For training and testing, optical and SAR image patches are extracted from TerraSAR-X and PRISM image pairs covering greater urban areas spread over Europe. The artificially generated patches are then used to improve the conditions for three known matching approaches based on normalized cross-correlation (NCC), scale-invariant feature transform (SIFT), and binary robust invariant scalable key (BRISK), which are normally not usable for the matching of optical and SAR images. The results validate that a NCC-, SIFT-, and BRISK-based matching greatly benefit, in terms of matching accuracy and precision, from the use of the artificial templates. The comparison with two state-of-the-art optical and SAR matching approaches shows the potential of the proposed method but also revealed some challenges and the necessity for further developments.

[1]  Ian Dowman,et al.  An improved model for automatic feature-based registration of SAR and SPOT images , 2001 .

[2]  Yun Zhang,et al.  A critical review of image registration methods , 2010 .

[3]  Su Ruan,et al.  Medical Image Synthesis with Context-Aware Generative Adversarial Networks , 2016, MICCAI.

[4]  Roland Siegwart,et al.  BRISK: Binary Robust invariant scalable keypoints , 2011, 2011 International Conference on Computer Vision.

[5]  Raquel Urtasun,et al.  Exploiting Deep Matching and SAR Data for the Geo-Localization Accuracy Improvement of Optical Satellite Images , 2017, Remote. Sens..

[6]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[7]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[8]  M. Bossard,et al.  CORINE land cover technical guide - Addendum 2000 , 2000 .

[9]  Florence Tupin,et al.  Iterative Weighted Maximum Likelihood Denoising With Probabilistic Patch-Based Weights , 2009, IEEE Transactions on Image Processing.

[10]  Peter Reinartz,et al.  Mutual-Information-Based Registration of TerraSAR-X and Ikonos Imagery in Urban Areas , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[11]  Youkyung Han,et al.  Automatic and accurate registration of VHR optical and SAR images using a quadtree structure , 2015 .

[12]  Songde Ma,et al.  Multisource data registration based on NURBS description of contours , 2008 .

[13]  Mark R. Pickering,et al.  Robust Automatic Registration of Multimodal Satellite Images Using CCRE With Partial Volume Interpolation , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[14]  Jan Flusser,et al.  Image registration methods: a survey , 2003, Image Vis. Comput..

[15]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Lisa M. Brown,et al.  A survey of image registration techniques , 1992, CSUR.

[18]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[19]  M. Saquib Sarfraz,et al.  Automatic registration of SAR and optical images based on mutual information assisted Monte Carlo , 2012, 2012 IEEE International Geoscience and Remote Sensing Symposium.

[20]  Wilhelm Burger,et al.  Principles of Digital Image Processing - Core Algorithms , 2009, Undergraduate Topics in Computer Science.

[21]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[22]  Rupert Müller,et al.  On the possibility of conditional adversarial networks for multi-sensor image matching , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[23]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[24]  Chunhong Pan,et al.  Registration of Optical and SAR Satellite Images by Exploring the Spatial Relationship of the Improved SIFT , 2013, IEEE Geoscience and Remote Sensing Letters.

[25]  Peter Steigenberger,et al.  Imaging Geodesy—Toward Centimeter-Level Ranging Accuracy With TerraSAR-X , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[26]  R. Schowengerdt,et al.  A robust technique for precise registration of radar and optical satellite images , 2005 .

[27]  Wenxian Yu,et al.  An improved Normalized Cross Correlation algorithm for SAR image registration , 2012, 2012 IEEE International Geoscience and Remote Sensing Symposium.

[28]  Yueting Zhang,et al.  Synthetic Aperture Radar Image Synthesis by Using Generative Adversarial Nets , 2017, IEEE Geoscience and Remote Sensing Letters.

[29]  B. S. Manjunath,et al.  A contour-based approach to multisensor image registration , 1995, IEEE Trans. Image Process..

[30]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[31]  Camille Couprie,et al.  Semantic Segmentation using Adversarial Networks , 2016, NIPS 2016.

[32]  Peter Reinartz,et al.  Urban Atlas – DLR Processing Chain for Orthorectification of Prism and AVNIR-2 Images and TerraSAR-X as possible GCP Source , 2010 .

[33]  Wei Shi,et al.  A visual circle based image registration algorithm for optical and SAR imagery , 2012, 2012 IEEE International Geoscience and Remote Sensing Symposium.