K-Means Clustering Guided Generative Adversarial Networks for SAR-Optical Image Matching

Synthetic Aperture Radar and optical (SAR-optical) image matching is a technique of finding correspondences between SAR and optical images. SAR-optical image matching can be simplified to single-mode image matching through image synthesis. However, the existing SAR-optical image synthesis methods are unable to provide qualified images for SAR-optical image matching. In this work, we present a K-means Clustering Guide Generative Adversarial Networks (KCG-GAN) to improve the image quality of synthesizing by constraining spatial information synthesis. KCG-GAN uses k-means segmentations as one of the image generator’s inputs and introduces feature matching loss, segmentation loss, and L1 loss to the objective function. Meanwhile, to provide repeatable k-means segmentations, we develop a straightforward 1D k-means algorithm. We compare KCG-GAN with a leading image synthesis method—pix2pixHD. Qualitative results illustrate that KCG-GAN preserves more spatial structures than pix2pixHD. Quantitative results show that, compared with pix2pixHD, images synthesized by KCG-GAN are more similar to original optical images, and SAR-optical image matching based on KCG-GAN obtains at most 3.15 times more qualified matchings. Robustness tests demonstrate that SAR-optical image matching based on KCG-GAN is robust to rotation and scale changing. We also test three SIFT-like algorithms on matching original SAR-optical image pairs and matching KCG-GAN synthesized optical-optical image pairs. Experimental results show that our KCG-GAN significantly improves the performances of the three algorithms on SAR-optical image matching.

[1]  Lorenzo Bruzzone,et al.  Robust Registration of Multimodal Remote Sensing Images Based on Structural Similarity , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[2]  Xiao Xiang Zhu,et al.  The SEN1-2 Dataset for Deep Learning in SAR-Optical Data Fusion , 2018, ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[3]  Junjun Jiang,et al.  Feature-guided Gaussian mixture model for image matching , 2019, Pattern Recognit..

[4]  Francesca Bovolo,et al.  A Fast and Robust Matching Framework for Multimodal Remote Sensing Image Registration , 2018, ArXiv.

[5]  Shuang Wang,et al.  Deep Generative Matching Network for Optical and SAR Image Registration , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[6]  Raquel Urtasun,et al.  Exploiting Deep Matching and SAR Data for the Geo-Localization Accuracy Improvement of Optical Satellite Images , 2017, Remote. Sens..

[7]  Patrick Aravena Pelizari,et al.  Object-Based Analysis and Fusion of Optical and SAR Satellite Data for Dwelling Detection in Refugee Camps , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[8]  Jianjiang Zhou,et al.  Feature-Guided SAR-to-Optical Image Translation , 2020, IEEE Access.

[9]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[10]  Jiri Matas,et al.  Working hard to know your neighbor's margins: Local descriptor learning loss , 2017, NIPS.

[11]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[12]  H. C. Longuet-Higgins,et al.  A computer algorithm for reconstructing a scene from two projections , 1981, Nature.

[13]  Rupert Müller,et al.  On the possibility of conditional adversarial networks for multi-sensor image matching , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[14]  Yang Wang,et al.  MARTA GANs: Unsupervised Representation Learning for Remote Sensing Image Classification , 2016, IEEE Geoscience and Remote Sensing Letters.

[15]  Hongsheng Zhang,et al.  Exploring the optimal integration levels between SAR and optical data for better urban land cover mapping in the Pearl River Delta , 2018, Int. J. Appl. Earth Obs. Geoinformation.

[16]  Fang Liu,et al.  SAR Image Segmentation Based on Hierarchical Visual Semantic and Adaptive Neighborhood Multinomial Latent Model , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[17]  Julie Delon,et al.  SAR-SIFT: A SIFT-Like Algorithm for SAR Images , 2015, IEEE Trans. Geosci. Remote. Sens..

[18]  Maoguo Gong,et al.  Remote Sensing Image Registration With Modified SIFT and Enhanced Feature Matching , 2017, IEEE Geoscience and Remote Sensing Letters.

[19]  Gabriele Moser,et al.  Multimodal Classification of Remote Sensing Images: A Review and Future Directions , 2015, Proceedings of the IEEE.

[20]  Michael Schmitt,et al.  Matching of TerraSAR-X derived ground control points to optical image patches using deep learning , 2019 .

[21]  Taesung Park,et al.  Semantic Image Synthesis With Spatially-Adaptive Normalization , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Maoguo Gong,et al.  A Conditional Adversarial Network for Change Detection in Heterogeneous Images , 2019, IEEE Geoscience and Remote Sensing Letters.

[23]  Sergei Vassilvitskii,et al.  k-means++: the advantages of careful seeding , 2007, SODA '07.

[24]  Olaf Hellwich,et al.  Machine-learning based detection of corresponding interest points in optical and SAR images , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[25]  Junichi Susaki,et al.  SAR and optical data fusion for land use and cover change detection , 2014, 2014 IEEE Geoscience and Remote Sensing Symposium.

[26]  Jan Kautz,et al.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Joachim Denzler,et al.  Deep learning and process understanding for data-driven Earth system science , 2019, Nature.

[28]  Han Zhang,et al.  Registration of Multimodal Remote Sensing Image Based on Deep Fully Convolutional Neural Network , 2019, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[29]  Daniela Stroppiana,et al.  Exploitation of SAR and Optical Sentinel Data to Detect Rice Crop and Estimate Seasonal Dynamics of Leaf Area Index , 2017, Remote. Sens..

[30]  Fang Liu,et al.  Hierarchical semantic model and scattering mechanism based PolSAR image classification , 2015, Pattern Recognit..

[31]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[32]  Michael Schmitt,et al.  Deep Learning for SAR-Optical Image Matching , 2019, IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium.

[33]  Peter Reinartz,et al.  Exploring the Potential of Conditional Adversarial Networks for Optical and SAR Image Matching , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[34]  Itaru Kitahara,et al.  SAR2OPT: Image Alignment Between Multi-Modal Images Using Generative Adversarial Networks , 2019, IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium.

[35]  D. Sculley,et al.  Web-scale k-means clustering , 2010, WWW '10.

[36]  Xiao Xiang Zhu,et al.  Identifying Corresponding Patches in SAR and Optical Images With a Pseudo-Siamese CNN , 2018, IEEE Geoscience and Remote Sensing Letters.

[37]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Kristof Van Tricht,et al.  Synergistic Use of Radar Sentinel-1 and Optical Sentinel-2 Imagery for Crop Mapping: A Case Study for Belgium , 2018, Remote. Sens..

[39]  Olaf Hellwich,et al.  Exploiting GAN-Based SAR to Optical Image Transcoding for Improved Classification via Deep Learning , 2018 .

[40]  Yuming Xiang,et al.  Automatic Registration of Optical and SAR Images VIA Improved Phase Congruency , 2019, IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium.

[41]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[42]  Shang-Hong Lai,et al.  AugGAN: Cross Domain Adaptation with GAN-Based Data Augmentation , 2018, ECCV.

[43]  Patrick Oswald,et al.  Combined Landsat and L-Band SAR Data Improves Land Cover Classification and Change Detection in Dynamic Tropical Landscapes , 2018, Remote. Sens..

[44]  Damian Gromek,et al.  Geometrical Matching of SAR and Optical Images Utilizing ASIFT Features for SAR-based Navigation Aided Systems , 2019, Sensors.

[45]  Wei Jin,et al.  A SAR-to-Optical Image Translation Method Based on Conditional Generation Adversarial Network (cGAN) , 2020, IEEE Access.

[46]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[47]  Yongjun Zhang,et al.  A novel extended phase correlation algorithm based on Log-Gabor filtering for multimodal remote sensing image registration , 2019, International Journal of Remote Sensing.

[48]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[49]  Junjun Jiang,et al.  Robust Feature Matching Using Spatial Clustering With Heavy Outliers , 2020, IEEE Transactions on Image Processing.

[50]  Xiao Xiang Zhu,et al.  Mining Hard Negative Samples for SAR-Optical Image Matching Using Generative Adversarial Networks , 2018, Remote. Sens..

[51]  Haizhou Wang,et al.  Ckmeans.1d.dp: Optimal k-means Clustering in One Dimension by Dynamic Programming , 2011, R J..

[52]  Chuan Li,et al.  Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks , 2016, ECCV.

[53]  Corentin Henry,et al.  SAR-to-Optical Image Translation Based on Conditional Generative Adversarial Networks - Optimization, Opportunities and Limits , 2019, Remote. Sens..

[54]  Yue Wu,et al.  Multimodal Remote Sensing Image Registration Based on Image Transfer and Local Features , 2019, IEEE Geoscience and Remote Sensing Letters.

[55]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[56]  Richard I. Hartley,et al.  In Defense of the Eight-Point Algorithm , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[57]  Andrea Vedaldi,et al.  Vlfeat: an open and portable library of computer vision algorithms , 2010, ACM Multimedia.

[58]  Liangpei Zhang,et al.  Few-Shot Hyperspectral Image Classification With Unknown Classes Using Multitask Deep Learning , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[59]  Joachim Denzler,et al.  Registration of High Resolution Sar and Optical Satellite Imagery Using Fully Convolutional Networks , 2019, IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium.

[60]  Yun Fu,et al.  Segmentation Guided Image-to-Image Translation with Adversarial Networks , 2019, 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).

[61]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[62]  Fang Liu,et al.  A Two-Stage Evolutionary Fuzzy Clustering Framework for Noisy Image Segmentation , 2020, IEEE Access.

[63]  Luís Corte-Real,et al.  Measures for an Objective Evaluation of the Geometric Correction Process Quality , 2009, IEEE Geoscience and Remote Sensing Letters.

[64]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[65]  Junjun Jiang,et al.  Robust Feature Matching for Remote Sensing Image Registration via Locally Linear Transforming , 2015, IEEE Transactions on Geoscience and Remote Sensing.