Image translation between high-resolution optical and synthetic aperture radar (SAR) data

ABSTRACT This paper presents a novel study: remote-sensing image translation between high-resolution optical and Synthetic Aperture Radar (SAR) data through machine learning approaches. To this end, conditional Generative Adversarial Networks (cGANs) have been proposed with the guide of high-level image features. Efficiency of the proposed methods have been verified with different SAR parameters on three regions from the world: Toronto, Vancouver in Canada and Shanghai in China. The generated SAR and optical images have been evaluated by pixel-based image classification with detailed land cover types including: low and high-density residential area, industry area, construction site, golf course, water, forest, pasture and crops. Results showed that the translated image could effectively keep many land cover types with compatible classification accuracy to the ground truth data. In comparison with state-of-the-art image translation approaches, the proposed methods could improve the translation results under the criteria of common similarity indicators. This is one of first study on multi-source remote-sensing data translation by machine learning.

[1]  Leon A. Gatys,et al.  A Neural Algorithm of Artistic Style , 2015, ArXiv.

[2]  Zhongliang Fu,et al.  RESEARCH ON IMAGE TRANSLATION BETWEEN SAR AND OPTICAL IMAGERY , 2012 .

[3]  Xiao Xiang Zhu,et al.  Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources , 2017, IEEE Geoscience and Remote Sensing Magazine.

[4]  Masashi Matsuoka,et al.  Filmy Cloud Removal on Satellite Imagery with Multispectral Conditional Generative Adversarial Nets , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[5]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[6]  Jörg Stückler,et al.  Semi-Supervised Deep Learning for Monocular Depth Map Prediction , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[8]  Yong Dou,et al.  Image Translation Between High-Resolution Remote Sensing Optical and SAR Data Using Conditional GAN , 2018, PCM.

[9]  Hiroshi Ishikawa,et al.  Let there be color! , 2016, ACM Trans. Graph..

[10]  Jong-Sen Lee,et al.  Refined filtering of image noise using local statistics , 1981 .

[11]  Mihai Datcu,et al.  Dialectical GAN for SAR Image Translation: From Sentinel-1 to TerraSAR-X , 2018, Remote. Sens..

[12]  Gregory Shakhnarovich,et al.  Learning Representations for Automatic Colorization , 2016, ECCV.

[13]  Weiming Wang,et al.  Image Translation Between Sar and Optical Imagery with Generative Adversarial Nets , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[14]  Eric Pottier,et al.  A review of target decomposition theorems in radar polarimetry , 1996, IEEE Trans. Geosci. Remote. Sens..

[15]  Yong Dou,et al.  RADARSAT-2 fine-beam polarimetric and ultra-fine-beam SAR data for urban mapping: comparison and synergy , 2016 .

[16]  Hui Jiang,et al.  Annealed Gradient Descent for Deep Learning , 2015, UAI.

[17]  Yong Dou,et al.  DropFilterR: A Novel Regularization Method for Learning Convolutional Neural Networks , 2019, Neural Processing Letters.

[18]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Xiao Xiang Zhu,et al.  Deep learning in remote sensing: a review , 2017, ArXiv.

[20]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[21]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[22]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[23]  Xin Niu,et al.  An Adaptive Contextual SEM Algorithm for Urban Land Cover Mapping Using Multitemporal High-Resolution Polarimetric SAR Data , 2012, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[24]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[25]  Yue Yu,et al.  SAR-to-Optical Image Translation Using Supervised Cycle-Consistent Adversarial Networks , 2019, IEEE Access.

[26]  Yann LeCun,et al.  Deep multi-scale video prediction beyond mean square error , 2015, ICLR.

[27]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[28]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[30]  Bo Du,et al.  Deep Learning for Remote Sensing Data: A Technical Tutorial on the State of the Art , 2016, IEEE Geoscience and Remote Sensing Magazine.

[31]  Wei Shen,et al.  Learning Residual Images for Face Attribute Manipulation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).