S2OMGAN: Shortcut from Remote Sensing Images to Online Maps

Traditional online maps, widely used on Internet such as Google map and Baidu map, are rendered from vector data. Timely updating online maps from vector data, of which the generating is time-consuming, is a difficult mission. It is a shortcut to generate online maps in time from remote sensing images, which can be acquired timely without vector data. However, this mission used to be challenging or even impossible. Inspired by image-to-image translation (img2img) techniques based on generative adversarial network (GAN), we propose a semi-supervised structure-augmented online map GAN (S$^{2}$OMGAN) model to generate online maps directly from remote sensing images. In this model, we designed a semi-supervised learning strategy to pre-train S$^{2}$OMGAN on rich unpaired samples and finetune it on limited paired samples in reality. We also designed image gradient L1 loss and image gradient structure loss to generate an online map with global topological relationship and detailed edge curves of objects, which are important in cartography. Moreover, we propose edge structural similarity index (ESSI) as a metric to evaluate the quality of topological consistency between generated online maps and ground truths. Experimental results present that S$^{2}$OMGAN outperforms state-of-the-art (SOTA) works according to mean squared error, structural similarity index and ESSI. Also, S$^{2}$OMGAN wins more approval than SOTA in the human perceptual test on visual realism of cartography. Our work shows that S$^{2}$OMGAN is potentially a new paradigm to produce online maps. Our implementation of the S$^{2}$OMGAN is available at \url{this https URL}.

[1]  Filiberto Pla,et al.  A New Deep Generative Network for Unsupervised Remote Sensing Single-Image Super-Resolution , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[2]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[3]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[4]  Lorenzo Bruzzone,et al.  Quality assessment of classification and cluster maps without ground truth knowledge , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[5]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[6]  Alexei A. Efros,et al.  Toward Multimodal Image-to-Image Translation , 2017, NIPS.

[7]  Jung-Woo Ha,et al.  StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Jan Neumann.  The Topological Information Content of a Map / An Attempt At A Rehabilitation Of Information Theory In Cartography , 1994 .

[9]  Gözde B. Ünal,et al.  Patch-Based Image Inpainting with Generative Adversarial Networks , 2018, ArXiv.

[10]  Junjun Jiang,et al.  Edge-Enhanced GAN for Remote Sensing Image Superresolution , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[11]  Zhou Wang,et al.  Image Quality Assessment: From Error Measurement to Structural Similarity , 2004 .

[12]  Jun Li,et al.  Unsupervised Feature Extraction in Hyperspectral Images Based on Wasserstein Generative Adversarial Network , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[13]  Hayit Greenspan,et al.  Synthetic data augmentation using GAN for improved liver lesion classification , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[14]  Michał Januszewski,et al.  Segmentation-Enhanced CycleGAN , 2019, bioRxiv.

[15]  Karel Zimmermann,et al.  Data-Driven Policy Transfer With Imprecise Perception Simulation , 2018, IEEE Robotics and Automation Letters.

[16]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  David Hurych,et al.  Yes, we GAN: Applying Adversarial Techniques for Autonomous Driving , 2019, Autonomous Vehicles and Machines.

[18]  Sarfraz Khurshid,et al.  DeepRoad: GAN-Based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[19]  Hua Huang,et al.  Image Quality Assessment Using Directional Anisotropy Structure Measurement , 2017, IEEE Transactions on Image Processing.

[20]  Kwang In Kim,et al.  Unsupervised Attention-guided Image to Image Translation , 2018, NeurIPS.

[21]  Chunhong Pan,et al.  Enhancing Pix2Pix for Remote Sensing Image Classification , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[22]  Qian Du,et al.  Caps-TripleGAN: GAN-Assisted CapsNet for Hyperspectral Image Classification , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[23]  Kaiqi Huang,et al.  GP-GAN: Towards Realistic High-Resolution Image Blending , 2017, ACM Multimedia.

[24]  Tal Hassner,et al.  Extreme 3D Face Reconstruction: Seeing Through Occlusions , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Yong Yu,et al.  Guiding the One-to-one Mapping in CycleGAN via Optimal Transport , 2018, AAAI.

[26]  Jan Kautz,et al.  Unsupervised Image-to-Image Translation Networks , 2017, NIPS.

[27]  Philip Bachman,et al.  Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data , 2018, ICML.

[28]  Jan Kautz,et al.  Multimodal Unsupervised Image-to-Image Translation , 2018, ECCV.

[29]  Werner Kuhn,et al.  A Keystroke Level Analysis of Manual Map Digitizing , 1993, COSIT.

[30]  Lin Zhu,et al.  Generative Adversarial Networks for Hyperspectral Image Classification , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[31]  Lior Wolf,et al.  Unsupervised Cross-Domain Image Generation , 2016, ICLR.

[32]  Geoffrey I. Webb,et al.  Encyclopedia of Machine Learning , 2011, Encyclopedia of Machine Learning.

[33]  Swetava Ganguli,et al.  GeoGAN: A Conditional GAN with Reconstruction and Style Loss to Generate Standard Layer of Maps from Satellite Images , 2019, ArXiv.

[34]  Riad I. Hammoud,et al.  Deep Learning Based Single Image Dehazing , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[35]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[36]  Yu Cheng,et al.  Bayesian Cycle-Consistent Generative Adversarial Networks via Marginalizing Latent Sampling. , 2020, IEEE transactions on neural networks and learning systems.

[37]  Dimitris N. Metaxas,et al.  StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[38]  Hazim Kemal Ekenel,et al.  Cycle-Dehaze: Enhanced CycleGAN for Single Image Dehazing , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[39]  Yu Cheng,et al.  Bayesian CycleGAN via Marginalizing Latent Sampling , 2018, ArXiv.

[40]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Snehamoy Chatterjee,et al.  A Modified Generative Adversarial Nets Integrated With Stochastic Approach for Realizing Super-Resolution Reservoir Simulation , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[42]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Song Gao,et al.  Transferring multiscale map styles using generative adversarial networks , 2019, International Journal of Cartography.

[44]  Mark Sandler,et al.  CycleGAN, a Master of Steganography , 2017, ArXiv.

[45]  Lei Zhang,et al.  Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index , 2013, IEEE Transactions on Image Processing.