Joint Learning from Earth Observation and OpenStreetMap Data to Get Faster Better Semantic Maps

In this work, we investigate the use of OpenStreetMap data for semantic labeling of Earth Observation images. Deep neural networks have been used in the past for remote sensing data classification from various sensors, including multispectral, hyperspectral, SAR and LiDAR data. While OpenStreetMap has already been used as ground truth data for training such networks, this abundant data source remains rarely exploited as an input information layer. In this paper, we study different use cases and deep network architectures to leverage OpenStreetMap data for semantic labeling of aerial and satellite images. Especially, we look into fusion based architectures and coarseto- fine segmentation to include the OpenStreetMap layer into multispectral-based deep fully convolutional networks. We illustrate how these methods can be successfully used on two public datasets: ISPRS Potsdam and DFC2017. We show that OpenStreetMap data can efficiently be integrated into the vision-based deep learning models and that it significantly improves both the accuracy performance and the convergence speed of the networks.

[1]  Jamie Sherrah,et al.  Effective semantic pixel labelling with convolutional networks and Conditional Random Fields , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[2]  Bertrand Le Saux,et al.  Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks , 2016, ACCV.

[3]  Michele Volpi,et al.  Dense Semantic Labeling of Subdecimeter Resolution Images With Convolutional Neural Networks , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[4]  Markus Gerke,et al.  The ISPRS benchmark on urban object classification and 3D building reconstruction , 2012 .

[5]  T. Oke,et al.  Local Climate Zones for Urban Temperature Studies , 2012 .

[6]  Xiao Xiang Zhu,et al.  Spatiotemporal scene interpretation of space videos via deep neural network and tracklet analysis , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[7]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[8]  Luisa Verdoliva,et al.  Land Use Classification in Remote Sensing Images by Convolutional Neural Networks , 2015, ArXiv.

[9]  Xiao Xiang Zhu,et al.  FusioNet: A two-stream convolutional neural network for urban scene classification using PolSAR and hyperspectral data , 2017, 2017 Joint Urban Remote Sensing Event (JURSE).

[10]  Uwe Stilla,et al.  Classification With an Edge: Improving Semantic Image Segmentation with Boundary Detection , 2016, ISPRS Journal of Photogrammetry and Remote Sensing.

[11]  Jamie Sherrah,et al.  Fully Convolutional Networks for Dense Semantic Labelling of High-Resolution Aerial Imagery , 2016, ArXiv.

[12]  Jiangye Yuan,et al.  Automatic Building Extraction in Aerial Scenes Using Convolutional Networks , 2016, ArXiv.

[13]  Steffen Fritz,et al.  Contributing to WUDAPT: A Local Climate Zone Classification of Two Cities in Ukraine , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[14]  Geoffrey E. Hinton,et al.  Learning to Detect Roads in High-Resolution Aerial Images , 2010, ECCV.

[15]  Bertrand Le Saux,et al.  Fusion of heterogeneous data in convolutional networks for urban semantic labeling , 2017, 2017 Joint Urban Remote Sensing Event (JURSE).

[16]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[17]  Nikos Paragios,et al.  Simultaneous registration, segmentation and change detection from multisensor, multitemporal satellite image pairs , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[18]  Marius Leordeanu,et al.  Aerial image geolocalization from recognition and matching of roads and intersections , 2016, BMVC.

[19]  Pierre Alliez,et al.  Fully convolutional neural networks for remote sensing image classification , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[20]  Daniel Cremers,et al.  FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture , 2016, ACCV.

[21]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Ian D. Reid,et al.  RefineNet : MultiPath Refinement Networks with Identity Mappings for High-Resolution Semantic Segmentation , 2016 .

[23]  Michael Cramer,et al.  The DGPF-Test on Digital Airborne Camera Evaluation - Over- view and Test Design , 2010 .

[24]  Nikos Komodakis,et al.  Building detection in very high resolution multispectral data with deep learning features , 2015, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[25]  Hannes Taubenböck,et al.  Joint use of remote sensing data and volunteered geographic information for exposure estimation: evidence from Valparaíso, Chile , 2017, Natural Hazards.

[26]  Geoffrey E. Hinton,et al.  Machine Learning for Aerial Image Labeling , 2013 .

[27]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Alexandre Boulch,et al.  Processing of Extremely High-Resolution LiDAR and RGB Data: Outcome of the 2015 IEEE GRSS Data Fusion Contest–Part A: 2-D Contest , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[30]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Wolfram Burgard,et al.  Multimodal deep learning for robust RGB-D object recognition , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[32]  Jiaoyan Chen,et al.  DeepVGI: Deep Learning with Volunteered Geographic Information , 2017, WWW.

[33]  Marco Minghini,et al.  Using OpenStreetMap to Create Land Use and Land Cover Maps , 2017 .

[34]  Ian D. Reid,et al.  RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.