Urban land cover classification with missing data using deep convolutional neural networks

Fusing different sensors with different data modalities is a common technique to improve land cover classification performance in remote sensing. However, all modalities are rarely available for all test data, and this missing data problem poses severe challenges for multi-modal learning. Inspired by recent successes in deep learning, we propose as a remedy a convolutional neural network architecture for urban remote sensing image segmentation trained on data modalities which are not all available at test time. We train our architecture with a cost function particularly suited for imbalanced classes, as this is a frequent problem in remote sensing. We demonstrate the method using a benchmark dataset containing RGB and DSM images. Assuming that the DSM images are missing during testing, our method outperforms both a CNN trained on RGB images as well as an ensemble of two CNNs trained on the RGB images, by exploiting the training time information of the missing modality.

[1]  Michael Cramer,et al.  The DGPF-Test on Digital Airborne Camera Evaluation - Over- view and Test Design , 2010 .

[2]  Bertrand Le Saux,et al.  Segment-before-Detect: Vehicle Detection and Classification through Semantic Segmentation of Aerial Images , 2017, Remote. Sens..

[3]  Markus Gerke,et al.  Use of the stair vision library within the ISPRS 2D semantic labeling benchmark (Vaihingen) , 2014 .

[4]  Michael Kampffmeyer,et al.  Semantic Segmentation of Small Objects and Modeling of Uncertainty in Urban Remote Sensing Images Using Deep Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[5]  Rob Fergus,et al.  Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[6]  Robert Jenssen,et al.  Land-cover classification of partly missing data using support vector machines , 2012 .

[7]  Arnt-Børre Salberg,et al.  Land Cover Classification of Cloud-Contaminated Multitemporal High-Resolution Images , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[8]  K. Koperski,et al.  Land cover classification with multi-sensor fusion of partly missing data. , 2009 .

[9]  Pierre Alliez,et al.  Fully convolutional neural networks for remote sensing image classification , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[10]  Trevor Darrell,et al.  Learning with Side Information through Modality Hallucination , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Francesca Bovolo,et al.  The Time Variable in Data Fusion: A Change Detection Perspective , 2015, IEEE Geoscience and Remote Sensing Magazine.

[12]  Joseph L Schafer,et al.  Analysis of Incomplete Multivariate Data , 1997 .

[13]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Carlo Gatta,et al.  Unsupervised Deep Feature Extraction for Remote Sensing Image Classification , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Gang Yang,et al.  Missing Information Reconstruction of Remote Sensing Data: A Technical Review , 2015, IEEE Geoscience and Remote Sensing Magazine.

[16]  Executive Summary.  World Urbanization Prospects: The 2018 Revision , 2019 .

[17]  Lorenzo Bruzzone,et al.  Earthquake Damage Assessment of Buildings Using VHR Optical and SAR Imagery , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[18]  Michele Volpi,et al.  Dense Semantic Labeling of Subdecimeter Resolution Images With Convolutional Neural Networks , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[19]  Raffay Hamid,et al.  Large-scale damage detection using satellite imagery , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Lorenzo Bruzzone,et al.  Domain adaptation based on deep denoising auto-encoders for classification of remote sensing images , 2016, Remote Sensing.

[21]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[22]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[23]  Jamie Sherrah,et al.  Effective semantic pixel labelling with convolutional networks and Conditional Random Fields , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[24]  Razvan Pascanu,et al.  Understanding the exploding gradient problem , 2012, ArXiv.

[25]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[26]  Jixian Zhang.  Multi-source remote sensing data fusion: status and trends , 2010 .

[27]  Bertrand Le Saux,et al.  Fusion of heterogeneous data in convolutional networks for urban semantic labeling , 2017, 2017 Joint Urban Remote Sensing Event (JURSE).

[28]  P. Bocquier.  WORLD URBANIZATION PROSPECTS: AN ALTERNATIVE TO THE UN MODEL OF PROJECTION COMPATIBLE WITH URBAN TRANSITION THEORY 1 , 2005 .

[29]  Aníbal R. Figueiras-Vidal,et al.  Pattern classification with missing data: a review , 2010, Neural Computing and Applications.

[30]  Taeho Jo,et al.  A Multiple Resampling Method for Learning from Imbalanced Data Sets , 2004, Comput. Intell..

[31]  Gabriele Moser,et al.  Large urban zone classification on SPOT-5 imagery with convolutional neural networks , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[32]  Jitendra Malik,et al.  Learning Rich Features from RGB-D Images for Object Detection and Segmentation , 2014, ECCV.

[33]  Alexandre Boulch,et al.  Benchmarking classification of earth-observation data: From learning explicit features to convolutional networks , 2015, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[34]  Gregoire Mercier,et al.  Self-Organizing Maps for Processing of Data with Missing Values and Outliers: Application to Remote Sensing Images , 2010 .

[35]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[36]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.