Multilevel Feature Fusion-Based CNN for Local Climate Zone Classification From Sentinel-2 Images: Benchmark Results on the So2Sat LCZ42 Dataset

As a unique classification scheme for urban forms and functions, the local climate zone (LCZ) system provides essential general information for any studies related to urban environments, especially on a large scale. Remote sensing data-based classification approaches are the key to large-scale mapping and monitoring of LCZs. The potential of deep learning-based approaches is not yet fully explored, even though advanced convolutional neural networks (CNNs) continue to push the frontiers for various computer vision tasks. One reason is that published studies are based on different datasets, usually at a regional scale, which makes it impossible to fairly and consistently compare the potential of different CNNs for real-world scenarios. This article is based on the big So2Sat LCZ42 benchmark dataset dedicated to LCZ classification. Using this dataset, we studied a range of CNNs of varying sizes. In addition, we proposed a CNN to classify LCZs from Sentinel-2 images, Sen2LCZ-Net. Using this base network, we propose fusing multilevel features using the extended Sen2LCZ-Net-MF. With this proposed simple network architecture, and the highly competitive benchmark dataset, we obtain results that are better than those obtained by the state-of-the-art CNNs, while requiring less computation with fewer layers and parameters. Large-scale LCZ classification examples of completely unseen areas are presented, demonstrating the potential of our proposed Sen2LCZ-Net-MF as well as the So2Sat LCZ42 dataset. We also intensively investigated the influence of network depth and width, and the effectiveness of the design choices made for Sen2LCZ-Net-MF. This article will provide important baselines for future CNN-based algorithm developments for both LCZ classification and other urban land cover land use classification. Code and pretrained models are available at https://github.com/ChunpingQiu/benchmark-on-So2SatLCZ42-dataset-a-simple-tour.

[1]  Edward Ng,et al.  Spatial Variability of Geriatric Depression Risk in a High ‐ Density City : A Data ‐ Driven Socio ‐ Environmental Vulnerability Mapping , 2022 .

[2]  Xiao Xiang Zhu,et al.  Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources , 2017, IEEE Geoscience and Remote Sensing Magazine.

[3]  Xiao Xiang Zhu,et al.  SEN12MS - A Curated Dataset of Georeferenced Multi-Spectral Sentinel-1/2 Imagery for Deep Learning and Data Fusion , 2019, ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[4]  Daniel G. Aliaga,et al.  WUDAPT: An Urban Weather, Climate, and Environmental Modeling Infrastructure for the Anthropocene , 2018, Bulletin of the American Meteorological Society.

[5]  Hannes Taubenböck,et al.  TanDEM-X mission—new perspectives for the inventory and monitoring of global settlement patterns , 2012 .

[6]  Bin Chen,et al.  Stable classification with limited sample: transferring a 30-m resolution sample set collected in 2015 to mapping 10-m resolution global land cover in 2017. , 2019, Science bulletin.

[7]  Paolo Gamba,et al.  Urban Extent Extraction Combining Sentinel Data in the Optical and Microwave Range , 2019, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[8]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[9]  Iain Stewart,et al.  Mapping Local Climate Zones for a Worldwide Database of the Form and Function of Cities , 2015, ISPRS Int. J. Geo Inf..

[10]  Linda See,et al.  Generating WUDAPT Level 0 data – Current status of production and evaluation , 2019, Urban Climate.

[11]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[12]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.

[13]  Naoto Yokoya,et al.  Open Data for Global Multimodal Land Use Classification: Outcome of the 2017 IEEE GRSS Data Fusion Contest , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[14]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[15]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[16]  Xiao Xiang Zhu,et al.  Fusing Multi-Seasonal Sentinel-2 Images with Residual Convolutional Neural Networks for Local Climate Zone-Derived Urban Land Cover Classification , 2019, IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium.

[17]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Onkar Dikshit,et al.  SPECTRAL-SPATIAL CLASSIFICATION OF HYPERSPECTRAL REMOTE SENSING IMAGES USING VARIATIONAL AUTOENCODER AND CONVOLUTION NEURAL NETWORK , 2018, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[19]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Frieke Van Coillie,et al.  Quality of Crowdsourced Data on Urban Morphology—The Human Influence Experiment (HUMINEX) , 2017 .

[21]  Xavier Gastaldi,et al.  Shake-Shake regularization , 2017, ArXiv.

[22]  Paul Osmond,et al.  Understanding Land Surface Temperature Differences of Local Climate Zones Based on Airborne Remote Sensing Data , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[23]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[25]  Benjamin Bechtel,et al.  Global transferability of local climate zone models , 2019, Urban Climate.

[26]  Naoto Yokoya,et al.  Invariant Attribute Profiles: A Spatial-Frequency Joint Feature Extractor for Hyperspectral Image Classification , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[27]  T. Oke,et al.  Local Climate Zones for Urban Temperature Studies , 2012 .

[28]  Sven Behnke,et al.  Evaluation of Pooling Operations in Convolutional Architectures for Object Recognition , 2010, ICANN.

[29]  Quoc V. Le,et al.  Neural Architecture Search with Reinforcement Learning , 2016, ICLR.

[30]  Quoc V. Le,et al.  GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism , 2018, ArXiv.

[31]  Cheolhee Yoo,et al.  Comparison between convolutional neural networks and random forest for local climate zone classification in mega urban areas using Landsat images , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[32]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Jürgen Schmidhuber,et al.  Multi-column deep neural networks for image classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Timothy R. Oke,et al.  Evaluation of the ‘local climate zone’ scheme using temperature observations and model simulations , 2014 .

[35]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[36]  Georges Quénot,et al.  Coupled Ensembles of Neural Networks , 2017, 2018 International Conference on Content-Based Multimedia Indexing (CBMI).

[37]  Andreas Dengel,et al.  EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[38]  Xiao Xiang Zhu,et al.  So2Sat LCZ42: A Benchmark Data Set for the Classification of Global Local Climate Zones [Software and Data Sets] , 2020, IEEE Geoscience and Remote Sensing Magazine.

[39]  Ehsan Adeli,et al.  Towards Principled Design of Deep Convolutional Networks: Introducing SimpNet , 2018, ArXiv.

[40]  Sergey Levine,et al.  Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks , 2017, ICML.

[41]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Begüm Demir,et al.  Bigearthnet: A Large-Scale Benchmark Archive for Remote Sensing Image Understanding , 2019, IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium.

[43]  Daniel Fenner,et al.  Micro-Scale Variability of Air Temperature within a Local Climate Zone in Berlin, Germany, during Summer , 2018 .

[44]  Thomas Brox,et al.  Striving for Simplicity: The All Convolutional Net , 2014, ICLR.

[45]  S. Savić,et al.  Inter-/intra-zonal seasonal variability of the surface urban heat island based on local climate zones in three central European cities , 2019, Building and Environment.

[46]  Pierre Soille,et al.  Automated global delineation of human settlements from 40 years of Landsat satellite data archives , 2019, Big Earth Data.

[47]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[48]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Xiao Xiang Zhu,et al.  AGGREGATING CLOUD-FREE SENTINEL-2 IMAGES WITH GOOGLE EARTH ENGINE , 2019, ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[50]  Naoto Yokoya,et al.  Learnable manifold alignment (LeMA): A semi-supervised cross-modality learning framework for land cover and land use classification , 2019, ISPRS journal of photogrammetry and remote sensing : official publication of the International Society for Photogrammetry and Remote Sensing.

[51]  Thomas Esch,et al.  Urban Footprint Processor—Fully Automated Processing Chain Generating Settlement Masks From Global Data of the TanDEM-X Mission , 2013, IEEE Geoscience and Remote Sensing Letters.

[52]  Nassir Navab,et al.  Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks , 2018, MICCAI.

[53]  Yue Zhang,et al.  Effective Classification of Local Climate Zones Based on Multi-Source Remote Sensing Data , 2019, IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium.

[54]  Rajashree Kotharkar,et al.  Evaluating urban heat island in the critical local climate zones of an Indian city , 2018 .

[55]  Eberhard Parlow,et al.  Attribution of local climate zones using a multitemporal land use/land cover classification scheme , 2017 .

[56]  Zhiming Zhang,et al.  Revealing Kunming's (China) Historical Urban Planning Policies Through Local Climate Zones , 2019, Remote. Sens..

[57]  Benjamin Bechtel,et al.  Urban climate zone classification using convolutional neural network and ground-level images , 2019, Progress in Physical Geography: Earth and Environment.

[58]  Ronald Kemker,et al.  Low-Shot Learning for the Semantic Segmentation of Remote Sensing Imagery , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[59]  Xiao Xiang Zhu,et al.  Local climate zone-based urban land cover classification from multi-seasonal Sentinel-2 images with a recurrent residual network , 2019, ISPRS journal of photogrammetry and remote sensing : official publication of the International Society for Photogrammetry and Remote Sensing.

[60]  Petr Dobrovolný,et al.  Spatial modelling of summer climate indices based on local climate zones: expected changes in the future climate of Brno, Czech Republic , 2019, Climatic Change.

[61]  Naoto Yokoya,et al.  CoSpace: Common Subspace Learning From Hyperspectral-Multispectral Correspondences , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[62]  Benjamin Bechtel,et al.  Mapping Europe into local climate zones , 2019, PloS one.

[63]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[64]  Naoto Yokoya,et al.  Learning-Shared Cross-Modality Representation Using Multispectral-LiDAR and Hyperspectral Data , 2019, IEEE Geoscience and Remote Sensing Letters.

[65]  Xiao Xiang Zhu,et al.  Mining Hard Negative Samples for SAR-Optical Image Matching Using Generative Adversarial Networks , 2018, Remote. Sens..

[66]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[67]  Nikos Komodakis,et al.  Wide Residual Networks , 2016, BMVC.

[68]  Song Han,et al.  ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware , 2018, ICLR.

[69]  Johannes Rosentreter,et al.  Towards large-scale mapping of local climate zones using multitemporal Sentinel 2 data and convolutional neural networks , 2020 .

[70]  Mohammad Rouhani,et al.  Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures , 2016, ArXiv.

[71]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[72]  Kilian Q. Weinberger,et al.  Snapshot Ensembles: Train 1, get M for free , 2017, ICLR.

[73]  Xiao Xiang Zhu,et al.  So2Sat LCZ42: A Benchmark Dataset for Global Local Climate Zones Classification , 2019, ArXiv.

[74]  X. X. Zhu,et al.  A framework for large-scale mapping of human settlement extent from Sentinel-2 images via fully convolutional neural networks , 2020, ISPRS journal of photogrammetry and remote sensing : official publication of the International Society for Photogrammetry and Remote Sensing.

[75]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[76]  Xavier Gastaldi,et al.  Shake-Shake regularization of 3-branch residual networks , 2017, ICLR.

[77]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.