Local climate zone classification using a multi-scale, multi-level attention network

Abstract Local Climate Zones (LCZ) offer a climate-aware and standardized classification scheme composed of 17 urban and natural landscape classes. Recent deep learning-based LCZ classification studies have adopted a scene classification approach with computer vision-inspired models. In light of these advancements, this study introduces a multi-scale, multi-level attention network (MSMLA-Net) for deep learning-based LCZ classification. MSMLA-Net integrates a multi-scale (MS) module to generate multi-scale features from the input data and a novel multi-level attention (MLA) module as a branch unit from the model’s main ResNet backbone. MLA uses the convolutional block attention module (CBAM) at multiple stages to generate multi-level spatially and spectrally enhanced features for context aggregation. This study presents comprehensive model-based experiments on model depth, the individual and combined influence of MS and MLA modules, and the addition of attention mechanisms. Furthermore, data-based experiments are conducted to determine optimal Sentinel-2 spectral bands, while OpenStreetMap (OSM) building data, ALOS World 3D DSM height information, and a national land cover map are included as ancillary bands. LCZ classification tests are conducted on six major cities in South Korea. With regards to model-based performance, MSMLA-Net was built using a modified SE-ResNet50 backbone (MSMLA-50) and obtained the best classification results using 48 by 48-pixel input patches with a combination of all Sentinel-2 and ancillary bands, outperforming three state-of-the-art LCZ classification models. In particular, only MSMLA-50 reached over 70% built-up overall accuracy and maintained high accuracy even when tested on completely unseen areas in a “citywise” leave-one-out sampling strategy. For data-based results, the combination of Sentinel-2 Red Edge bands were helpful, mainly due to the greater number of available bands. Training only ancillary data generated up to 75.0% for built-up overall accuracy, albeit at the cost of low natural overall accuracy. Using all available bands and ancillary data produced the best results, but a combination of only OSM and Sentinel-2 bands also generated comparable accuracy. Ultimately, the proposed MSMLA-Net bridges advanced computer vision techniques such as attention mechanisms and multi-level context aggregation to improve LCZ classification. The MS and MLA modules can be applied on different backbones, sophisticated sampling schemes, and data inputs to flexibly achieve improved classification results.

[1]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  Gerald Mills,et al.  Cities as agents of global change , 2007 .

[3]  Min Wang,et al.  Context Aggregation Network for Semantic Labeling in Aerial Images , 2019, Remote. Sens..

[4]  Christopher. Simons,et al.  Machine learning with Python , 2017 .

[5]  T. Oke,et al.  Local Climate Zones for Urban Temperature Studies , 2012 .

[6]  Alexandr A. Kalinin,et al.  Albumentations: fast and flexible image augmentations , 2018, Inf..

[7]  Monika Kuffer,et al.  Object-based image analysis to map local climate zones: The case of Bandung, Indonesia , 2019, Applied Geography.

[8]  Iain Stewart,et al.  Mapping Local Climate Zones for a Worldwide Database of the Form and Function of Cities , 2015, ISPRS Int. J. Geo Inf..

[9]  Alexander Kmoch,et al.  Vertical Accuracy of Freely Available Global Digital Elevation Models (ASTER, AW3D30, MERIT, TanDEM-X, SRTM, and NASADEM) , 2020, Remote. Sens..

[10]  Frieke Van Coillie,et al.  Quality of Crowdsourced Data on Urban Morphology—The Human Influence Experiment (HUMINEX) , 2017 .

[11]  Xiao Xiang Zhu,et al.  AGGREGATING CLOUD-FREE SENTINEL-2 IMAGES WITH GOOGLE EARTH ENGINE , 2019, ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[12]  Benjamin Bechtel,et al.  Urban climate zone classification using convolutional neural network and ground-level images , 2019, Progress in Physical Geography: Earth and Environment.

[13]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[14]  Xiao Xiang Zhu,et al.  Local climate zone-based urban land cover classification from multi-seasonal Sentinel-2 images with a recurrent residual network , 2019, ISPRS journal of photogrammetry and remote sensing : official publication of the International Society for Photogrammetry and Remote Sensing.

[15]  Xiao Xiang Zhu,et al.  Feature Importance Analysis of Sentinel-2 Imagery for Large-Scale Urban Local Climate Zone Classification , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[16]  Iryna Dronova,et al.  Urban Landscape Change Analysis Using Local Climate Zones and Object-Based Classification in the Salt Lake Metro Region, Utah, USA , 2019, Remote. Sens..

[17]  Cheolhee Yoo,et al.  Improving Local Climate Zone Classification Using Incomplete Building Data and Sentinel 2 Images Based on Convolutional Neural Networks , 2020, Remote Sensing.

[18]  Jianguo Tan,et al.  The urban heat island and its impact on heat waves and human health in Shanghai , 2010, International journal of biometeorology.

[19]  Takeo Tadono,et al.  Generation of the 30 M-MESH global digital surface model by alos prism , 2016 .

[20]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[21]  Benjamin Bechtel,et al.  Global transferability of local climate zone models , 2019, Urban Climate.

[22]  Xiao Xiang Zhu,et al.  Multilevel Feature Fusion-Based CNN for Local Climate Zone Classification From Sentinel-2 Images: Benchmark Results on the So2Sat LCZ42 Dataset , 2020, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[23]  Elie Bou-Zeid,et al.  Synergistic Interactions between Urban Heat Islands and Heat Waves: The Impact in Cities Is Larger than the Sum of Its Parts* , 2013 .

[24]  J. Dinesh Peter,et al.  Classification of Mammogram Images Using Multiscale all Convolutional Neural Network (MA-CNN) , 2019, Journal of Medical Systems.

[25]  C. Ren,et al.  Mapping the local climate zones of urban areas by GIS-based and WUDAPT methods: A case study of Hong Kong , 2017, Urban Climate.

[26]  Benjamin Bechtel,et al.  Multitemporal Landsat data for urban heat island assessment and classification of local climate zones , 2011, 2011 Joint Urban Remote Sensing Event.

[27]  Benjamin Bechtel,et al.  Classification of Local Climate Zones Based on Multiple Earth Observation Data , 2012, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[28]  Xiao Xiang Zhu,et al.  So2Sat LCZ42: A Benchmark Data Set for the Classification of Global Local Climate Zones [Software and Data Sets] , 2020, IEEE Geoscience and Remote Sensing Magazine.

[29]  Marc Rußwurm,et al.  Meta-Learning for Few-Shot Land Cover Classification , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[30]  Timothy R. Oke,et al.  Evaluation of the ‘local climate zone’ scheme using temperature observations and model simulations , 2014 .

[31]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Chao Ren,et al.  Mapping local climate zones for a Japanese large city by an extended workflow of WUDAPT Level 0 method , 2020 .

[33]  Qian Shi,et al.  Local climate zone mapping as remote sensing scene classification using deep learning: A case study of metropolitan China , 2020 .

[34]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Benjamin Bechtel,et al.  Classification of Local Climate Zones Using SAR and Multispectral Data in an Arid Environment , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[36]  Frieke Van Coillie,et al.  Influence of neighbourhood information on 'Local Climate Zone' mapping in heterogeneous cities , 2017, Int. J. Appl. Earth Obs. Geoinformation.

[37]  Johannes Rosentreter,et al.  Towards large-scale mapping of local climate zones using multitemporal Sentinel 2 data and convolutional neural networks , 2020 .

[38]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[39]  Linda See,et al.  Generating WUDAPT Level 0 data – Current status of production and evaluation , 2019, Urban Climate.

[40]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[41]  Xiao Xiang Zhu,et al.  Feature Importance Analysis for Local Climate Zone Classification Using a Residual Convolutional Neural Network with Multi-Source Datasets , 2018, Remote. Sens..

[42]  Sotiris Vardoulakis,et al.  The Urban Heat Island: Implications for Health in a Changing Environment , 2017, Current Environmental Health Reports.

[43]  Linda See,et al.  Using OpenStreetMap (OSM) to enhance the classification of local climate zones in the framework of WUDAPT , 2019, Urban Climate.

[44]  Cheolhee Yoo,et al.  Comparison between convolutional neural networks and random forest for local climate zone classification in mega urban areas using Landsat images , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[45]  Xiao Xiang Zhu,et al.  Fusion of Heterogeneous Earth Observation Data for the Classification of Local Climate Zones , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[46]  Lorenzo Bruzzone,et al.  Multi-Scale Context Aggregation for Semantic Segmentation of Remote Sensing Images , 2020, Remote. Sens..