Spatial resolution enhancement method for Landsat imagery using a Generative Adversarial Network

ABSTRACT Landsat and Sentinel-2 are two freely accessible satellite data that are relevant for global land cover monitoring. However, the uses of the latter data set are growing because of its higher spatial resolutions and the availability of benchmark data sets for deep learning applications. In this study, we integrate a style transfer (perceptual loss estimation from Sentinel 2 benchmark data) into a Generative Adversarial Network (GAN) to construct a single image super-resolution model. The proposed model upscales Landsat 8 images (using red, green, blue, and near-infrared bands at 30 m and Panchromatic band 15 m for high-resolution features exploiting) to 10 m (with Sentinel-2 as reference). Compared to pan-sharpening and other upscaling methods, the proposed method can produce more realistic, spatial convincing images at 10 m resolution and more similar to Sentinel-2 images than the other commonly used super-resolution imaging algorithms. As a result, the proposed method extends the usage of high-resolution benchmark data sets for lower resolution imagery to enrich supplement data sources for land cover classification.

[1]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[2]  C. Justice,et al.  The Harmonized Landsat and Sentinel-2 surface reflectance data set , 2018, Remote Sensing of Environment.

[3]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Jian Yang,et al.  Image Super-Resolution via Deep Recursive Residual Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[6]  Qian Du,et al.  Remote sensing images super-resolution with deep convolution networks , 2019, Multimedia Tools and Applications.

[7]  Steffen Fritz,et al.  A global dataset of crowdsourced land cover and land use reference data , 2016, Scientific Data.

[8]  Giuseppe Scarpa,et al.  Fast Super-Resolution of 20 m Sentinel-2 Bands Using Convolutional Neural Networks , 2019, Remote. Sens..

[9]  Tao Liu,et al.  Deep learning-based fusion of Landsat-8 and Sentinel-2 images for a harmonized surface reflectance product , 2019 .

[10]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[12]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[13]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Tong Tong,et al.  Image Super-Resolution Using Dense Skip Connections , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[15]  Andreas Dengel,et al.  EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[16]  Rasim Latifovic,et al.  Landsat Super-Resolution Enhancement Using Convolution Neural Networks and Sentinel-2 for Training , 2018, Remote. Sens..

[17]  Konrad Schindler,et al.  Super-Resolution of Sentinel-2 Images: Learning a Globally Applicable Deep Neural Network , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[18]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[20]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Peter M. Atkinson,et al.  Fusion of Landsat 8 OLI and Sentinel-2 MSI Data , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[22]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[23]  Yuqi Bai,et al.  Annual dynamics of global land cover and its long-term changes from 1982 to 2015 , 2020, Earth System Science Data.

[24]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.