Sentinel-1 and Sentinel-2 Spatio-Temporal Data Fusion for Clouds Removal

The abundance of clouds, located both spatially and temporally, often makes remote sensing applications with optical images difficult or even impossible. In this manuscript a novel method for clouds-corrupted optical images restoration has been presented and developed, based on a joint data fusion paradigm, where three deep neural networks have been combined in order to fuse spatio-temporal features extracted from Sentinel-1 and Sentinel-2 time-series of data. It is worth to highlight that both the code and the dataset have been implemented from scratch and made available to interested research for further analysis and investigation.

[1]  Silvia Liberata Ullo,et al.  Automatic dataset builder for Machine Learning applications to satellite imagery , 2020, SoftwareX.

[2]  Qiangqiang Yuan,et al.  Thick cloud and cloud shadow removal in multitemporal imagery using progressively spatio-temporal patch group deep learning , 2020, ISPRS Journal of Photogrammetry and Remote Sensing.

[3]  Chao Zeng,et al.  Missing Data Reconstruction in Remote Sensing Image With a Unified Spatial–Temporal–Spectral Deep Convolutional Neural Network , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[4]  Heng Pan.  Cloud Removal for Remote Sensing Imagery via Spatial Attention Generative Adversarial Network , 2020, ArXiv.

[5]  Robert Eckardt,et al.  Removal of Optically Thick Clouds from Multi-Spectral Satellite Images Using Multi-Frequency SAR Data , 2013, Remote. Sens..

[6]  Nikos Komodakis,et al.  Cloud-Gan: Cloud Removal for Sentinel-2 Imagery Using a Cyclic Consistent Generative Adversarial Networks , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[7]  Xiaoxiang Zhu,et al.  Multisensor Data Fusion for Cloud Removal in Global and All-Season Sentinel-2 Imagery , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[8]  Stefano Ermon,et al.  Cloud Removal from Satellite Images using Spatiotemporal Generator Networks , 2020 .

[9]  Xiao Xiang Zhu,et al.  Cloud removal in Sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion , 2020, ISPRS journal of photogrammetry and remote sensing : official publication of the International Society for Photogrammetry and Remote Sensing.

[10]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[11]  Silvia Liberata Ullo,et al.  Paradigm selection for Data Fusion of SAR and Multispectral Sentinel data applied to Land-Cover Classification , 2021, ArXiv.

[12]  Nima Tajbakhsh,et al.  UNet++: A Nested U-Net Architecture for Medical Image Segmentation , 2018, DLMIA/ML-CDS@MICCAI.

[13]  Antonio J. Plaza,et al.  Thin Cloud Removal Based on Signal Transmission Principles and Spectral Mixture Analysis , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[14]  Naoto Yokoya,et al.  Multi-temporal Sentinel-1 and -2 Data Fusion for Optical Image Simulation , 2018, ISPRS Int. J. Geo Inf..

[15]  Chao-Hung Lin,et al.  Cloud Removal From Multitemporal Satellite Images Using Information Cloning , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[16]  Hai Zhang,et al.  Cloud Removal with Fusion of High Resolution Optical and SAR Images Using Generative Adversarial Networks , 2020, Remote. Sens..

[17]  Xiaoyi Li,et al.  Thin cloud removal from remote sensing images using multidirectional dual tree complex wavelet transform and transfer least square support vector regression , 2015 .

[18]  Michael Schmitt,et al.  A Conditional Generative Adversarial Network to Fuse Sar And Multispectral Optical Data For Cloud Removal From Sentinel-2 Images , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[19]  C. Woodcock,et al.  Continuous change detection and classification of land cover using all available Landsat data , 2014 .

[20]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[21]  Stefan Oehmcke,et al.  Creating cloud-free satellite imagery from image time series with deep learning , 2020, BigSpatial@SIGSPATIAL.

[22]  Olivier Arino,et al.  Ten-Meter Sentinel-2A Cloud-Free Composite - Southern Africa 2016 , 2017, Remote. Sens..

[23]  Hyeungu Choi,et al.  Cloud detection in Landsat imagery of ice sheets using shadow matching technique and automatic normalized difference snow index threshold value decision , 2004 .

[24]  Mark R. Pickering,et al.  Thin cloud removal from optical remote sensing images using the noise-adjusted principal components transform , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[25]  Xinghua Li,et al.  Cloud removal in remote sensing images using nonnegative matrix factorization and error correction , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[26]  Yong Wang,et al.  An empirical and radiative transfer model based algorithm to remove thin clouds in visible bands , 2016 .

[27]  Liangpei Zhang,et al.  Sparse-based reconstruction of missing information in remote sensing images from spectral/temporal complementary information , 2015 .

[28]  Mohammad Reza Mohammadi,et al.  Cloud Removal in Remote Sensing Images Using Generative Adversarial Networks and SAR-to-Optical Image Translation , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[29]  Nima Tajbakhsh,et al.  UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation , 2020, IEEE Transactions on Medical Imaging.

[30]  Tianxing Wang,et al.  Detection and Removal of Clouds and Associated Shadows in Satellite Imagery Based on Simulated Radiance Fields , 2019, Journal of Geophysical Research: Atmospheres.

[31]  Fred A. Kruse,et al.  The Spectral Image Processing System (SIPS) - Interactive visualization and analysis of imaging spectrometer data , 1993 .