Cloud Removal from Satellite Images using Spatiotemporal Generator Networks

Satellite images hold great promise for continuous environmental monitoring and earth observation. Occlusions cast by clouds, however, can severely limit coverage, making ground information extraction more difficult. Existing pipelines typically perform cloud removal with simple temporal composites and hand-crafted filters. In contrast, we cast the problem of cloud removal as a conditional image synthesis challenge, and we propose a trainable spatiotemporal generator network (STGAN) to remove clouds. We train our model on a new large-scale spatiotemporal dataset that we construct, containing 97640 image pairs covering all continents. We demonstrate experimentally that the proposed STGAN model outperforms standard models and can generate realistic cloud-free images with high PSNR and SSIM values across a variety of atmospheric conditions, leading to improved performance in downstream tasks such as land cover classification.

[1]  Sang Michael Xie,et al.  Combining satellite imagery and machine learning to predict poverty , 2016, Science.

[2]  Tushar Sandhan,et al.  Simultaneous Detection and Removal of High Altitude Clouds from an Image , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[3]  Sylvie Le Hégarat-Mascle,et al.  Use of Markov Random Fields for automatic cloud/shadow detection on high resolution optical images , 2009 .

[4]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[5]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[6]  Jenq-Neng Hwang,et al.  Single Image Snow Removal via Composition Generative Adversarial Networks , 2019, IEEE Access.

[7]  Andreas Dengel,et al.  Contextual Enrichment of Remote-Sensed Events with Social Media Streams , 2016, ACM Multimedia.

[8]  Masashi Matsuoka,et al.  Filmy Cloud Removal on Satellite Imagery with Multispectral Conditional Generative Adversarial Nets , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[9]  Vishal M. Patel,et al.  Image De-Raining Using a Conditional Generative Adversarial Network , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[10]  Wenhan Yang,et al.  Attentive Generative Adversarial Network for Raindrop Removal from A Single Image , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[11]  Andreas Uhl,et al.  Improvement of the Fmask algorithm for Sentinel-2 images: Separating clouds from bright surfaces based on parallax effects , 2018, Remote Sensing of Environment.

[12]  Ken Perlin,et al.  [Computer Graphics]: Three-Dimensional Graphics and Realism , 2022 .

[13]  Nataliia Kussul,et al.  Deep Learning Classification of Land Cover and Crop Types Using Remote Sensing Data , 2017, IEEE Geoscience and Remote Sensing Letters.

[14]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[15]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[16]  Chun-Liang Chien,et al.  Automatic cloud removal from multi-temporal SPOT images , 2008, Appl. Math. Comput..

[17]  Andreas Dengel,et al.  EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[18]  W. Paul Menzel,et al.  Spatial and Temporal Distribution of Clouds Observed by MODIS Onboard the Terra and Aqua Satellites , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[19]  Kalinka Regina Lucas Jaquie Castelo Branco,et al.  Precision Agriculture: Using Low-Cost Systems to Acquire Low-Altitude Images , 2016, IEEE Computer Graphics and Applications.

[20]  Alexei A. Efros,et al.  Toward Multimodal Image-to-Image Translation , 2017, NIPS.

[21]  Yang Wang,et al.  A Remote Sensing Image Dataset for Cloud Removal , 2019, ArXiv.

[22]  Luis Guanter,et al.  Ready-to-Use Methods for the Detection of Clouds, Cirrus, Snow, Shadow, Water and Clear Sky Pixels in Sentinel-2 MSI Images , 2016, Remote. Sens..

[23]  Nikos Komodakis,et al.  Building detection in very high resolution multispectral data with deep learning features , 2015, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[24]  Olivier Arino,et al.  Ten-Meter Sentinel-2A Cloud-Free Composite - Southern Africa 2016 , 2017, Remote. Sens..

[25]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  E. Helmer,et al.  Cloud-Free Satellite Image Mosaics with Regression Trees and Histogram Matching. , 2005 .

[27]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[28]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Atsushi Hashimoto,et al.  Restoration of Sea Surface Temperature Satellite Images Using a Partially Occluded Training Set , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).