Predicting Landsat Reflectance with Deep Generative Fusion

Public satellite missions are commonly bound to a trade-off between spatial and temporal resolution as no single sensor provides fine-grained acquisitions with frequent coverage. This hinders their potential to assist vegetation monitoring or humanitarian actions, which require detecting rapid and detailed terrestrial surface changes. In this work, we probe the potential of deep generative models to produce high-resolution optical imagery by fusing products with different spatial and temporal characteristics. We introduce a dataset of co-registered Moderate Resolution Imaging Spectroradiometer (MODIS) and Landsat surface reflectance time series and demonstrate the ability of our generative model to blend coarse daily reflectance information into low-paced finer acquisitions. We benchmark our proposed model against state-of-the-art reflectance fusion algorithms.

[1]  Konrad Schindler,et al.  Super-Resolution of Sentinel-2 Images: Learning a Globally Applicable Deep Neural Network , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[2]  Xiaolin Zhu,et al.  An enhanced spatial and temporal adaptive reflectance fusion model for complex heterogeneous regions , 2010 .

[3]  Peijuan Wang,et al.  Fusing Landsat and MODIS Data for Vegetation Monitoring , 2015, IEEE Geoscience and Remote Sensing Magazine.

[4]  Yoshua Bengio,et al.  HighRes-net: Multi-Frame Super-Resolution by Recursive Fusion , 2019 .

[5]  Yoshua Bengio,et al.  HighRes-net: Recursive Fusion for Multi-Frame Super-Resolution of Satellite Imagery , 2020, ArXiv.

[6]  Michael Schmitt,et al.  A Conditional Generative Adversarial Network to Fuse Sar And Multispectral Optical Data For Cloud Removal From Sentinel-2 Images , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[7]  David M. Szpakowski,et al.  A Review of the Applications of Remote Sensing in Fire Ecology , 2019, Remote. Sens..

[8]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[9]  Lillian Kay Petersen,et al.  Real-Time Prediction of Crop Yields From MODIS Relative Vegetation Health: A Continent-Wide Analysis of Africa , 2018, Remote. Sens..

[10]  Edward J. Knight,et al.  Landsat-8 Operational Land Imager Design, Characterization and Performance , 2014, Remote. Sens..

[11]  J. Boardman,et al.  Discrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm , 1992 .

[12]  Paul D. Bates,et al.  Remote sensing and flood inundation modelling , 2004 .

[13]  Jevgenij Gamper,et al.  Conditional Denoising of Remote Sensing Imagery Using Cycle-Consistent Deep Generative Models , 2019, ArXiv.

[14]  K. Beurs,et al.  Evaluation of Landsat and MODIS data fusion products for analysis of dryland forest phenology , 2012 .

[15]  Martha C. Anderson,et al.  Landsat-8: Science and Product Vision for Terrestrial Global Change Research , 2014 .

[16]  David Berthelot,et al.  Creating High Resolution Images with a Latent Adversarial Generator , 2020, ArXiv.

[17]  Fernando França da Cunha,et al.  Fusion of MODIS and Landsat-Like Images for Daily High Spatial Resolution NDVI , 2020, Remote. Sens..

[18]  Vincent Dumoulin,et al.  Deconvolution and Checkerboard Artifacts , 2016 .

[19]  Mathew R. Schwaller,et al.  On the blending of the Landsat and MODIS surface reflectance: predicting daily Landsat surface reflectance , 2006, IEEE Transactions on Geoscience and Remote Sensing.

[20]  Robert E. Wolfe,et al.  A Landsat surface reflectance dataset for North America, 1990-2000 , 2006, IEEE Geoscience and Remote Sensing Letters.

[21]  Marc Rußwurm,et al.  Multi3Net: Segmenting Flooded Buildings via Fusion of Multiresolution, Multisensor, and Multitemporal Satellite Imagery , 2019, AAAI.

[22]  Thomas S. Pagano,et al.  Prelaunch characteristics of the Moderate Resolution Imaging Spectroradiometer (MODIS) on EOS-AM1 , 1998, IEEE Trans. Geosci. Remote. Sens..

[23]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[24]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[25]  Pablo J. Zarco-Tejada,et al.  Fieldcopter: unmanned aerial systems for crop monitoring services , 2013 .

[26]  Robert E. Wolfe,et al.  Key characteristics of MODIS data products , 1998, IEEE Trans. Geosci. Remote. Sens..

[27]  D. Baldocchi,et al.  Using data from Landsat, MODIS, VIIRS and PhenoCams to monitor the phenology of California oak/grass savanna and open grassland across spatial scales , 2017 .

[28]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[29]  Corentin Henry,et al.  SAR-to-Optical Image Translation Based on Conditional Generative Adversarial Networks - Optimization, Opportunities and Limits , 2019, Remote. Sens..

[30]  Enrico Magli,et al.  DeepSUM: Deep Neural Network for Super-Resolution of Unregistered Multitemporal Images , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[31]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[32]  Yue Yu,et al.  SAR-to-Optical Image Translation Using Supervised Cycle-Consistent Adversarial Networks , 2019, IEEE Access.

[33]  Joanne C. White,et al.  Generation of dense time series synthetic Landsat data through data blending with MODIS using a spatial and temporal adaptive reflectance fusion model. , 2009 .

[34]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Hongli Liu,et al.  An Improved STARFM with Help of an Unmixing-Based Method to Generate High Spatial and Temporal Resolution Remote Sensing Data in Complex Heterogeneous Regions , 2016, Sensors.

[36]  Xixi Lu,et al.  Application of Remote Sensing in Flood Management with Special Reference to Monsoon Asia: A Review , 2004 .