Reconstruction of Global Long-Term Gap-Free Daily Surface Soil Moisture from 2002 to 2020 Based on a Pixel-Wise Machine Learning Method

Global, long-term, gap-free, high quality soil moisture products are extremely important for hydrological monitoring and climate change research. However, soil moisture products produced from satellite observations have data gaps due to the limited capabilities of satellite orbit/swath and retrieval algorithms, which limit the regional and global applications of soil moisture data in hydrology and agriculture studies. To solve this problem, we proposed a gap-filling method to reconstruct a global gap-free surface soil moisture product by applying the machine learning (Random Forest) algorithm on a pixel-by-pixel basis, taking into account the nonlinear relationship between surface soil moisture and the related surface environmental variables. The gap-filling method was applied to the NN-SM surface soil moisture product, which has a fraction of data gaps of around 50% globally on a multi-year average. A global daily gap-free surface soil moisture dataset from 2002 to 2020 was then generated. The reconstructed values of several sub-regions after manually eliminating the original values were cross-verified with the original data, and this clearly demonstrated the reliability of the reconstruction method with the correlation coefficient (R) ranging between 0.770 and 0.918, the Root Mean Square Error (RMSE) between 0.057 and 0.082 m3/m3, the unbiased Root Mean Square Error (ubRMSE) between 0.053 and 0.081 m3/m3, and Bias between −0.012 and 0.008 m3/m3. The accuracy of the reconstructed surface soil moisture dataset was evaluated using in situ observations of surface soil moisture at 12 sites from the International Soil Moisture Network (ISMN) and the Long-Term Agroecosystem Research (LTAR) network, and the results showed good accuracy in terms of R (0.610), RMSE (0.067 m3/m3), ubRMSE (0.045 m3/m3) and Bias (0.031 m3/m3). Overall, the reconstructed surface soil moisture dataset retained the characteristics of the NN-SM product, such as high accuracy and good spatiotemporal pattern. However, with the advantage of continuous spatiotemporal coverage, it is more suitable for further applications in the analysis of global surface soil moisture trends, land surface hydrological processes, and land-atmosphere energy and water exchanges, etc.

[1]  L. Jia,et al.  A 21-year dataset (2000–2020) of gap-free global daily surface soil moisture at 1-km grid resolution , 2023, Scientific Data.

[2]  M. Menenti,et al.  Global soil moisture data fusion by Triple Collocation Analysis from 2011 to 2018 , 2022, Scientific Data.

[3]  Jiancheng Shi,et al.  Global spatiotemporally continuous MODIS land surface temperature dataset , 2022, Scientific data.

[4]  Massimo Menenti,et al.  Optimal Estimate of Global Biome - Specific Parameter Settings to Reconstruct NDVI Time Series with the Harmonic ANalysis of Time Series (HANTS) Method , 2021, Remote. Sens..

[5]  B. Minsley,et al.  The Biophysical Role of Water and Ice Within Permafrost Nearing Collapse: Insights From Novel Geophysical Observations , 2021, Journal of Geophysical Research: Earth Surface.

[6]  M. Cosh,et al.  A long term global daily soil moisture dataset derived from AMSR-E and AMSR2 (2002–2019) , 2021, Scientific data.

[7]  Q. Yuan,et al.  Generating seamless global daily AMSR2 soil moisture (SGD-SM) long-term products for the years 2013–2019 , 2021 .

[8]  Hao Sun,et al.  Evaluating Machine Learning and Geostatistical Methods for Spatial Gap-Filling of Monthly ESA CCI Soil Moisture in China , 2021, Remote. Sens..

[9]  Andreas Colliander,et al.  Reconciling Flagging Strategies for Multi-Sensor Satellite Soil Moisture Climate Data Records , 2020, Remote. Sens..

[10]  J. Thepaut,et al.  The ERA5 global reanalysis , 2020, Quarterly Journal of the Royal Meteorological Society.

[11]  Michela Taufer,et al.  Spatial Gap-Filling of ESA CCI Satellite-Derived Soil Moisture Based on Geostatistical Techniques and Multiple Regression , 2020, Remote. Sens..

[12]  Seyed Amir Naghibi,et al.  Machine learning approaches for spatial modeling of agricultural droughts in the south-east region of Queensland Australia. , 2020, The Science of the total environment.

[13]  D. Entekhabi,et al.  Terrestrial Evaporation and Moisture Drainage in a Warmer Climate , 2019, Geophysical Research Letters.

[14]  F. Yuan,et al.  Flash droughts characterization over China: From a perspective of the rapid intensification rate. , 2019, The Science of the total environment.

[15]  H. Lei,et al.  Generation of spatially complete and daily continuous surface soil moisture of high spatial resolution , 2019, Remote Sensing of Environment.

[16]  F. Yuan,et al.  Performance of Two Long-Term Satellite-Based and GPCC 8.0 Precipitation Products for Drought Monitoring over the Yellow River Basin in China , 2019, Sustainability.

[17]  Xiaoli Yang,et al.  Drought Monitoring and Evaluation by ESA CCI Soil Moisture Products Over the Yellow River Basin , 2019, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[18]  Qing Xiao,et al.  Accuracy Assessment on MODIS (V006), GLASS and MuSyQ Land-Surface Albedo Products: A Case Study in the Heihe River Basin, China , 2018, Remote. Sens..

[19]  Catherine Ottlé,et al.  Confronting Soil Moisture Dynamics from the ORCHIDEE Land Surface Model With the ESA-CCI Product: Perspectives for Data Assimilation , 2018, Remote. Sens..

[20]  Adriaan J. Teuling,et al.  A hot future for European droughts , 2018, Nature Climate Change.

[21]  Yaping Yang,et al.  Comparison of Different Machine Learning Approaches for Monthly Satellite-Based Soil Moisture Downscaling over Northeast China , 2017, Remote. Sens..

[22]  Yi Y. Liu,et al.  ESA CCI Soil Moisture for improved Earth system understanding : State-of-the art and future directions , 2017 .

[23]  J. Watts,et al.  A global satellite environmental data record derived from AMSR-E and AMSR2 microwave Earth observations , 2017 .

[24]  Wei Cheng,et al.  Evaluation of AMSR-E retrieval by detecting soil moisture decrease following massive dryland re-vegetation in the Loess Plateau, China , 2017 .

[25]  W. Wagner,et al.  Global-scale assessment and combination of SMAP with ASCAT (active) and AMSR2 (passive) soil moisture products , 2017 .

[26]  Kelly K. Caylor,et al.  Validation of SMAP surface soil moisture products with core validation sites , 2017, Remote Sensing of Environment.

[27]  Amen Al-Yaari,et al.  Rebuilding Long Time Series Global Soil Moisture Products Using the Neural Network Adopting the Microwave Vegetation Index , 2017, Remote. Sens..

[28]  Chun-Hsu Su,et al.  Does AMSR2 produce better soil moisture retrievals than AMSR-E over Australia? , 2017 .

[29]  M. Vohland,et al.  Downscaling land surface temperatures at regional scales with random forest regression , 2016 .

[30]  Giorgio Boni,et al.  Impact of different satellite soil moisture products on the predictions of a continuous distributed hydrological model , 2016, Int. J. Appl. Earth Obs. Geoinformation.

[31]  Jindi Wang,et al.  Spatially and Temporally Complete Satellite Soil Moisture Data Based on a Data Assimilation Method , 2016, Remote. Sens..

[32]  Yann Kerr,et al.  Assessment of the SMAP Passive Soil Moisture Product , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[33]  J. Zeng,et al.  Evaluation of remotely sensed and reanalysis soil moisture products over the Tibetan Plateau using in-situ observations , 2015 .

[34]  Seokhyeon Kim,et al.  A global comparison of alternate AMSR2 soil moisture products: Why do they differ? , 2015 .

[35]  Shengli Wu,et al.  Inter-Calibration of Satellite Passive Microwave Land Observations from AMSR-E and AMSR2 Using Overlapping FY3B-MWRI Sensor Measurements , 2014, Remote. Sens..

[36]  Wei Wu,et al.  Influences of soil moisture and vegetation on convective precipitation forecasts over the United States Great Plains , 2014 .

[37]  A. Al Bitar,et al.  Global-scale evaluation of two satellite-based passive microwave soil moisture datasets (SMOS and AMSR-E) with respect to Land Data Assimilation System estimates , 2014, Remote Sensing of Environment.

[38]  R.A.M. de Jeu,et al.  Using satellite based soil moisture to quantify the water driven variability in NDVI: A case study over mainland Australia , 2014 .

[39]  Lazhu,et al.  A MULTISCALE SOIL MOISTURE AND FREEZE-THAW MONITORING NETWORK ON THE THIRD POLE , 2013 .

[40]  Matthias Drusch,et al.  Global Automated Quality Control of In Situ Soil Moisture Data from the International Soil Moisture Network , 2013 .

[41]  Samuel E. Tuttle,et al.  A new approach for validating satellite estimates of soil moisture using large-scale precipitation: Comparing AMSR-E products , 2012 .

[42]  Zhongbo Su,et al.  Soil moisture mapping over the central part of the Tibetan Plateau using a series of ASAR WS images , 2012 .

[43]  Yi Liu,et al.  A three-dimensional gap filling method for large geophysical datasets: Application to global satellite soil moisture observations , 2012, Environ. Model. Softw..

[44]  José Martínez-Fernández,et al.  Validation of the SMOS L2 Soil Moisture Data in the REMEDHUS Network (Spain) , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[45]  Mario Chica-Olmo,et al.  An assessment of the effectiveness of a random forest classifier for land-cover classification , 2012 .

[46]  Yaoming Ma,et al.  The Tibetan plateau observatory of plateau scale soil moisture and soil temperature, Tibet - Obs, for quantifying uncertainties in coarse resolution satellite and model products , 2011 .

[47]  Hu Yang,et al.  The FengYun-3 Microwave Radiation Imager On-Orbit Verification , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[48]  Linna Chai,et al.  A new soil freeze/thaw discriminant algorithm using AMSR‐E passive microwave imagery , 2011 .

[49]  A. Robock,et al.  The International Soil Moisture Network: a data hosting facility for global in situ soil moisture measurements , 2011 .

[50]  Lixin Zhang,et al.  Estimate of Phase Transition Water Content in Freeze–Thaw Process Using Microwave Radiometer , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[51]  A. Kalra,et al.  Estimating soil moisture using remote sensing data: A machine learning approach , 2010 .

[52]  C. Ottlé,et al.  The AMMA-CATCH experiment in the cultivated Sahelian area of south-west Niger – Investigating water cycle response to a fluctuating climate and changing environment , 2009 .

[53]  Thierry Pellarin,et al.  Hydrological modelling and associated microwave emission of a semi-arid region in South-western Niger , 2009 .

[54]  Frédéric Baup,et al.  The AMMA-CATCH Gourma observatory site in Mali: Relating climatic variations to changes in vegetation, surface hydrology, fluxes and natural resources , 2009 .

[55]  P. Rosnay,et al.  Multi-scale soil moisture measurements at the Gourma meso-scale site in Mali , 2009 .

[56]  Jonathan Cheung-Wai Chan,et al.  Evaluation of random forest and adaboost tree-based ensemble classification and spectral band selection for ecotope mapping using airborne hyperspectral imagery , 2008 .

[57]  Kenneth G. Renard,et al.  Preface to special section on Fifty Years of Research and Data Collection: U.S. Department of Agriculture Walnut Gulch Experimental Watershed , 2008 .

[58]  D. Bosch,et al.  Little River Experimental Watershed, Tifton, Georgia, United States: A geographic database , 2007 .

[59]  G. Heathman,et al.  Temporal stability of surface soil moisture in the Little Washita River watershed and its applications in satellite soil moisture product validation , 2006 .

[60]  Ralf Ludwig,et al.  Derivation of surface soil moisture from ENVISAT ASAR wide swath and image mode data in agricultural areas , 2006, IEEE Transactions on Geoscience and Remote Sensing.

[61]  Faisal Hossain,et al.  Numerical investigation of the impact of uncertainties in satellite rainfall estimation and land surface model parameters on simulation of soil moisture , 2005 .

[62]  P. Dirmeyer,et al.  Comparison, Validation, and Transferability of Eight Multiyear Global Soil Wetness Products , 2004 .

[63]  Thomas J. Jackson,et al.  Soil moisture retrieval from AMSR-E , 2003, IEEE Trans. Geosci. Remote. Sens..

[64]  S. Kanae,et al.  Global Hydrological Cycle and World Water Resources , 2003 .

[65]  D. Roy,et al.  The MODIS Land product quality assessment approach , 2002 .

[66]  I. Sandholt,et al.  A simple interpretation of the surface temperature/vegetation index space for assessment of surface moisture status , 2002 .

[67]  Clayton L. Hanson,et al.  Long‐Term Soil Water Content Database, Reynolds Creek Experimental Watershed, Idaho, United States , 2001 .

[68]  A. Robock,et al.  The Global Soil Moisture Data Bank , 2000 .

[69]  W. Verhoef,et al.  Reconstructing cloudfree NDVI composites using Fourier analysis of time series , 2000 .