Effect of missing data on short time series and their application in the characterization of surface temperature by detrended fluctuation analysis

Abstract Climate change is deeply impacting the society on different scales. Decision making becomes a complex task when, in adverse weather conditions, the meteorological records show missing data due to failures of measuring instruments. Several investigations have proposed optimized regression methods, K-nearest-neighbor imputation, and multiple imputations for the treatment of missing data; however, there is less information about the application of imputation methods for the treatment of missing data on short meteorological records. Therefore, the expected confidence in the results requires using robust analysis methods that depend the least as possible on the length of the records and the number of missing data. In this research, the performance of detrended fluctuation analysis applied on temperature short record was studied when K-nearest-neighbor and neural networks are used as imputation techniques, and compared with the performance without data imputation. The results showed the robustness when it is applied to a short time series with missing data and without data imputation. In this aspect, the DFA method only requires removing the seasonality from the temperature records to get good performance.

[1]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[2]  B. Bonaccorso,et al.  A new approach for processing climate missing databases applied to daily rainfall data in Soummam watershed, Algeria , 2019, Heliyon.

[3]  Robert J. Kuligowski,et al.  USING ARTIFICIAL NEURAL NETWORKS TO ESTIMATE MISSING RAINFALL DATA 1 , 1998 .

[4]  M. Mahdavi,et al.  NEURAL NETWORK APPLICATION FOR MONTHLY PRECIPITATION DATA RECONSTRUCTION , 2011 .

[5]  S. Havlin,et al.  Indication of a Universal Persistence Law Governing Atmospheric Variability , 1998 .

[6]  Gustavo E. A. P. A. Batista,et al.  An analysis of four missing data treatment methods for supervised learning , 2003, Appl. Artif. Intell..

[7]  J. A. Martín-Fernández,et al.  A modified EM alr-algorithm for replacing rounded zeros in compositional data sets , 2008, Comput. Geosci..

[8]  J. Feynman.  Has solar variability caused climate change that affected human culture , 2007 .

[9]  Éva Mekis,et al.  An Overview of the Second Generation Adjusted Daily Precipitation Dataset for Trend Analysis in Canada , 2011, Data, Models and Analysis.

[10]  Shlomo Havlin,et al.  Power-law persistence in the atmosphere and in the oceans , 2002 .

[11]  Jon D. Pelletier,et al.  Analysis and Modeling of the Natural Variability of Climate , 1997 .

[12]  Dennis Trewin.  The evolution of national statistical systems: Trends and implications , 2007 .

[13]  Zhenbin Wu,et al.  Investigation on the adsorption of phosphorus in all fractions from sediment by modified maifanite , 2018, Scientific Reports.

[14]  C. Fuentes,et al.  Spatial and temporal Hurst exponent variability of rainfall series based on the climatological distribution in a semiarid region in Mexico , 2018, Atmósfera.

[15]  Thomas R. H. Holmes,et al.  Estimating the soil temperature profile from a single depth observation: A simple empirical heatflow solution , 2008 .

[16]  Peter Filzmoser,et al.  Imputation of values above an upper detection limit in compositional data , 2020, Comput. Geosci..

[17]  A. Speranza,et al.  Annual and semiannual cycles of midlatitude near-surface temperature and tropospheric baroclinicity: reanalysis data and AOGCM simulations , 2017 .

[18]  E. Friis-christensen,et al.  Length of the Solar Cycle: An Indicator of Solar Activity Closely Associated with Climate , 1991, Science.

[19]  Shlomo Havlin,et al.  Analysis of daily temperature fluctuations , 1996 .

[20]  Luca Martino,et al.  Physics-aware Gaussian processes in remote sensing , 2018, Appl. Soft Comput..

[21]  M. Hutchinson,et al.  A comparison of two statistical methods for spatial interpolation of Canadian monthly mean climate data , 2000 .

[22]  H. Stanley,et al.  Multifractal Detrended Fluctuation Analysis of Nonstationary Time Series , 2002, physics/0202070.

[23]  Philip J. Sallis,et al.  A machine-learning algorithm for wind gust prediction , 2011, Comput. Geosci..

[24]  L. Bodri.  Short-term climate variability and its stochastic modeling , 1995 .

[25]  Gary King,et al.  Amelia II: A Program for Missing Data , 2011 .

[26]  P. Hopke,et al.  Application of modified alternating least squares regression to spectroscopic image analysis , 2003 .

[27]  Gerald R. North,et al.  Surface Temperature Reconstructions for the Last 1000 Years , 2006 .

[28]  P. Jones,et al.  Uncertainty estimates in regional and global observed temperature changes: A new data set from 1850 , 2006 .

[29]  L. AlejandrodelPozo,et al.  Mapa agroclimático de Chile. , 1989 .

[30]  J. R. Wallis,et al.  Robustness of the rescaled range R/S in the measurement of noncyclic long run statistical dependence , 1969 .

[31]  A Bunde,et al.  Power-law persistence and trends in the atmosphere: a detailed study of long temperature records. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[32]  Michael Schulz,et al.  Spectrum: spectral analysis of unevenly spaced paleoclimatic time series , 1997 .

[33]  Balamurali Saminathan,et al.  An investigation on the relationship between the Hurst exponent and the predictability of a rainfall time series , 2019, Meteorological Applications.

[34]  C. Peng,et al.  Mosaic organization of DNA nucleotides. , 1994, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[35]  H. Stanley,et al.  Effect of trends on detrended fluctuation analysis. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[36]  K. Fraedrich.  FICKIAN DIFFUSION AND NEWTONIAN COOLING: A CONCEPT FOR NOISE INDUCED CLIMATE VARIABILITY WITH LONG-TERM MEMORY? , 2002 .

[37]  Ali Reza Nafarzadegan,et al.  A modified distance-weighted approach for filling annual precipitation gaps: application to different climates of Iran , 2014, Theoretical and Applied Climatology.

[38]  Chris G. Tzanis,et al.  Multifractal Detrended Cross-Correlation Analysis of Global Methane and Temperature , 2020, Remote. Sens..

[39]  Steven G. Johnson,et al.  FFTW: an adaptive software architecture for the FFT , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[40]  K. Torre,et al.  Fractal analyses for 'short' time series: A re-assessment of classical methods , 2006 .

[41]  Yan Liu,et al.  Recurrent Neural Networks for Multivariate Time Series with Missing Values , 2016, Scientific Reports.

[42]  M. L. Kurnaz.  APPLICATION OF DETRENDED FLUCTUATION ANALYSIS TO MONTHLY AVERAGE OF THE MAXIMUM DAILY TEMPERATURES TO RESOLVE DIFFERENT CLIMATES , 2004 .

[43]  John O. Carter,et al.  Using spatial interpolation to construct a comprehensive archive of Australian climate data , 2001, Environ. Model. Softw..

[44]  M. Ausloos.  Statistical physics in meteorology , 2004, physics/0401066.

[45]  C. Ramírez,et al.  A bioclimatic classification of Chile: woodland communities in the temperate zone , 1998, Plant Ecology.

[46]  Gustau Camps-Valls,et al.  Latent force models for earth observation time series prediction , 2016, 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP).

[47]  J. Shukla,et al.  Predictability in the midst of chaos: A scientific basis for climate forecasting , 1998, Science.

[48]  Jianbo Gao,et al.  Multifractal analysis of sunspot time series: the effects of the 11-year cycle and Fourier truncation , 2009 .

[49]  Monidipa Das,et al.  A Deep-Learning-Based Forecasting Ensemble to Predict Missing Data for Remote Sensing Analysis , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[50]  Talkner,et al.  Power spectrum and detrended fluctuation analysis: application to daily temperatures , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[51]  I. Kondor,et al.  Statistical analysis of 5 s index data of the Budapest Stock Exchange , 1999 .

[52]  Leonardo Franco,et al.  Missing data imputation using statistical and machine learning methods in a real breast cancer problem , 2010, Artif. Intell. Medicine.

[53]  W. Hildreth,et al.  Laguna del Maule Volcanic Field: Eruptive history of a Quaternary basalt-to-rhyolite distributed volcanic field on the Andean rangecrest in central Chile , 2010 .

[54]  J. G. Contreras,et al.  Performance of multifractal detrended fluctuation analysis on short time series. , 2013, Physical review. E, Statistical, nonlinear, and soft matter physics.

[55]  Despina Deligiorgi,et al.  Multifractal scaling properties of daily air temperature time series , 2017 .

[56]  Chao Zeng,et al.  Missing Data Reconstruction in Remote Sensing Image With a Unified Spatial–Temporal–Spectral Deep Convolutional Neural Network , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[57]  Harri Niska,et al.  Methods for imputation of missing values in air quality data sets , 2004 .

[58]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[59]  A. Philippe,et al.  Generators of long-range dependent processes: A survey , 2003 .

[60]  P. Sarricolea,et al.  Climatic regionalisation of continental Chile , 2017 .

[61]  Crowley,et al.  Atmospheric science: Methane rises from wetlands , 2011, Nature.

[62]  John A. Eddy,et al.  The Maunder Minimum , 1976, Science.

[63]  E. Bacry,et al.  Characterizing long-range correlations in DNA sequences from wavelet analysis. , 1995, Physical review letters.

[64]  Kenneth G. Hubbard,et al.  The Effects of Data Gaps on the Calculated Monthly Mean Maximum and Minimum Temperatures in the Continental United States: A Spatial and Temporal Study , 1999 .

[65]  P. Abry,et al.  The wavelet based synthesis for fractional Brownian motion , 1996 .

[66]  Shlomo Havlin,et al.  Multifractality of river runoff and precipitation: Comparison of fluctuation analysis and wavelet methods , 2003 .

[67]  Manfred Mudelsee,et al.  REDFIT-X: Cross-spectral analysis of unevenly spaced paleoclimate time series , 2016, Comput. Geosci..

[68]  B. D. Clader,et al.  Climate sensitivity of the Earth to solar irradiance , 2002 .

[69]  D. G. Regulwar,et al.  ARTIFICIAL NEURAL NETWORK METHOD FOR ESTIMATION OF MISSING DATA , 2012, International Journal of Advanced Technology in Civil Engineering.

[70]  Mehran Amiri,et al.  Missing data imputation using fuzzy-rough methods , 2016, Neurocomputing.

[71]  Marcel Ausloos,et al.  Statistical physics in foreign exchange currency and stock markets , 2000 .

[72]  Ramesh S. V. Teegavarapu,et al.  Improved weighting methods, deterministic and stochastic data-driven models for estimation of missing precipitation records , 2005 .

[73]  Patrick Minnis,et al.  Forcings and chaos in interannual to decadal climate change , 1997 .