Comparison of stochastic and machine learning methods for multi-step ahead forecasting of hydrological processes

Research within the field of hydrology often focuses on the statistical problem of comparing stochastic to machine learning (ML) forecasting methods. The performed comparisons are based on case studies, while a study providing large-scale results on the subject is missing. Herein, we compare 11 stochastic and 9 ML methods regarding their multi-step ahead forecasting properties by conducting 12 extensive computational experiments based on simulations. Each of these experiments uses 2000 time series generated by linear stationary stochastic processes. We conduct each simulation experiment twice; the first time using time series of 100 values and the second time using time series of 300 values. Additionally, we conduct a real-world experiment using 405 mean annual river discharge time series of 100 values. We quantify the forecasting performance of the methods using 18 metrics. The results indicate that stochastic and ML methods may produce equally useful forecasts.

[1]  Jun Guo,et al.  Monthly streamflow forecasting based on improved support vector machine model , 2011, Expert Syst. Appl..

[2]  C. Sutton Classification and Regression Trees, Bagging, and Boosting , 2005 .

[3]  Ping-Feng Pai,et al.  A recurrent support vector regression model in rainfall forecasting , 2007 .

[4]  M. Taqqu,et al.  Fractionally differenced ARIMA models applied to hydrologic time series: Identification, estimation, and simulation , 1997 .

[5]  F. Pappenberger,et al.  Communicating uncertainty in hydro‐meteorological forecasts: mission impossible? , 2010 .

[6]  Juan B. Valdés,et al.  NONLINEAR MODEL FOR DROUGHT FORECASTING BASED ON A CONJUNCTION OF WAVELET TRANSFORMS AND NEURAL NETWORKS , 2003 .

[7]  Paresh Chandra Deka,et al.  Support vector machine applications in the field of hydrology: A review , 2014, Appl. Soft Comput..

[8]  Shie-Yui Liong,et al.  Forecasting of hydrologic time series with ridge regression in feature space , 2007 .

[9]  Guy G. Gable,et al.  Integrating case study and survey research methods: an example in information systems , 1994 .

[10]  Lingzhi Wang,et al.  A Novel Nonlinear Combination Model Based on Support Vector Machine for Rainfall Prediction , 2011, 2011 Fourth International Joint Conference on Computational Sciences and Optimization.

[11]  San Cristóbal Mateo,et al.  The Lack of A Priori Distinctions Between Learning Algorithms , 1996 .

[12]  J. Suykens,et al.  Time Series Prediction using LS-SVMs , 2008 .

[13]  D. G. Watts,et al.  Application of Linear Random Models to Four Annual Streamflow Series , 1970 .

[14]  P. Coulibaly,et al.  Two decades of anarchy? Emerging themes and outstanding challenges for neural network river forecasting , 2012 .

[15]  Maria-Helena Ramos,et al.  How do I know if my forecasts are better? Using benchmarks in hydrological ensemble prediction , 2015 .

[16]  J. P. King,et al.  Comparison of performance of statistical models in forecasting monthly streamflow of Kizil River, China , 2010 .

[17]  Demetris Koutsoyiannis,et al.  HESS Opinions "A random walk on water" , 2009 .

[18]  Demetris Koutsoyiannis,et al.  Discussion of “Generalized regression neural networks for evapotranspiration modelling” , 2007 .

[19]  X. Wen,et al.  A comparative study of artificial neural network, adaptive neuro fuzzy inference system and support vector machine for forecasting river flow in the semiarid mountain region , 2014 .

[20]  Ravi Sankar,et al.  Time Series Prediction Using Support Vector Machines: A Survey , 2009, IEEE Computational Intelligence Magazine.

[21]  Amir F. Atiya,et al.  An Empirical Comparison of Machine Learning Models for Time Series Forecasting , 2010 .

[22]  Amir F. Atiya,et al.  A comparison between neural-network forecasting techniques-case study: river flow forecasting , 1999, IEEE Trans. Neural Networks.

[23]  R. Brown Statistical forecasting for inventory control , 1960 .

[24]  Holger R. Maier,et al.  Neural networks for the prediction and forecasting of water resource variables: a review of modelling issues and applications , 2000, Environ. Model. Softw..

[25]  Demetris Koutsoyiannis,et al.  Forecasting of geophysical processes using stochastic and machine learning algorithms: Supplementary material , 2017 .

[26]  Jian Hu,et al.  EMD-KNN Model for Annual Average Rainfall Forecasting , 2013 .

[27]  Brian D. Ripley,et al.  Feed-Forward Neural Networks and Multinomial Log-Linear Models , 2015 .

[28]  Georgia Papacharalampous Theoretical and empirical comparison of stochastic and machine learning methods for hydrological processes forecasting , 2016 .

[29]  Florian Pappenberger,et al.  Do probabilistic forecasts lead to better decisions , 2012 .

[30]  Rob J Hyndman,et al.  Forecasting with Exponential Smoothing: The State Space Approach , 2008 .

[31]  Demetris Koutsoyiannis,et al.  Hurst‐Kolmogorov Dynamics and Uncertainty 1 , 2011 .

[32]  Jean-Philippe Vert,et al.  Consistency of Random Forests , 2014, 1405.2881.

[33]  N. J. de Vos,et al.  Echo state networks as an alternative to traditional artificial neural networks in rainfall–runoff modelling , 2013 .

[34]  E. S. Gardner EXPONENTIAL SMOOTHING: THE STATE OF THE ART, PART II , 2006 .

[35]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[36]  S. P. Neuman,et al.  On model selection criteria in multimodel analysis , 2007 .

[37]  H. Akaike A new look at the statistical model identification , 1974 .

[38]  Zaher Mundher Yaseen,et al.  Non-tuned machine learning approach for hydrological time series forecasting , 2016, Neural Computing and Applications.

[39]  Andy Liaw,et al.  Classification and Regression by randomForest , 2007 .

[40]  Hoshin Vijai Gupta,et al.  Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling , 2009 .

[41]  Demetris Koutsoyiannis,et al.  On the prediction of persistent processes using the output of deterministic models , 2017 .

[42]  Ani Shabri,et al.  Streamflow forecasting using least-squares support vector machines , 2012 .

[43]  Avi Ostfeld,et al.  Data-driven modelling: some past experiences and new approaches , 2008 .

[44]  Shie-Yui Liong,et al.  FLOOD STAGE FORECASTING WITH SUPPORT VECTOR MACHINES 1 , 2002 .

[45]  John F. MacGregor,et al.  Some Recent Advances in Forecasting and Control , 1968 .

[46]  Fangqiong Luo,et al.  A novel nonlinear combination model based on Support Vector Machine for stock market prediction , 2010, 2010 8th World Congress on Intelligent Control and Automation.

[47]  Singh Manjushree,et al.  Application of Software Packages for Monthly Stream Flow Forecasting of Kangsabati River in India , 2011 .

[48]  Chuntian Cheng,et al.  Using support vector machines for long-term discharge prediction , 2006 .

[49]  Fionn Murtagh,et al.  Multilayer perceptrons for classification and regression , 1991, Neurocomputing.

[50]  G. G. Moisen,et al.  Classification and regression trees , 2008 .

[51]  Stephen R. Marsland,et al.  Machine Learning - An Algorithmic Perspective , 2009, Chapman and Hall / CRC machine learning and pattern recognition series.

[52]  Junfei Chen,et al.  Statistical Uncertainty Estimation Using Random Forests and Its Application to Drought Forecast , 2012 .

[53]  P. Phillips,et al.  Testing the null hypothesis of stationarity against the alternative of a unit root: How sure are we that economic time series have a unit root? , 1992 .

[54]  Demetris Koutsoyiannis “Hurst-Kolomogorov Dynamics and Uncertainty” , 2010 .

[55]  Rosangela Ballini,et al.  Multi-step-ahead monthly streamflow forecasting by a neurofuzzy network model , 2001, Proceedings Joint 9th IFSA World Congress and 20th NAFIPS International Conference (Cat. No. 01TH8569).

[56]  J. Nash,et al.  River flow forecasting through conceptual models part I — A discussion of principles☆ , 1970 .

[57]  Tao Hong,et al.  Probabilistic electric load forecasting: A tutorial review , 2016 .

[58]  Fred L. Collopy,et al.  Error Measures for Generalizing About Forecasting Methods: Empirical Comparisons , 1992 .

[59]  Vladimir Vapnik,et al.  An overview of statistical learning theory , 1999, IEEE Trans. Neural Networks.

[60]  Gianluca Bontempi,et al.  Machine Learning Strategies for Time Series Forecasting , 2012, eBISS.

[61]  K. Nikolopoulos,et al.  The theta model: a decomposition approach to forecasting , 2000 .

[62]  Erwan Scornet,et al.  Rejoinder on: A random forest guided tour , 2016 .

[63]  Wei-Chiang Hong,et al.  Rainfall forecasting by technological machine learning models , 2008, Appl. Math. Comput..

[64]  Ozgur Kisi,et al.  Streamflow Forecasting Using Different Artificial Neural Network Algorithms , 2007 .

[65]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[66]  Everette S. Gardner,et al.  Exponential smoothing: The state of the art , 1985 .

[67]  Demetris Koutsoyiannis,et al.  A Bayesian statistical model for deriving the predictive distribution of hydroclimatic variables , 2014, Climate Dynamics.

[68]  Robert E. Criss,et al.  Do Nash values have value? Discussion and alternate proposals , 2008 .

[69]  Bellie Sivakumar,et al.  Chaos theory in geophysics: past, present and future , 2004 .

[70]  William N. Venables,et al.  Modern Applied Statistics with S , 2010 .

[71]  William W. S. Wei,et al.  Time series analysis - univariate and multivariate methods , 1989 .

[72]  A. H. Murphy,et al.  What Is a Good Forecast? An Essay on the Nature of Goodness in Weather Forecasting , 1993 .

[73]  Demetris Koutsoyiannis,et al.  Simultaneous estimation of the parameters of the Hurst–Kolmogorov stochastic process , 2011 .

[74]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[75]  Gordon Fraser,et al.  Parameter tuning or default values? An empirical investigation in search-based software engineering , 2013, Empirical Software Engineering.

[76]  Richard E. Neapolitan,et al.  Artificial Intelligence: With an Introduction to Machine Learning, Second Edition , 2018 .

[77]  Vujica Yevjevich,et al.  Stochastic models in hydrology , 1987 .

[78]  Demetris Koutsoyiannis,et al.  Medium-range flow prediction for the Nile: a comparison of stochastic and deterministic methods / Prévision du débit du Nil à moyen terme: une comparaison de méthodes stochastiques et déterministes , 2008 .

[79]  Yuan-Fong Su,et al.  On the criteria of model performance evaluation for real-time flood forecasting , 2017, Stochastic Environmental Research and Risk Assessment.

[80]  Roman Krzysztofowicz,et al.  The case for probabilistic forecasting in hydrology , 2001 .

[81]  Fotios Petropoulos,et al.  forecast: Forecasting functions for time series and linear models , 2018 .

[82]  David Terman,et al.  State space , 2008, Scholarpedia.

[83]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[84]  Evangelos Spiliotis,et al.  Statistical and Machine Learning forecasting methods: Concerns and ways forward , 2018, PloS one.

[85]  Christina Gloeckner,et al.  Modern Applied Statistics With S , 2003 .

[86]  Vladimir Cherkassky,et al.  The Nature Of Statistical Learning Theory , 1997, IEEE Trans. Neural Networks.

[87]  Ronny Berndtsson,et al.  Impact of complexity on daily and multi-step forecasting of streamflow with chaotic, stochastic, and black-box models , 2017, Stochastic Environmental Research and Risk Assessment.

[88]  Robert J. Abrahart,et al.  Neural Network Hydroinformatics: Maintaining Scientific Rigour , 2009 .

[89]  Galit Shmueli,et al.  To Explain or To Predict? , 2010 .

[90]  Shie-Yui Liong,et al.  Rainfall and runoff forecasting with SSA-SVM approach , 2001 .

[91]  Özgür Kisi,et al.  Forecasting daily lake levels using artificial intelligence approaches , 2012, Comput. Geosci..

[92]  C. Holt Author's retrospective on ‘Forecasting seasonals and trends by exponentially weighted moving averages’ , 2004 .

[93]  Ethem Alpaydin,et al.  Introduction to machine learning , 2004, Adaptive computation and machine learning.

[94]  Ian M. Mitchell,et al.  Reproducible research for scientific computing: Tools and strategies for changing the culture , 2012, Computing in Science & Engineering.

[95]  Clifford M. Hurvich,et al.  A CORRECTED AKAIKE INFORMATION CRITERION FOR VECTOR AUTOREGRESSIVE MODEL SELECTION , 1993 .

[96]  Min Han,et al.  Support Vector Echo-State Machine for Chaotic Time-Series Prediction , 2007, IEEE Transactions on Neural Networks.

[97]  Andreas S. Andreou,et al.  Nonlinear analysis and forecasting of a brackish Karstic spring , 2000 .

[98]  Gérard Biau,et al.  Analysis of a Random Forests Model , 2010, J. Mach. Learn. Res..

[99]  Erwan Scornet,et al.  A random forest guided tour , 2015, TEST.

[100]  D. K. Srivastava,et al.  Application of ANN for Reservoir Inflow Prediction and Operation , 1999 .

[101]  S. Sorooshian,et al.  Automatic calibration of conceptual rainfall-runoff models: sensitivity to calibration data , 1996 .

[102]  Pinar Donmez,et al.  Introduction to Machine Learning, 2nd ed., by Ethem Alpaydın. Cambridge, MA: The MIT Press 2010. ISBN: 978-0-262-01243-0. $54/£ 39.95 + 584 pages , 2013, Nat. Lang. Eng..

[103]  Yihui Xie,et al.  Dynamic Documents with R and knitr , 2015 .

[104]  Lutgarde M. C. Buydens,et al.  Using support vector machines for time series prediction , 2003 .

[105]  Rob J Hyndman,et al.  Forecasting Time Series With Complex Seasonal Patterns Using Exponential Smoothing , 2011 .

[106]  Özgür Kisi,et al.  Precipitation forecasting by using wavelet-support vector machine conjunction model , 2012, Eng. Appl. Artif. Intell..

[107]  Hadley Wickham,et al.  Tools to Make Developing R Packages Easier , 2016 .

[108]  Andrew Harvey,et al.  A unified view of statistical forecasting procedures , 1984 .

[109]  Hoshin Vijai Gupta,et al.  Do Nash values have value? , 2007 .

[110]  David H. Wolpert,et al.  The Lack of A Priori Distinctions Between Learning Algorithms , 1996, Neural Computation.

[111]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[112]  Rob J Hyndman,et al.  Unmasking the Theta Method , 2003 .

[113]  Michael Y. Hu,et al.  Forecasting with artificial neural networks: The state of the art , 1997 .

[114]  Rob J Hyndman,et al.  Automatic Time Series Forecasting: The forecast Package for R , 2008 .

[115]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[116]  Holger R. Maier,et al.  Improved validation framework and R-package for artificial neural network models , 2017, Environ. Model. Softw..

[117]  Lars Schmidt-Thieme,et al.  Beyond Manual Tuning of Hyperparameters , 2015, KI - Künstliche Intelligenz.

[118]  R. Lippmann,et al.  An introduction to computing with neural nets , 1987, IEEE ASSP Magazine.

[119]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[120]  A. Zeileis Econometric Computing with HC and HAC Covariance Matrix Estimators , 2004 .

[121]  Spyros Makridakis,et al.  Confidence intervals: An empirical investigation of the series in the M-competition , 1987 .

[122]  I SapankevychNicholas,et al.  Time series prediction using support vector machines , 2009 .

[123]  Wayan Firdaus Mahmudy,et al.  Drought forecasting using ANFIS on tuban regency, Indonesia , 2017, 2017 International Conference on Sustainable Information Engineering and Technology (SIET).

[124]  Richard A. Wasniowski Using support vector machines in data mining , 2004 .

[125]  Hadley Wickham,et al.  The Split-Apply-Combine Strategy for Data Analysis , 2011 .

[126]  Spyros Makridakis,et al.  The M3-Competition: results, conclusions and implications , 2000 .

[127]  MohammadSajjad Khan,et al.  Application of Support Vector Machine in Lake Water Level Prediction , 2006 .

[128]  Murad S. Taqqu,et al.  A seasonal fractional ARIMA Model applied to the Nile River monthly flows at Aswan , 2000 .

[129]  Demetris Koutsoyiannis,et al.  Error Evolution in Multi-Step Ahead Streamflow Forecasting for the Operation of Hydropower Reservoirs , 2017 .

[130]  Demetris Koutsoyiannis,et al.  Predictability of monthly temperature and precipitation using automatic time series forecasting methods , 2018, Acta Geophysica.

[131]  Kohske Takahashi,et al.  Create Elegant Data Visualisations Using the Grammar of Graphics [R package ggplot2 version 3.3.2] , 2020 .

[132]  V. Singh,et al.  Drought Forecasting Using a Hybrid Stochastic and Neural Network Model , 2007 .

[133]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[134]  이의훈 European Geosciences Union General Assembly 2017 참가 후기 , 2017 .

[135]  P. Kitanidis,et al.  Real‐time forecasting with a conceptual hydrologic model: 2. Applications and results , 1980 .

[136]  Galit Shmueli,et al.  To Explain or To Predict? , 2010, 1101.0891.

[137]  Gunnar Rätsch,et al.  Using support vector machines for time series prediction , 1999 .

[138]  Peter K. Kitanidis,et al.  Real‐time forecasting with a conceptual hydrologic model: 1. Analysis of uncertainty , 1980 .

[139]  Gang Luo,et al.  A review of automatic selection methods for machine learning algorithms and hyper-parameter values , 2016, Network Modeling Analysis in Health Informatics and Bioinformatics.

[140]  Robert Fildes,et al.  The evaluation of extrapolative forecasting methods , 1992 .

[141]  Paulo Cortez,et al.  Data Mining with Neural Networks and Support Vector Machines Using the R/rminer Tool , 2010, ICDM.

[142]  Rangasami L. Kashyap,et al.  Optimal Choice of AR and MA Parts in Autoregressive Moving Average Models , 1982, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[143]  Parthasarathy Ramachandran,et al.  A Comparison of Machine Learning Techniques for Modeling River Flow Time Series: The Case of Upper Cauvery River Basin , 2014, Water Resources Management.

[144]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[145]  Steven P. Millard,et al.  EnvStats: An R Package for Environmental Statistics , 2013 .

[146]  P. Krause,et al.  COMPARISON OF DIFFERENT EFFICIENCY CRITERIA FOR HYDROLOGICAL MODEL ASSESSMENT , 2005 .

[147]  Yihui Xie,et al.  Dynamic Documents with R and knitr, Second Edition , 2015 .

[148]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[149]  Ozgur Kisi,et al.  A wavelet-support vector machine conjunction model for monthly streamflow forecasting , 2011 .

[150]  Yihui Xie,et al.  knitr: A Comprehensive Tool for Reproducible Research in R , 2018, Implementing Reproducible Research.

[151]  J. Armstrong,et al.  Evaluating Forecasting Methods , 2001 .

[152]  D. Legates,et al.  Evaluating the use of “goodness‐of‐fit” Measures in hydrologic and hydroclimatic model validation , 1999 .

[153]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[154]  Terry L. Lanc,et al.  The importance of input variables to a neural network fault-diagnostic system for nuclear power plants , 1992 .

[155]  Georgia Papacharalampous,et al.  Variable Selection in Time Series Forecasting Using Random Forests , 2017, Algorithms.

[156]  B. Efron,et al.  Simultaneous Estimation of Parameters , 1972 .

[157]  Rob J Hyndman,et al.  A state space framework for automatic forecasting using exponential smoothing methods , 2002 .

[158]  Communicating uncertainty. , 2002, Minnesota medicine.

[159]  Edward H. Wiser,et al.  Stochastic Models in Hydrology , 1967 .

[160]  François Anctil,et al.  A neural network experiment on the simulation of daily nitrate-nitrogen and suspended sediment fluxes from a small agricultural catchment , 2009 .

[161]  Kurt Hornik,et al.  kernlab - An S4 Package for Kernel Methods in R , 2004 .

[162]  George Athanasopoulos,et al.  Forecasting: principles and practice , 2013 .

[163]  Chuntian Cheng,et al.  A comparison of performance of several artificial intelligence , 2009 .

[164]  S. Liong,et al.  EC-SVM approach for real-time hydrologic forecasting , 2004 .

[165]  Baki Billah,et al.  Empirical information criteria for time series forecasting model selection , 2005 .

[166]  M. Valipour,et al.  Comparison of the ARMA, ARIMA, and the autoregressive artificial neural network models in forecasting the monthly inflow of Dez dam reservoir , 2013 .

[167]  Chuntian Cheng,et al.  A new indirect multi-step-ahead prediction model for a long-term hydrologic prediction , 2008 .

[168]  Leo Breiman,et al.  Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author) , 2001 .

[169]  Demetris Koutsoyiannis,et al.  Negligent killing of scientific concepts: the stationarity case , 2015 .

[170]  Rob J Hyndman,et al.  Prediction intervals for exponential smoothing using two new classes of state space models 30 January 2003 , 2003 .

[171]  Duncan Snidal,et al.  Rational Deterrence Theory and Comparative Case Studies , 1989, World Politics.

[172]  Demetris Koutsoyiannis,et al.  Generic and parsimonious stochastic modelling for hydrology and beyond , 2016 .

[173]  Rob J Hyndman,et al.  Another look at measures of forecast accuracy , 2006 .

[174]  Demetris Koutsoyiannis,et al.  One-step ahead forecasting of geophysical processes within a purely statistical framework , 2018, Geoscience Letters.

[175]  William Stafford Noble,et al.  Support vector machine , 2013 .

[176]  Guoqiang Peter Zhang,et al.  An investigation of neural networks for linear time-series forecasting , 2001, Comput. Oper. Res..

[177]  Rob J Hyndman,et al.  25 years of time series forecasting , 2006 .

[178]  R. Hyndman Automatic time series forecasting , 2006 .

[179]  Kurt Hornik,et al.  The Design and Analysis of Benchmark Experiments , 2005 .

[180]  S. Weijs,et al.  Why hydrological predictions should be evaluated using information theory , 2010 .

[181]  A. Raftery,et al.  Space-time modeling with long-memory dependence: assessing Ireland's wind-power resource. Technical report , 1987 .

[182]  Rob J. Hyndman,et al.  Forecasting with Exponential Smoothing , 2008 .

[183]  Brian D. Ripley,et al.  Modern Applied Statistics with S Fourth edition , 2002 .

[184]  Ozgur Kisi,et al.  River Flow Modeling Using Artificial Neural Networks , 2004 .

[185]  Ming Ye,et al.  Maximum likelihood Bayesian averaging of spatial variability models in unsaturated fractured tuff , 2003 .