On estimating the reliability of ecological forecasts.

Recent work has highlighted the utility of nonparametric forecasting methods for predicting ecological time series (Perretti et al., 2013. Proc. Natl. Acad. Sci. U.S.A. 110, 5253-5257). However, one topic that has received considerably less attention is the quantification of uncertainty in nonparametric forecasts. This important topic was brought to the forefront in the recent work by Jabot (2014. J. Theor. Biol.). Here, we add to this emerging discussion by reviewing the available methods for quantifying forecast uncertainty in nonparametric models. We conclude with a demonstration of one such method using the simulation model of Jabot (2014. J. Theor. Biol.). We find that nonparametric forecast error is accurately estimated with as few as 10 observations in the time series.

[1]  J. Shao.  AN ASYMPTOTIC THEORY FOR LINEAR MODEL SELECTION , 1997 .

[2]  George Sugihara,et al.  Nonparametric forecasting outperforms parametric methods for a simulated multispecies system , 2013 .

[3]  D. Cox,et al.  An Analysis of Transformations , 1964 .

[4]  Ben Collen,et al.  Complexity is costly: a meta‐analysis of parametric and non‐parametric methods for short‐term population forecasting , 2014 .

[5]  Leonard J. Tashman,et al.  Out-of-sample tests of forecasting accuracy: an analysis and review , 2000 .

[6]  Jeffrey S. Racine,et al.  Nonparametric Econometrics: The np Package , 2008 .

[7]  M. Stone.  An Asymptotic Equivalence of Choice of Model by Cross‐Validation and Akaike's Criterion , 1977 .

[8]  William Remus,et al.  Neural Network Models for Time Series Forecasts , 1996 .

[9]  E. Nadaraya.  On Estimating Regression , 1964 .

[10]  Marc Mangel,et al.  Bayesian nonparametric analysis of stock- recruitment relationships , 2005 .

[11]  W. Härdle,et al.  Kernel regression smoothing of time series , 1992 .

[12]  R. Tibshirani,et al.  Improvements on Cross-Validation: The 632+ Bootstrap Method , 1997 .

[13]  George Sugihara,et al.  Model-free forecasting outperforms the correct mechanistic model for simulated and experimental data , 2013, Proceedings of the National Academy of Sciences.

[14]  Annette M. Molinaro,et al.  Prediction error estimation: a comparison of resampling methods , 2005, Bioinform..

[15]  Franck Jabot,et al.  Why preferring parametric forecasting to nonparametric methods? , 2015, Journal of theoretical biology.

[16]  Jeffrey S. Racine,et al.  Nonparametric estimation of regression functions with both categorical and continuous data , 2004 .

[17]  George Sugihara,et al.  Complex dynamics may limit prediction in marine fisheries , 2014 .

[18]  S. Carpenter,et al.  Ecological forecasts: an emerging imperative. , 2001, Science.

[19]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[20]  Kotaro Ono,et al.  A Bayesian approach to identifying and compensating for model misspecification in population models. , 2014, Ecology.

[21]  N. Stenseth,et al.  Ecological forecasting under climate change: the case of Baltic cod , 2010, Proceedings of the Royal Society B: Biological Sciences.

[22]  Stephen P. Ellner,et al.  Chaos in a Noisy World: New Methods and Evidence from Time-Series Analysis , 1995, The American Naturalist.

[23]  George Sugihara,et al.  Nonlinear forecasting for the classification of natural time series , 1994, Philosophical Transactions of the Royal Society of London. Series A: Physical and Engineering Sciences.

[24]  B. Efron,et al.  A Leisurely Look at the Bootstrap, the Jackknife, and , 1983 .