Ecologists should not use statistical significance tests to interpret simulation model results

Simulation models are widely used to represent the dynamics of ecological systems. A common question with such models is how changes to a parameter value or functional form in the model alter the results. Some authors have chosen to answer that question using frequentist statistical hypothesis tests (e.g. ANOVA). �怀is is inappropriate for two reasons. First, p-values are determined by statistical power (i.e. replication), which can be arbitrarily high in a simulation context, producing minuscule p-values regardless of the effect size. Second, the null hypothesis of no difference between treatments (e.g. parameter values) is known a priori to be false, invalidating the premise of the test. Use of p-values is troublesome (rather than simply irrelevant) because small p-values lend a false sense of importance to observed differences. We argue that modelers should abandon this practice and focus on evaluating the magnitude of differences between simulations. A growing number of authors in the ecological literature use statistical methods common to experimental ecology to analyze the output of ecological simulation models. For example, authors may use analysis of variance (ANOVA) to test whether model runs with different parameter values or different functional forms produce statistically different outputs. We view significance testing applied to simulation model output as a misuse of statistical theory. In this article we explain our reasoning with the goals of discouraging the practice, encouraging instead a focus on the magnitude of differences between simulations (i.e. effect sizes), and sparking discussion regarding when ‐ if ever ‐ statistical significance tests could be appropriate. �怀e perils of placing too much emphasis on statistical