An evaluation of diagnostic tests and their roles in validating forest biometric models

Model validation is an important part of model development. It is performed to increase the credibility and gain sufficient confidence about a model. This paper evaluated the usefulness of 10 statistical tests, five parametric and five nonparametric, in validating forest biometric models. The five parametric tests are the paired t test, the Χ2 test, the separate t test, the simultaneous F test, and the novel test. The five nonparametric tests are the Brown-Mood test, the Kolmogorov–Smirnov test, the modified Kolmogorov–Smirnov test, the sign test, and the Wilcoxon signed-rank test. Nine benchmark data sets were selected to evaluate the behavior of these tests in model validation; three were collected from Alberta and six were published elsewhere. It was shown that the usefulness of statistical tests in model validation is very limited. None of the tests seems to be generic enough to work well across a wide range of models and data. Each model passed one or more tests, but not all of them. Because of this,...

[1]  Martin W. Ritchie,et al.  Implications of disaggregation in forest growth and yield modeling , 1997 .

[2]  Marion R. Reynolds,et al.  Regression methodology for estimating model prediction error , 1986 .

[3]  David G. Mayer,et al.  Regression of real-world data on model output: An appropriate overall test of validity , 1994 .

[4]  Jack P. C. Kleijnen,et al.  Validation of Trace-Driven Simulation Models: A Novel Regression Test , 1998 .

[5]  Douglas A. Wolfe,et al.  Nonparametric Statistical Methods , 1973 .

[6]  Oscar García,et al.  Evaluating forest Growth Models , 1997 .

[7]  M. R. Reynolds.  Estimating the error in model predictions. , 1984 .

[8]  Leslie Anne Morehead,et al.  Determining the Factors Influential in the Validation of Computer-based Problem Solving Systems , 1996 .

[9]  Frederick Mosteller,et al.  Data Analysis and Regression , 1978 .

[10]  C. Borror.  Practical Nonparametric Statistics, 3rd Ed. , 2001 .

[11]  F. Uzoh,et al.  Individual tree height increment model for managed even-aged stands of ponderosa pine throughout the western United States using linear mixed effects models , 2006 .

[12]  R. Dennis Cook,et al.  Cross-Validation of Regression Models , 1984 .

[13]  W. W. Daniel.  Applied Nonparametric Statistics , 1979 .

[14]  M. Stephens.  EDF Statistics for Goodness of Fit and Some Comparisons , 1974 .

[15]  S. Harrison,et al.  Regression of a model on real-system output: An invalid test of model validity , 1990 .

[16]  R. Scheaffer,et al.  Probability and statistics for engineers , 1986 .

[17]  Robert A. Monserud,et al.  Large-scale management experiments in the moist maritime forests of the Pacific Northwest , 2002 .

[18]  A. Kozak,et al.  Does cross validation provide additional information in the evaluation of regression models , 2003 .

[19]  D. Reed,et al.  Evaluating forest stress factors using various forest growth modeling approaches , 1994 .

[20]  Timothy G. Gregoire,et al.  Using Simultaneous Regression Techniques with Individual-Tree Growth Models , 1998 .

[21]  Patricia J. Wozniak.  Applied Nonparametric Statistics (2nd ed.) , 1991 .

[22]  Richard L. Scheaffer,et al.  Probability and statistics for engineers , 1986 .

[23]  Robert A. Monserud,et al.  Performance and comparison of stand growth models based on individual tree and diameter class growth , 1979 .

[24]  Paula Soares,et al.  A critical look at procedures for validating growth and yield models. , 2003 .

[25]  Timothy G. Gregoire,et al.  Accuracy Testing and Estimation Alternatives , 1988, Forest Science.

[26]  Peter L. Marshall,et al.  Testing the distributional assumptions of least squares linear regression , 1995 .

[27]  J. O. Rawlings,et al.  Applied Regression Analysis: A Research Tool , 1988 .

[28]  Robert A. Monserud,et al.  Applicability of the forest stand growth simulator PROGNAUS for the Austrian part of the Bohemian Massif , 1997 .

[29]  Ronald D. Snee,et al.  Validation of Regression Models: Methods and Examples , 1977 .

[30]  J. N. R. Jeffers,et al.  Mathematical Models in Ecology. , 1972 .

[31]  P. Sprent,et al.  Applied Nonparametric Statistical Methods, Third Edition , 2000 .

[32]  H. Shugart.  A Theory of Forest Dynamics , 1984 .

[33]  Elizabeth A. Peck,et al.  Introduction to Linear Regression Analysis , 2001 .

[34]  N. Draper,et al.  Applied Regression Analysis , 1966 .

[35]  Andrew P. Robinson,et al.  Criteria for comparing the adaptability of forest growth models , 2003 .

[36]  Norman R. Draper,et al.  Applied regression analysis (2. ed.) , 1981, Wiley series in probability and mathematical statistics.

[37]  J. Neter,et al.  Applied Linear Regression Models , 1983 .

[38]  R. A. Groeneveld,et al.  Practical Nonparametric Statistics (2nd ed). , 1981 .

[39]  Robert G. Sargent,et al.  Validation and verification of simulation models , 1999, Proceedings of the 2004 Winter Simulation Conference, 2004..

[40]  Daniel A. Yaussy,et al.  Comparison of an empirical forest growth and yield simulator and a forest gap simulator using actual 30-year growth from two even-aged forests in Kentucky , 2000 .

[41]  Shongming Huang,et al.  Estimating a system of nonlinear simultaneous individual tree models for white spruce in boreal mixed-species stands , 1999 .

[42]  Saul I. Gass,et al.  Letter to the Editor - Guidelines for Model Evaluation: An Abridged Version of the U.S. General Accounting Office Exposure Draft , 1980, Oper. Res..

[43]  J.C.G. Goelz,et al.  Development of a well-behaved site index equation: jack pine in north central Ontario , 1992 .

[44]  R. Curtis.  Yield Tables Past and Present , 1972 .

[45]  K. Popper,et al.  Conjectures and Refutations , 1963 .

[46]  Leslie A. Real,et al.  Monte Carlo assessments of goodness-of-fit for ecological simulation models , 2003 .

[47]  Ralph B. D'Agostino,et al.  Goodness-of-Fit-Techniques , 2020 .

[48]  S. Huang,et al.  Validation of ecoregion-based taper equations for white spruce in Alberta , 1999 .