Validation of Biophysical Models: Issues and Methodologies

The potential of mathematical models is widely acknowledged for examining components and interactions of natural systems, estimating the changes and uncertainties on outcomes, and fostering communication between scientists with different backgrounds and between scientists, managers and the community. For favourable reception of models, a systematic accrual of a good knowledge base is crucial for both science and decision-making. As the roles of models grow in importance, there is an increase in the need for appropriate methods with which to test their quality and performance. For biophysical models, the heterogeneity of data and the range of factors influencing usefulness of their outputs often make it difficult for full analysis and assessment. As a result, modelling studies in the domain of natural sciences often lack elements of good modelling practice related to model validation, that is correspondence of models to its intended purpose. Here we review validation issues and methods currently available for assessing the quality of biophysical models. The review covers issues of validation purpose, the robustness of model results, data quality, model prediction and model complexity. The importance of assessing input data quality and interpretation of phenomena is also addressed. Details are then provided on the range of measures commonly used for validation. Requirements for a methodology for assessment during the entire model-cycle are synthesised. Examples are used from a variety of modelling studies which mainly include agronomic modelling, e.g. crop growth and development, climatic modelling, e.g. climate scenarios, and hydrological modelling, e.g. soil hydrology, but the principles are essentially applicable to any area. It is shown that conducting detailed validation requires multi-faceted knowledge, and poses substantial scientific and technical challenges. Special emphasis is placed on using combined multiple statistics to expand our horizons in validation whilst also tailoring the validation requirements to the specific objectives of the application.

[1]  J. Abecassis,et al.  Parametric modelling of wheat grain morphology: a new perspective , 2003 .

[2]  S J Woodward,et al.  Validating a model that predicts daily growth and feed quality of New Zealand dairy pastures. , 2001, Environment international.

[3]  A. Aitken,et al.  Assessing systematic errors in rainfall-runoff models , 1973 .

[4]  Andrew P. Whitmore,et al.  method for assessing the goodness of computer simulation of soil processes , 1991 .

[5]  Y. Pachepsky,et al.  Development of pedotransfer functions in soil hydrology , 2004 .

[6]  Gianni Bellocchi,et al.  IMPROVEMENTS IN TIME MISMATCH ANALYSIS OF MODEL ESTIMATES , 2004 .

[7]  M. Janssen,et al.  Multi-Agent Systems for the Simulation of Land-Use and Land-Cover Change: A Review , 2003 .

[8]  Edwin A. Wesolowski.  Verification of water-quality model to simulate effects of discharging treated wastewater during ice-cover conditions to the Red River of the North at Fargo, North Dakota, and Moorhead, Minnesota , 1995 .

[9]  I. A. Chaikovsky,et al.  Terminology for model credibility , 1979 .

[10]  Gianni Bellocchi,et al.  IRENE_DLL: A Class Library for Evaluating Numerical Estimates , 2003 .

[11]  Osman Balci,et al.  Validating Expert System Performance , 1987, IEEE Expert.

[12]  S. Zacharias,et al.  Robust Quantitative Techniques for Validating Pesticide Transport Models , 1996 .

[13]  Mike Rivington,et al.  Adaptation assessments for crop production in response to climate change in Cameroon , 2009, Agronomy for Sustainable Development.

[14]  Craig Aumann.  A Methodology for Building Credible Models for Policy Evaluation , 2008 .

[15]  William H. Press,et al.  Numerical Recipes in Fortran 77: The Art of Scientific Computing 2nd Editionn - Volume 1 of Fortran Numerical Recipes , 1992 .

[16]  L. Breiman,et al.  Submodel selection and evaluation in regression. The X-random case , 1992 .

[17]  L. A. Kramer,et al.  Validation of EPIC for Two Watersheds in Southwest Iowa , 1999 .

[18]  Kevin Judd,et al.  Modelling the dynamics of nonlinear time series using canonical variate analysis , 2002 .

[19]  N. Fodor,et al.  Sensitivity of 4M maize model to the inaccuracy of weather and soil input data , 2003 .

[20]  D. Hinkle,et al.  Applied statistics for the behavioral sciences , 1979 .

[21]  M. Donatelli,et al.  Modelling cropping systems¿highlights of the symposium and preface to the special issues , 2002 .

[22]  Belinda E. Medlyn,et al.  Design and use of a database of model parameters from elevated [CO2] experiments , 1999 .

[23]  R. Parrish,et al.  A method for testing whether model predictions fall within a prescribed factor of true values, with an application to pesticide leaching , 1990 .

[24]  N Oreskes,et al.  Evaluation (not validation) of quantitative models. , 1998, Environmental health perspectives.

[25]  Gianni Bellocchi,et al.  New indices to quantify patterns of residuals produced by model estimates , 2004 .

[26]  N Oreskes,et al.  Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences , 1994, Science.

[27]  D.R. Wallace,et al.  Software verification and validation: an overview , 1989, IEEE Software.

[28]  E. Scott Bair,et al.  MODEL (IN) VALIDATION – A VIEW FROM THE COURTROOM , 1994 .

[29]  Sallie Keller-McNulty,et al.  Workshop on Statistical Approaches for the Evaluation of Complex Computer Models , 2002 .

[30]  R. Rabbinge,et al.  Theoretical Production Ecology: Reflections and Prospects , 1990 .

[31]  B. W. Rust,et al.  Importance of validation in ecosystem analysis , 1975 .

[32]  C. Mallows.  Some Comments on Cp , 2000, Technometrics.

[33]  Vic Barnett,et al.  Response to Comments on : Testing winter wheat simulation models predictions against observed UK grain yields by Landau et al. [Agric. For. Meteorol. 89 (1998) 85-99] by Jamieson et al. [Agric. For. Meteorol., this issue] , 1999 .

[34]  Mikhail Kanevski,et al.  Machine Learning Algorithms for GeoSpatial Data. Applications and Software Tools , 2008 .

[35]  Beverly J. Volicer,et al.  Biostatistics: A Foundation for Analysis in the Health Sciences , 1979 .

[36]  M. Rivington,et al.  An integrated assessment approach to conduct analyses of climate change impacts on whole-farm systems , 2007, Environ. Model. Softw..

[37]  M. Analla,et al.  Model validation through the linear regression fit to actual versus predicted values , 1998 .

[38]  Guk-Rwang Won.  American Society for Testing and Materials , 1987 .

[39]  P. Heuberger,et al.  Calibration of process-oriented models , 1995 .

[40]  J. M. Hanft,et al.  Visual Indicators of Physiological Maturity of Hard Red Spring Wheat 1 , 1982 .

[41]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences , 2022, The SAGE Encyclopedia of Research Design.

[42]  Gianni Bellocchi,et al.  irene: a software to evaluate model performance , 2003 .

[43]  J. B. Dent,et al.  Systems Simulation in Agriculture , 1979, Springer Netherlands.

[44]  R. Pielke,et al.  FORUM ON MODELING THE ATMOSPHERIC BOUNDARY LAYER , 2005 .

[45]  M. Jeuffroy,et al.  The Azodyn crop model as a decision support tool for choosing cultivars , 2006 .

[46]  Thomas R. Sinclair,et al.  Criteria for publishing papers on crop modeling , 2000 .

[47]  V. Thomas Parker,et al.  Conceptual problems and scale limitations of defining ecological communities: a critique of the CI concept (Community of Individuals) , 2001 .

[48]  Alan R. Ek,et al.  The consequences of hierarchy for modeling in forest ecosystems , 2000 .

[49]  C. J. Doyle,et al.  Modelling the comparative productivity and profitability of grass and legume systems of silage production in northern Europe , 2004 .

[50]  G. Hoogenboom.  Contribution of agrometeorology to the simulation of crop production and its applications , 2000 .

[51]  K. Loague,et al.  Statistical and graphical methods for evaluating solute transport models: Overview and application , 1991 .

[52]  P. Bates,et al.  Model Validation - Perspectives in Hydrological Science , 2001 .

[53]  D. Stephenson,et al.  Criteria for comparison of single event models , 1986 .

[54]  Leif T. Jensen,et al.  A comparison of the performance of nine soil organic matter models using datasets from seven long-term experiments , 1997 .

[55]  Michio Sugeno,et al.  An introductory survey of fuzzy control , 1985, Inf. Sci..

[56]  J F Louis,et al.  Guidance for the Performance Evaluation of Three-Dimensional Air Quality Modeling Systems for Particulate Matter and Visibility , 2000, Journal of the Air & Waste Management Association.

[57]  Philip W. Gassman,et al.  Evaluation of EPIC for Three Minnesota Cropping Systems , 2000 .

[58]  Ashok Kumar,et al.  Evaluation of Three Air Dispersion Models: ISCST2, ISCLT2, and Screen2 for Mercury Emissions in an Urban Area , 1998 .

[59]  Frank A. P. C. Gobas,et al.  Development and Field Validation of a Multimedia Exposure Assessment Model for Waste Load Allocation in Aquatic Ecosystems: Application to 2,3,7,8-Tetrachlorodibenzo-p-dioxin and 2,3,7,8-Tetrachlorodibenzofuran in the Fraser River Watershed , 1998 .

[60]  Alex Simpkins,et al.  System Identification: Theory for the User, 2nd Edition (Ljung, L.; 1999) [On the Shelf] , 2012, IEEE Robotics & Automation Magazine.

[61]  D. J. Greenwood,et al.  Statistical methods for evaluating a crop nitrogen simulation model, N_ABLE , 2000 .

[62]  K. Matthews,et al.  Implementation of a spatial decision support system for rural land use planning: integrating GIS and environmental models with search and optimisation algorithms , 1999 .

[63]  Susan Craw,et al.  Implementation of a spatial decision support system for rural land use planning: integrating geographic information system and environmental models with search and optimisation algorithms , 1999 .

[64]  Attila Nemes,et al.  Integrated indices for pedotransfer functions evaluation , 2004 .

[65]  V. Barnett,et al.  Testing winter wheat simulation models' predictions against observed UK grain yields , 1998 .

[66]  M. Rivington,et al.  Quantifying the uncertainty in spatially-explicit land- use model predictions arising from the use of substituted climate data , 2003 .

[67]  B. Efron.  How Biased is the Apparent Error Rate of a Prediction Rule , 1986 .

[68]  Kenneth H. Reckhow,et al.  Engineering approaches for lake management , 1983 .

[69]  K. B. Matthewsa,et al.  Raising the Bar – Is evaluating the outcomes of decision and information support tools a bridge too far ? , 2008 .

[70]  Gianni Bellocchi,et al.  Modeling reference evapotranspiration over complex terrains from minimum climatological data , 2007 .

[71]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[72]  Ralph T. Cheng,et al.  Interfacing data analysis and numerical modeling for tidal hydrodynamic phenomena , 1991 .

[73]  Daniel Wallach,et al.  Parameter Estimation for Crop Models , 2001 .

[74]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[75]  Jack P. C. Kleijnen,et al.  Validation of Trace-Driven Simulation Models: Bootstrap Tests , 2001, Manag. Sci..

[76]  J. J. Neeteson,et al.  Response of potatoes to N fertilizer: Dynamic model , 1985, Plant and Soil.

[77]  C. L. Mallows.  Some comments on C_p , 1973 .

[78]  Daniel Wallach,et al.  Mean squared error of prediction as a criterion for evaluating and comparing system models , 1989 .

[79]  A. Agresti.  Categorical data analysis , 1993 .

[80]  W. E. Ricker,et al.  Computation and uses of central trend lines , 1984 .

[81]  A. H. Marcus,et al.  Some useful statistical methods for model validation. , 1998, Environmental health perspectives.

[82]  M. Rivington,et al.  Evaluation of three model estimations of solar radiation at 24 UK stations , 2005 .

[83]  Andrew P. Whitmore,et al.  Computer simulation of changes in soil mineral nitrogen and crop nitrogen during autumn, winter and spring , 1987, The Journal of Agricultural Science.

[84]  Lucie A. Vincent,et al.  A Technique for the Identification of Inhomogeneities in Canadian Temperature Series , 1998 .

[85]  Dolores R. Wallace,et al.  Software verification and validation , 1989 .

[86]  John D. Sterman,et al.  System Dynamics: Systems Thinking and Modeling for a Complex World , 2002 .

[87]  Xinwei Deng,et al.  Experimental design , 2012, WIREs Data Mining Knowl. Discov..

[88]  M. Herbst,et al.  Towards model evaluation using Self-Organizing Maps , 2008 .

[89]  G. Meehl,et al.  OVERVIEW OF THE COUPLED MODEL INTERCOMPARISON PROJECT , 2005 .

[90]  H. Akaike.  A new look at the statistical model identification , 1974 .

[91]  Roberto Pastres,et al.  Testing the robustness of primary production models in shallow coastal areas: a case study , 2004 .

[92]  M. A. Hamilton,et al.  Model validation: an annotated bibliography , 1991 .

[93]  Keith Beven,et al.  Towards integrated environmental models of everywhere: uncertainty, data and modelling as a learning process , 2007 .

[94]  David G. Mayer,et al.  Regression of real-world data on model output: An appropriate overall test of validity , 1994 .

[95]  G. G. Wright,et al.  Combining Metric Aerial Photography and Near‐Infrared Videography to Define Within‐Field Soil Sampling Frameworks , 2003 .

[96]  William H. Press,et al.  Numerical recipes in FORTRAN (2nd ed.): the art of scientific computing , 1992 .

[97]  Richard S. Sojda,et al.  Empirical Evaluation of Decision Support Systems: Concepts and an Example for Trumpeter Swan Management , 2004 .

[98]  C. T. de Wit,et al.  Philosophy and terminology , 1993 .

[99]  R. Barnes,et al.  Use of Mean Square Prediction Error Analysis and Reproducibility Measures to Study near Infrared Calibration Equation Performance , 1999 .

[100]  Kevin W. King,et al.  Data Uncertainty Estimation Tool for Hydrology and Water Quality (DUET-H/WQ): Estimating Measurement Uncertainty for Monitoring and Modelling Applications , 2008 .

[101]  R. Gil Pontius,et al.  Land-cover change model validation by an ROC method for the Ipswich watershed, Massachusetts, USA , 2001 .

[102]  Douglas G. Fox,et al.  Judging Air Quality Model Performance , 1981 .

[103]  Vic Barnett,et al.  A PARSIMONIOUS, MULTIPLE-REGRESSION MODEL OF WHEAT YIELD RESPONSE TO ENVIRONMENT , 2000 .

[104]  M. Salam,et al.  Comparing Simulated and Measured Values Using Mean Squared Deviation and its Components , 2000 .

[105]  P. L. Mitchell.  Misuse of regression for empirical validation of models , 1997 .

[106]  N. W. T. Quinn.  Integration of sensor networks and decision support tools for basin-scale, real-time water quality management , 2008 .

[107]  J. F. Lopes,et al.  3D ecological modelling of the Aveiro coast (Portugal) , 2008 .

[108]  M. Kuhnert,et al.  Comparing Raster Map Comparison Algorithms for Spatial Modeling and Analysis , 2005 .

[109]  Luis Orlindo Tedeschi,et al.  Assessment of the adequacy of mathematical models , 2006 .

[110]  William K. Lauenroth,et al.  Models in Ecosystem Science , 2003, Models in Ecosystem Science.

[111]  Z. Ying,et al.  Model‐Checking Techniques Based on Cumulative Residuals , 2002, Biometrics.

[112]  T. O. Kvålseth.  Cautionary Note about R 2 , 1985 .

[113]  Keith Beven,et al.  Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology , 2001 .

[114]  Kenneth H. Reckhow,et al.  Statistical Evaluation of Mechanistic Water‐Quality Models , 1990 .

[115]  J. Campbell.  Introduction to remote sensing , 1987 .

[116]  L. Belbin,et al.  Evaluation of statistical models used for predicting plant species distributions: Role of artificial data and theory , 2006 .

[117]  Mikhail A. Semenov,et al.  Comments on “Testing winter wheat simulation models predictions against observed UK grain yields” by , 1999 .

[118]  C. Willmott.  Some Comments on the Evaluation of Model Performance , 1982 .

[119]  D. Holzworth,et al.  Common Sense In Model Testing , 2005 .

[120]  William W. Doe,et al.  Landscape erosion and evolution modeling , 2001 .

[121]  David Favis-Mortlock,et al.  The Limits of Erosion Modeling , 2001 .

[122]  Edward J. Rykiel,et al.  Testing ecological models: the meaning of validation , 1996 .

[123]  Keith Beven,et al.  On the sensitivity of soil-vegetation-atmosphere transfer (SVAT) schemes: equifinality and the problem of robust calibration , 1997 .

[124]  D.Lynn Shaeffer,et al.  A model evaluation methodology applicable to environmental assessment models , 1980 .

[125]  M. Rivington,et al.  Characterising the agro-meteorological implications of climate change scenarios for land management stakeholders , 2008 .

[126]  G. Bellocchi,et al.  Evaluating uncertainty introduced to process-based simulation model estimates by alternative sources of meteorological data , 2006 .

[127]  Gianni Bellocchi,et al.  An indicator of solar radiation model performance based on a fuzzy expert system , 2002 .

[128]  M. Janssen,et al.  Multi-Agent Systems for the Simulation of Land-Use and LandCover Change : A Review , 2002 .

[129]  Bernd Diekkrüger,et al.  Validity of agroecosystem models a comparison of results of different models applied to the same data set , 1995 .

[130]  P. Reich,et al.  A handbook of protocols for standardised and easy measurement of plant functional traits worldwide , 2003 .

[131]  R. Daren Harmel,et al.  Consideration of measurement uncertainty in the evaluation of goodness-of-fit in hydrologic and water quality modeling , 2007 .

[132]  Marcello Vichi,et al.  Link or sink: a modelling interpretation of the open Baltic biogeochemistry , 2004 .

[133]  Kazuhiko Kobayashi,et al.  Comments on another way of partitioning mean squared deviation proposed by Gauch et al. (2003) , 2004 .

[134]  Muhittin Oral,et al.  In search of a valid view of model validation for operations research , 1993 .

[135]  Anthony J. Jakeman,et al.  Ten iterative steps in development and evaluation of environmental models , 2006, Environ. Model. Softw..

[136]  Osman Balci,et al.  Verification, Validation And Accreditation Of Simulation Models , 1997, Winter Simulation Conference Proceedings,.

[137]  Kurt D. Pennell,et al.  Evaluation of Five Simulation Models for Predicting Aldicarb and Bromide Behavior Under Field Conditions , 1990 .

[138]  C. Willmott.  ON THE VALIDATION OF MODELS , 1981 .

[139]  Hsin-I Wu,et al.  A model comparison for daylength as a function of latitude and day of year , 1995 .

[140]  Kenneth W. Bauer,et al.  Covalidation of dissimilarly structured models , 1997, WSC '97.

[141]  D. Altman,et al.  Comparing methods of measurement: why plotting difference against standard method is misleading , 1995, The Lancet.

[142]  John Norton.  Prediction for decision-making under uncertainty , 2003 .

[143]  Jens Christian Refsgaard,et al.  Modelling guidelinesterminology and guiding principles , 2004 .

[144]  O. Klepper,et al.  A model of carbon flows in relation to macrobenthic food supply in the Oosterschelde estuary (S.W. Netherlands) , 1989 .

[145]  Robert Clement,et al.  On the validation of models of forest CO2 exchange using eddy covariance data: some perils and pitfalls. , 2005, Tree physiology.

[146]  Stephen P. Prisley,et al.  A synthesis of literature on evaluation of models for policy applications, with implications for forest carbon accounting , 2004 .

[147]  Aude Barbottin,et al.  The Azodyn crop model as a decision support tool for choosing cultivars , 2006 .

[148]  Marion R. Reynolds,et al.  Comparisons of some tests for validation of stochastic simulation models , 1982 .

[149]  Claudio O. Stöckle,et al.  Evaluation of estimated weather data for calculating Penman-Monteith reference crop evapotranspiration , 2004, Irrigation Science.

[150]  John A. Davies,et al.  Evaluation of selected models for estimating solar radiation on horizontal surfaces , 1989 .

[151]  Osman Balci,et al.  Validation of multivariate response models using Hotelling's two-sample T2 test , 1982 .

[152]  P. A. Leffelaar,et al.  On scale problems in modelling: an example from soil ecology. , 1990 .

[153]  Osman Balci,et al.  Some examples of simulation model validation using hypothesis testing , 1982, WSC.

[154]  Peter Checkland,et al.  Systems Thinking, Systems Practice , 1981 .

[155]  Robert M. Argent,et al.  An overview of model integration for environmental applications--components, frameworks and semantics , 2004, Environ. Model. Softw..

[156]  Santiago Beguería,et al.  Validation and Evaluation of Predictive Models in Hazard Assessment and Risk Management , 2006 .

[157]  P. Jones,et al.  Regional climate model simulations of daily maximum and minimum near-surface temperatures across Europe compared with observed station data 1961–1990 , 2004 .

[158]  João Miguel Dias,et al.  Implementation and assessment of hydrodynamic, salt and heat transport models: The case of Ria de Aveiro Lagoon (Portugal) , 2006, Environ. Model. Softw..

[159]  Gianni Bellocchi,et al.  Estimating monthly ( R)USLE climate input in a Mediterranean region using limited data , 2007 .

[160]  Keith Beven,et al.  Prophecy, reality and uncertainty in distributed hydrological modelling , 1993 .

[161]  Richard A. Berk,et al.  The Use of Statistical Tools for Evaluating Computer Simulations – An Editorial Essay , 2001 .

[162]  Marcel Van Oijen,et al.  On the use of specific publication criteria for papers on process-based modelling in plant science , 2002 .

[163]  Jack P. C. Kleijnen,et al.  Validation of Trace-Driven Simulation Models: A Novel Regression Test , 1998 .

[164]  Steffen Bayer,et al.  Business dynamics: Systems thinking and modeling for a complex world , 2004 .

[165]  Eugene R. Yagow.  Auxiliary Procedures for the AGNPS Model in Urban Fringe Watersheds , 1997 .

[166]  Randy Wayne Clouse.  Evaluation of GLEAMS considering parameter uncertainty , 1996 .

[167]  Gerrit Hoogenboom,et al.  The impact of potential errors in rainfall observation on the simulation of crop growth, development and yield , 2002 .

[168]  G. Arthur Mihram,et al.  Some Practical Aspects of the Verification and Validation of Simulation Models , 1972 .

[169]  Jan-Tai Kuo,et al.  Procedure to Calibrate and Verify Numerical Models of Estuarine Hydrodynamics , 1999 .

[170]  Klaas Metselaar,et al.  Auditing predictive models : a case study in crop growth , 1999 .

[171]  Jeffrey J. McDonnell,et al.  On the dialog between experimentalist and modeler in catchment hydrology: Use of soft data for multicriteria model calibration , 2002 .

[172]  R. C. Muchow,et al.  Modelling sugarcane production systems: II: Analysis of system performance and methodology issues , 2000 .

[173]  Michael J. Savage,et al.  Comparison of estimates of daily solar radiation from air temperature range for application in crop simulations , 2008 .

[174]  M. Coughenour,et al.  Photosynthesis, transpiration, and primary productivity: Scaling up from leaves to canopies and regions using process models and remotely sensed data , 2004 .

[175]  S. Harrison,et al.  Regression of a model on real-system output: An invalid test of model validity , 1990 .

[176]  D. Wallach,et al.  Mean squared error of prediction in models for studying ecological and agronomic systems , 1987 .

[177]  Paul D. Bates,et al.  Hydrological Science: model credibility and scientific integrity , 2001 .

[178]  K. H. Pollock,et al.  Biostatistics: A Foundation for Analysis in the Health Sciences. , 1976 .

[179]  G. Bellocchi,et al.  A review of methodologies to evaluate agroecosystems simulation models , 1999 .

[180]  Wei Li,et al.  Model validation software for classification models using repeated partitioning: MVREP , 2003, Comput. Methods Programs Biomed..

[181]  Thorsten Arnold,et al.  Step by step calibration of an Integrated Model System for irrigation management , 2008 .

[182]  R. C. Muchow,et al.  Modelling sugarcane production systems I. Development and performance of the sugarcane module , 1999 .

[183]  P. Krause,et al.  COMPARISON OF DIFFERENT EFFICIENCY CRITERIA FOR HYDROLOGICAL MODEL ASSESSMENT , 2005 .

[184]  Miroslav Trnka,et al.  Global solar radiation in Central European lowlands estimated by various empirical formulae , 2005 .

[185]  David Makowski,et al.  Comparison of models and indicators for categorizing soft wheat fields according to their grain protein contents , 2008 .

[186]  Soroosh Sorooshian,et al.  Calibration of rainfall‐runoff models: Application of global optimization to the Sacramento Soil Moisture Accounting Model , 1993 .

[187]  M. Katz.  Validation of models , 2006 .

[188]  Gunnar Abrahamson,et al.  Terminology for model credibility , 1980 .

[189]  M. Stephens.  EDF Statistics for Goodness of Fit and Some Comparisons , 1974 .

[190]  K. Buchgraber,et al.  A simple statistical model for predicting herbage production from permanent grassland , 2006 .

[191]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[192]  J. Nash,et al.  River flow forecasting through conceptual models part I — A discussion of principles☆ , 1970 .

[193]  David W. Hulse,et al.  Modeling biocomplexity - actors, landscapes and alternative futures , 2007, Environ. Model. Softw..

[194]  Gianni Bellocchi,et al.  Modelling solar radiation over complex terrains using monthly climatological data , 2007 .

[195]  Hugh G. Gauch,et al.  Model Evaluation by Comparison of Model-Based Predictions and Measured Values , 2003 .

[196]  Alan Agresti,et al.  Categorical Data Analysis , 1991, International Encyclopedia of Statistical Science.

[197]  Huub Scholten,et al.  Quantitative validation of deterministic models: when is a model acceptable? , 1998 .