Choosing the best model: Level of detail, complexity, and model performance

he lack of a methodology, or at least detailed guidelines, for choosing the best model in a mathematical or computer modelling study stems from a poor understanding of the precise ways in which the success of the study depends upon the particular model used. As a result, the choice of the best model is regarded as more of an art than a science. In order to improve the model selection process, model performance needs to be clearly defined, and suitable model attributes identified that can be used to predict the performance of the alternative candidate models. This paper distinguishes the different aspects of model performance and considers the extent to which they can be measured. The most common attributes used to compare alternative models are level of detail and complexity although these terms are used in a number of different ways. The meanings of these concepts are therefore discussed and the likely relationships with the model performance elements considered. The related area of simplification is reviewed and the areas in which further work is required are set out.

[1]  Richard W. Eglese,et al.  Operational Research Tutorial Papers , 1990 .

[2]  Paul A. Fishwick,et al.  The role of process abstraction in simulation , 1988, IEEE Trans. Syst. Man Cybern..

[3]  Morten Flatau Review Article: When Order is no longer Order-Organizing and the New Science of Complexity , 1995 .

[4]  B O Palsson,et al.  Model complexity has a significant effect on the numerical value and interpretation of metabolic sensitivity coefficients. , 1993, Journal of theoretical biology.

[5]  G. Chaitin Randomness and Mathematical Proof , 1975 .

[6]  Günter Blöschl,et al.  Point snowmelt models with different degrees of complexity — Internal processes , 1991 .

[7]  Stephen Ward,et al.  Arguments for Constructively Simple Models , 1989 .

[8]  Averill M. Law,et al.  A forum on crucial issues in the simulation of manufacturing systems , 1993, WSC '93.

[9]  John L. Casti,et al.  Connectivity, Complexity, and Catastrophe in Large-Scale Systems , 1980 .

[10]  F. Sklar,et al.  Articulation, accuracy and effectiveness of mathematical models: A review of freshwater wetland applications☆ , 1985 .

[11]  George S. Innis,et al.  Simulation model simplification techniques , 1983 .

[12]  Sallie M. Henry,et al.  Software Structure Metrics Based on Information Flow , 1981, IEEE Transactions on Software Engineering.

[13]  Brian Falkenhainer,et al.  Compositional Modeling: Finding the Right Model for the Job , 1991, Artif. Intell..

[14]  Ewart R. Carson,et al.  Dealing with complexity: an introduction to the theory & applications of systemsscience , 1988 .

[15]  Suleyman Sevinc,et al.  AUTOMATION OF SIMPLIFICATION IN DISCRETE EVENT MODELLING AND SIMULATION , 1990 .

[16]  Tom DeMarco,et al.  Controlling software projects : management, measurement & estimation , 1982 .

[17]  William B. Rouse,et al.  On Measuring the Complexity of Monitoring and Controlling Large-Scale Systems , 1986, IEEE Transactions on Systems, Man, and Cybernetics.

[18]  Sanjaya Addanki,et al.  Graphs of Models , 1991, Artif. Intell..

[19]  Norman E. Fenton,et al.  Software Metrics: A Rigorous Approach , 1991 .

[20]  Enver Yücesan,et al.  Complexity of simulation models: a graph theoretic approach , 1993, WSC '93.

[21]  John M. Starkey,et al.  EFFECTS OF MODEL COMPLEXITY ON THE PERFORMANCE OF AUTOMATED VEHICLE STEERING CONTROLLERS : MODEL DEVELOPMENT, VALIDATION AND COMPARISON , 1995 .

[22]  William Gurney,et al.  Individual-based models: combining testability and generality , 1992 .

[23]  Thomas R. Gruber,et al.  Model formulation as a problem‐solving task: Computer‐assisted engineering modeling , 1993, Int. J. Intell. Syst..

[24]  Efraim Halfon,et al.  Is there a best model structure? II. Comparing the model structures of different fate models , 1983 .

[25]  J. Doyne Farmer,et al.  A Rosetta stone for connectionism , 1990 .

[26]  John Fripp How effective are models , 1985 .

[27]  Michael Pidd,et al.  Computer Simulation in Management Science (3rd Edition) , 1998 .

[28]  Donn G. Decoursey Developing Models with More Detail: Do More Algorithms Give More Truth? , 1992 .

[29]  J. Banks,et al.  Discrete-Event System Simulation , 1995 .

[30]  Pierre-Jacques Courtois,et al.  On time and space decomposition of complex structures , 1985, CACM.

[31]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[32]  Paul K. Davis,et al.  Families of models that cross levels of resolution: issues for design, calibration and management , 1993, WSC '93.

[33]  Janusz S. Kowalik,et al.  Mathematical modeling with computers , 1980 .

[34]  Eric Rexstad,et al.  Model simplification — Three applications , 1985 .

[35]  Harold J. Morowitz,et al.  The emergence of complexity , 1995, Complex..

[36]  Roger J. Brooks,et al.  Determining the range of predictions of a groundwater model which arises from alternative calibrations , 1994 .

[37]  Dennis B. Webster,et al.  Determining the level of detail in a simulation model. A case study , 1984 .

[38]  Saul I. Gass What is a computer-based mathematical model? , 1983 .

[39]  Anas N. Al-Rabadi,et al.  A comparison of modified reconstructability analysis and Ashenhurst‐Curtis decomposition of Boolean functions , 2004 .

[40]  W. Weaver Science and complexity. , 1948, American scientist.

[41]  Lee W. Schruben,et al.  Simulation modeling with event graphs , 1983, CACM.

[42]  Bernard P. Zeigler,et al.  Chapter 1 – MULTILEVEL MULTIFORMALISM MODELING: AN ECOSYSTEM EXAMPLE , 1979 .

[43]  A. Jakeman,et al.  How much complexity is warranted in a rainfall‐runoff model? , 1993 .

[44]  David Garfinkel Modeling of inherently complex biological systems: problems, strategies, and methods , 1984 .

[45]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[46]  W K Durfee,et al.  Control of standing and gait using electrical stimulation: influence of muscle model complexity on control strategy. , 1993, Progress in brain research.

[47]  Mario Bunge The Myth of Simplicity: Problems of Scientific Philosophy. , 1964 .

[48]  Efraim Halfon,et al.  Is there a best model structure? I. Modeling the fate of a toxic substance in a lake , 1983 .

[49]  Robert Rosen,et al.  COMPLEXITY AS A SYSTEM PROPERTY , 1977 .

[50]  Christopher G. Langton,et al.  Computation at the edge of chaos: Phase transitions and emergent computation , 1990 .

[51]  John David Salt,et al.  Simulation should be easy and fun! , 1993, WSC '93.

[52]  Victor E. Middleton,et al.  Proceedings of the 1993 Winter Simulation Conference , 1993 .

[53]  L. George TESTS FOR SYSTEM COMPLEXITY , 1977 .

[54]  Claudio O. Stöckle,et al.  Canopy photosynthesis and transpiration estimates using radiation interception models with different levels of detail , 1992 .

[55]  Bernard P. Zeigler Multifaceted modeling methodology: Grappling with the irreducible complexity of systems , 1984 .

[56]  HERBERT A. SIMON,et al.  The Architecture of Complexity , 1991 .

[57]  J. D. Johannes,et al.  Systems Simulation: The Art and Science , 1975, IEEE Transactions on Systems, Man, and Cybernetics.

[58]  V. Manno,et al.  A MEASURE OF THE DIFFICULTY OF SYSTEM DIAGNOSIS AND ITS RELATIONSHIP TO COMPLEXITY , 1989 .

[59]  Marius S. Potgieter,et al.  The simulation of Forbush decreases with time-dependent cosmic-ray modulation models of varying complexity , 1991 .

[60]  P. Pandurang Nayak Automated Modelling of Physical Systems , 1995 .

[61]  Bernard P. Zeigler,et al.  Theory of Modelling and Simulation , 1979, IEEE Transactions on Systems, Man and Cybernetics.

[62]  William S. Meisel,et al.  Repro-Modeling: An Approach to Efficient Model Utilization and Interpretation , 1973, IEEE Trans. Syst. Man Cybern..

[63]  M. J. Lawrence,et al.  The population genetics of the self-incompatibility polymorphism in Papaver rhoeas. XI. The effects of limited pollen and seed dispersal, overlapping generations and variation in plant size on the variance of S-allele frequencies in populations at equilibrium , 1996, Heredity.

[64]  Richard N. Palmer,et al.  COMPLEXITY IN COLUMBIA RIVER SYSTEMS MODELING , 1986 .

[65]  C. B. Tilanus Failures and successes of quantitative methods in management , 1983 .

[66]  Murray Gell-Mann,et al.  What is complexity? Remarks on simplicity and complexity by the Nobel Prize-winning author of The Quark and the Jaguar , 1995, Complex..