Validating behavioral models for reuse

When using a model to predict the behavior of a physical system of interest, engineers must be confident that, under the conditions of interest, the model is an adequate representation of the system. The process of building this confidence is called model validation. It requires that engineers have knowledge about the system and conditions of interest, properties of the model and their own tolerance for uncertainty in the predictions. To reduce time and costs, engineers often reuse preexisting models that other engineers have developed. However, if the user lacks critical parts of this knowledge, model validation can be as time consuming and costly as developing a similar model from scratch. In this article, we describe a general process for performing model validation for reused behavioral models that overcomes this problem by relying on the formalization and exchange of knowledge. We identify the critical elements of this knowledge, discuss how to represent it and demonstrate the overall process on a simple engineering example.

[1]  Michael Pidd,et al.  Simulation model reuse: definitions, benefits and obstacles , 2004, Simul. Model. Pract. Theory.

[2]  Robert T. Clemen,et al.  Making Hard Decisions: An Introduction to Decision Analysis , 1997 .

[3]  Robert G. Sargent,et al.  An expository on verification and validation of simulation models , 1985, WSC '85.

[4]  Ram D. Sriram,et al.  The NIST Design Repository Project , 1999 .

[5]  K. Popper,et al.  The Logic of Scientific Discovery , 1960 .

[6]  Varol Akman,et al.  Steps Toward Formalizing Context , 1996, AI Mag..

[7]  Scott Ferson,et al.  Probability bounds analysis , 1998 .

[8]  Thomas H. Naylor,et al.  Verification of Computer Simulation Models , 1967 .

[9]  Ramanathan V. Guha,et al.  Varieties of Contexts , 2003, CONTEXT.

[10]  P. Walley.  Statistical Reasoning with Imprecise Probabilities , 1990 .

[11]  Joseph Edward Shigley,et al.  Mechanical engineering design , 1972 .

[12]  R. Bajcsy.  IJCAI-93 : proceedings of the Thirteenth International Joint Conference on Artificial Intelligence , Chambéry, France, August 28-September 3, 1993 , 1993 .

[13]  Armen Der Kiureghian,et al.  MEASURES OF STRUCTURAL SAFETY UNDER IMPERFECT STATES OF KNOWLEDGE , 1989 .

[14]  Jack C. Wileden,et al.  Ontologies for supporting engineering analysis models , 2005, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[15]  Christiaan J. J. Paredis,et al.  Managing Multiple Sources of Epistemic Uncertainty in Engineering Decision Making , 2007 .

[16]  T. Biggerstaff,et al.  Reusability Framework, Assessment, and Directions , 1987, IEEE Software.

[17]  P. Sanders.  DoD Modeling and Simulation (M&S) Verification, Validation, and Accreditation (VV&A), , 1996 .

[18]  Russell S. Peak,et al.  A KNOWLEDGE REPOSITORY FOR BEHAVIORAL MODELS IN ENGINEERING DESIGN , 2004 .

[19]  Brian Falkenhainer,et al.  Compositional Modeling: Finding the Right Model for the Job , 1991, Artif. Intell..

[20]  Paul Herskovitz.  A Theoretical Framework For Simulation Validation: Popper’S Falsifications , 1991 .

[21]  Charles W. Krueger,et al.  Software reuse , 1992, CSUR.

[22]  Christiaan J. J. Paredis,et al.  Eliminating Design Alternatives Based on Imprecise Information , 2006 .

[23]  Douglas B. Lenat,et al.  Language, representation and contexts , 1992 .

[24]  Richard J. Malak,et al.  A FRAMEWORK FOR VALIDATING REUSABLE BEHAVIORAL MODELS IN ENGINEERING DESIGN , 2005 .

[25]  Gareth W. Parry,et al.  The characterization of uncertainty in probabilistic risk assessments of complex systems , 1996 .

[26]  C. Paredis,et al.  The Value of Using Imprecise Probabilities in Engineering Design , 2006 .

[27]  Yaman Barlas,et al.  Philosophical roots of model validation: Two paradigms , 1990 .

[28]  S Kafandaris,et al.  The Economic Value of Information , 2001, J. Oper. Res. Soc..

[29]  S. Ferson,et al.  Different methods are needed to propagate ignorance and variability , 1996 .

[30]  Russell R. Barton,et al.  Proceedings of the 2000 winter simulation conference , 2000 .

[31]  Levent Yilmaz,et al.  On the Need for Contextualized Introspective Models to Improve Reuse and Composability of Defense Simulations , 2004 .

[32]  Timothy G. Trucano,et al.  Verification and Validation in Computational Fluid Dynamics , 2002 .

[33]  William L. Oberkampf,et al.  Estimation of Total Uncertainty in Modeling and Simulation , 2000 .

[34]  I. A. Chaikovsky,et al.  Terminology for model credibility , 1979 .

[35]  Osman Balci,et al.  Well-defined intended uses: an explicit requirement for accreditation of modeling and simulation applications , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[36]  George B. Kleindorfer,et al.  THE PHILOSOPHY OF SCIENCE AND VAL1DATION IN SIMULATION , 1993 .

[37]  T. Penelhum.  A Treatise of Human Nature (review) , 2000 .

[38]  Hans-Ludwig Hausen,et al.  Software evaluation for certification - principles, practice, and legal liability , 1995, International software quality assurance series.

[39]  Herbert A. Simon,et al.  The Sciences of the Artificial , 1970 .

[40]  Varol Akman,et al.  The Use of Situation Theory in Context Modeling , 1997, Comput. Intell..

[41]  Efstratios Nikolaidis,et al.  Types of Uncertainty in Design Decision Making , 2004 .

[42]  Christiaan J. J. Paredis,et al.  Managing the collection of information under uncertainty using information economics , 2006 .

[43]  Gunnar Abrahamson,et al.  Terminology for model credibility , 1980 .

[44]  Roy Scrudder,et al.  Enabling simulation interoperability , 2006, Computer.

[45]  Victor R. Basili,et al.  How reuse influences productivity in object-oriented systems , 1996, CACM.

[46]  Efstratios Nikolaidis,et al.  Engineering Design Reliability Handbook , 2004 .

[47]  John McCarthy,et al.  Notes on Formalizing Context , 1993, IJCAI.

[48]  Paul F. Reynolds,et al.  A case study of model context for simulation composability and reusability , 2005, Proceedings of the Winter Simulation Conference, 2005..

[49]  H. Simon,et al.  The sciences of the artificial (3rd ed.) , 1996 .

[50]  E. L. Harder,et al.  The Institute of Electrical and Electronics Engineers, Inc. , 2019, 2019 IEEE International Conference on Software Architecture Companion (ICSA-C).