Calibrating Expert Assessments of Advanced Aerospace Technology Adoption Impact

CALIBRATING EXPERT ASSESSMENTS OF ADVANCED AEROSPACE TECHNOLOGY ADOPTION IMPACT Brace A. Conway Old Dominion University Director: Dr. Resit Unal This dissertation describes the development of expert judgment calibration methodology as part of elicitation of the expert j udgments to assist in the task of quantifying parameter uncertainty for proposed new aerospace vehicles. From previous work, it has been shown that experts in the field of aerospace systems design and development can provide valuable input into the sizing and conceptual design of future space launch vehicles employing advanced technology. In particular (and of specific interest in this case), assessment of operations and support cost implications o f adopting proposed new technology is frequently asked of the experts. Often the input consisting of estimates and opinions is imprecise and may be offered with less than a high degree of confidence in its efficacy. Since the sizing and design of advanced space or launch vehicles must ultimately have costs attached to them (for subsequent program advocacy and tradeoff studies), the lack of precision in parameter estimates will be detrimental to the development of viable cost models to support the advocacy and tradeoffs. It is postulated that a system, which could accurately apply a measure of calibration to the imprecise and/or low-confidence estimates of the surveyed experts, would greatly enhance the derived parametric data. The development of such a calibration aid has been the thrust of this effort. Bayesian network methodology, augmented by uncertainty modeling and aggregation techniques, among others, were employed in the tool Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. construction. Appropriate survey questionnaire instruments were compiled for use in acquiring the experts’ input; the responses served as input to a test case for validation of the resulting calibration model. Application of the derived techniques were applied as part o f a larger expert assessment elicitation and aggregation study. Results of this research show that calibration of expert judgments, particularly for far-term events, appears to be possible. Suggestions for refinement and extension of the development are

[1]  M. Degroot,et al.  Bayesian Analysis and Uncertainty in Economic Theory. , 1986 .

[2]  L. James,et al.  Estimating within-group interrater reliability with and without response bias. , 1984 .

[3]  Lazar Stankov,et al.  Age differences in the realism of confidence judgements: A calibration study using tests of fluid and crystallized intelligence , 1996 .

[4]  Ramon López de Mántaras,et al.  Approximate Reasoning Models , 1990 .

[5]  G. Keren.  Calibration and probability judgements: Conceptual and methodological issues , 1991 .

[6]  A. H. Murphy,et al.  Credible Interval Temperature Forecasting: Some Experimental Results , 1974 .

[7]  Terry L. Hardy.  Fuzzy Logic Approaches to Multi-Objective Decision-Making in Aerospace Applications , 1994 .

[8]  Johnnie E. V. Johnson,et al.  Calibration of Subjective Probability Judgments in a Naturalistic Setting. , 2001, Organizational behavior and human decision processes.

[9]  Rex V. Brown,et al.  Research and the credibility of estimates , 1969 .

[10]  Baruch Fischhoff,et al.  The effect of temporal setting on likelihood estimates , 1976 .

[11]  R. Cooke.  Elicitation of expert opinions for uncertainty and risks , 2003 .

[12]  Martin Neil,et al.  Building large-scale Bayesian networks , 2000, The Knowledge Engineering Review.

[13]  Colin F. Camerer,et al.  Taking Risks: The Management of Uncertainty , 1988 .

[14]  Max Henrion,et al.  Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis , 1990 .

[15]  B. Kosko.  Fuzzy Thinking: The New Science of Fuzzy Logic , 1993 .

[16]  Roger A. Lepsch,et al.  Using Expert Judgement Methodology to Address Uncertainty in Launch Vehicle Weight Estimates , 2002 .

[17]  Curley,et al.  The Effects of Subject-Defined Categories on Judgmental Accuracy in Confidence Assessment Tasks. , 1999, Organizational behavior and human decision processes.

[18]  Michael D. McNeese,et al.  Capturing multiple perspectives: a user-centered approach to knowledge and design acquisition , 1993 .

[19]  Lotfi A. Zadeh,et al.  Knowledge Representation in Fuzzy Logic , 1996, IEEE Trans. Knowl. Data Eng..

[20]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[21]  William A. Wallace,et al.  Elecitation of Knowledge from Multiple Experts Using Network Inference , 1997, IEEE Trans. Knowl. Data Eng..

[22]  F. Albert Wang,et al.  Overconfidence, Investor Sentiment, and Evolution , 2000 .

[23]  S. A. Schmitt,et al.  Measuring Uncertainty: An Elementary Introduction to Bayesian Statistics. , 1970 .

[24]  Jelle van Lenthe,et al.  Scoring-Rule Feedforward and the Elicitation of Subjective Probability Distributions , 1994 .

[25]  B. P. M. Duarte,et al.  The expected utility theory applied to an industrial decision problem - what technological alternative to implement to treat industrial solid residuals , 2001, Comput. Oper. Res..

[26]  Mark Simon,et al.  Cognitive biases, risk perception, and venture formation: How individuals decide to start companies , 2000 .

[27]  Grigoris Antoniou,et al.  A tutorial on default reasoning , 1998, The Knowledge Engineering Review.

[28]  George Wright,et al.  Coherence, Calibration, and Expertise in Judgmental Probability Forecasting , 1994 .

[29]  Mark T. Spence.  Problem-Problem Solver Characteristics Affectingthe Calibration of Judgments , 1996 .

[30]  George Wright,et al.  Changes in the realism and distribution of probability assessments as a function of question type , 1982 .

[31]  B. H. Beach.  Expert judgment about uncertainty: Bayesian decision making in realistic settings. , 1975, Organizational behavior and human performance.

[32]  George Wright,et al.  Delphi: A reevaluation of research and theory , 1991 .

[33]  Richard W. Monroe,et al.  A Synthesized Methodology for Eliciting Expert Judgment for Addressing Uncertainty in Decision Analysis , 1997 .

[34]  Linda C. van der Gaag,et al.  Sensitivity analysis: an aid for belief-network quantification , 2000, The Knowledge Engineering Review.

[35]  Mike Keppell.  Optimizing instructional designer---subject matter expert communication in the design and development of multimedia projects , 2001 .

[36]  R. E. Schaefer,et al.  The evaluation of individual and aggregated subjective probability distributions , 1976 .

[37]  John R. Olds,et al.  Multidisciplinary Conceptual Design Optimization of Space Transportation Systems , 1999 .

[38]  David J. Weiss,et al.  Performance-based assessment of expertise: How to decide if someone is an expert or not , 2002, Eur. J. Oper. Res..

[39]  Silja Renooij,et al.  Probability elicitation for belief networks: issues to consider , 2001, The Knowledge Engineering Review.

[40]  H. A. Lingstone,et al.  The Delphi Method: Techniques and Applications , 1976 .

[41]  W Zuk.  AN EXPERT SYSTEM FOR THE ESTHETIC RATING OF BRIDGES. FINAL REPORT , 1990 .

[42]  A. Dawid.  The Well-Calibrated Bayesian , 1982 .

[43]  Mark A. Kolb.  Object modeling with constraints for aerospace conceptual/preliminary design , 1994, Defense, Security, and Sensing.

[44]  P. Feltovich,et al.  Issues of expert flexibility in contexts characterized by complexity and change , 1997 .

[45]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[46]  Mohammed Khurshid Khan,et al.  Development of an Expert System for implementation of ISO 9000 quality systems , 1999 .