Harnessing Expert Knowledge: Defining Bayesian Network Model Priors From Expert Knowledge Only—Prior Elicitation for the Vibration Qualification Problem

As systems become more complex, systems engineers rely on experts to inform decisions. This challenges systems engineers as they strive to plan activities such as, qualification in an environment where technical constraints are coupled with the traditional cost, risk, and schedule constraints. Bayesian network (BN) models provide a framework to aid systems engineers in planning qualification efforts with complex constraints by harnessing expert knowledge and incorporating technical factors. By quantifying causal factors, a BN model can provide data about the risk of implementing a decision supplemented with information on driving factors. This allows a systems engineer to make informed decisions and examine “what-if” scenarios. This paper discusses a novel process developed to define the probability distributions, or priors, in a BN model based primarily on expert knowledge supplemented with extremely limited data (25 datasets or less). The model was developed to aid qualification decisions—specifically to predict the suitability of six degrees of freedom vibration testing for qualification. The process defined the model priors primarily with expert knowledge in an unbiased manner. Validation of the model provided evidence that this novel process might be an effective tool in harnessing expert knowledge for a BN model.

[1]  Mark Blackburn,et al.  Harnessing expert knowledge: Defining a Bayesian network decision model with limited data–Model structure for the vibration qualification problem , 2018, Syst. Eng..

[2]  Judea Pearl,et al.  Bayesian Networks , 1998, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[3]  Ken Stein,et al.  A comparison of two methods for expert elicitation in health technology assessments , 2016, BMC Medical Research Methodology.

[4]  William Marsh,et al.  From complex questionnaire and interviewing data to intelligent Bayesian Network models for medical decision support , 2016, Artif. Intell. Medicine.

[5]  Mark Blackburn,et al.  Use of Bayesian Networks for Qualification Planning: Early Results of Factor Analysis , 2016 .

[6]  Mark R. Blackburn,et al.  Use of Bayesian Networks for Qualification Planning: A Predictive Analysis Framework for a Technically Complex Systems Engineering Problem , 2015, Complex Adaptive Systems.

[7]  Jeremy E. Oakley,et al.  A web-based tool for eliciting probability distributions from experts , 2014, Environ. Model. Softw..

[8]  Miroslaw J. Skibniewski,et al.  Decision support analysis for safety control in complex project environments based on Bayesian Networks , 2013, Expert Syst. Appl..

[9]  Hans J. Pasman,et al.  Bayesian networks make LOPA more effective, QRA more transparent and flexible, and thus safety more definable! , 2013 .

[10]  Kerrie L. Mengersen,et al.  A proposed validation framework for expert elicited Bayesian Networks , 2013, Expert Syst. Appl..

[11]  Serena H. Chen,et al.  Good practice in Bayesian network modelling , 2012, Environ. Model. Softw..

[12]  Bruce G. Marcot,et al.  Metrics for evaluating performance and uncertainty of Bayesian network models , 2012 .

[13]  Andrés Cano,et al.  A Method for Integrating Expert Knowledge When Learning Bayesian Networks From Data , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[14]  Avi Pfeffer,et al.  Practical Probabilistic Programming , 2016, ILP.

[15]  Judith Rousseau,et al.  Combining expert opinions in prior elicitation , 2010 .

[16]  Hareton K. N. Leung,et al.  Assessment of Catastrophic Risk Using Bayesian Network Constructed from Domain Knowledge and Spatial Data , 2010, Risk analysis : an official publication of the Society for Risk Analysis.

[17]  Kaisa Simola,et al.  Application of a Bayesian model for the quantification of the European methodology for qualification of non-destructive testing , 2010 .

[18]  Kerrie Mengersen,et al.  Comparison of four expert elicitation methods: For Bayesian logistic regression and classification trees , 2009 .

[19]  Milo D Adkison,et al.  Drawbacks of complex models in frequentist and Bayesian approaches to natural-resource management. , 2009, Ecological applications : a publication of the Ecological Society of America.

[20]  David O. Smallwood,et al.  Evaluation of a six-DOF electrodynamic shaker system. , 2008 .

[21]  Lakhmi C. Jain,et al.  Introduction to Bayesian Networks , 2008 .

[22]  M. Kynn The ‘heuristics and biases’ bias in expert elicitation , 2007 .

[23]  Jeremy E. Oakley,et al.  Uncertain Judgements: Eliciting Experts' Probabilities , 2006 .

[24]  P.W. Marshall Interdisciplinary Aspects of Cost-Risk Tradeoffs , 2006, OCEANS 2006.

[25]  J. Berger The case for objective Bayesian analysis , 2006 .

[26]  Ravi Paul,et al.  Analyzing the structure of expert knowledge , 2006, Inf. Manag..

[27]  Mary Kynn,et al.  Eliciting Expert Knowledge for Bayesian Logistic Regression in Species Habitat Modelling , 2005 .

[28]  Vibration Test Evolution Single-Axis , Single-Shaker to 6 DoF , 2003 .

[29]  Hermann von Hasseln,et al.  Object-Oriented Dynamic Bayesian Network-Templates for Modelling Mechatronic Systems , 2002 .

[30]  David Heckerman,et al.  A Tutorial on Learning with Bayesian Networks , 1999, Innovations in Bayesian Networks.

[31]  Finn Verner Jensen,et al.  Introduction to Bayesian Networks , 2008, Innovations in Bayesian Networks.

[32]  E. Salas,et al.  Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. , 1993 .

[33]  G. Johanson,et al.  Paired Comparison Intransitivity: Useful Information or Nuisance?. , 1993 .

[34]  Peder J. Johnson,et al.  Assessing Structural Knowledge. , 1991 .

[35]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[36]  R. L. Winkler The Assessment of Prior Distributions in Bayesian Analysis , 1967 .