Engineering judgement in reliability and safety and its limits: what can we learn from research in psychology

Engineering judgement has an important role in safety or reliability assessment. This paper focuses on the use of engineering judgement for integrating diverse evidence into an assessment of the safety or reliability of a product. In many cases of stringent safety requirements, this form of engineering (or "expert") judgement, i.e., "informal inference from complex evidence", is the crucial resource for the decision maker, for lack of more solid, objective evidence. This dependence on judgement is especially evident in the assessment of the unreliability due to possible design faults in complex products, and computer software in particular. Although engineering judgement plays an essential role in the assessment, there are good reasons to doubt the ability of experts in some of the judgement tasks in which they are usually employed. Experimental research both about the way humans think and integrate evidence, and about the performance of experts in tasks similar to engineering judgement, support the idea that the ability of experts may be overrated. This paper summarises some literature about common fallacies and ways to guard against them, and argues for a more disciplined use of expert judgement.

[1]  Nancy G. Leveson,et al.  An investigation of the Therac-25 accidents , 1993, Computer.

[2]  H. Jungermann.  The Two Camps on Rationality , 1983 .

[3]  Bev Littlewood,et al.  Validation of ultrahigh dependability for software-based systems , 1993, CACM.

[4]  M. C. Thorne,et al.  The use of expert opinion in formulating conceptual models of underground disposal systems and the treatment of associated bias , 1993 .

[5]  Baruch Fischhoff,et al.  Calibration of Probabilities: The State of the Art , 1977 .

[6]  T. A. Wheeler,et al.  Use of expert judgment in NUREG--1150 , 1991 .

[7]  A. Tversky,et al.  Intuitive Prediction: Biases and Corrective Procedures , 1982 .

[8]  R. Dawes.  Judgment under uncertainty: The robust beauty of improper linear models in decision making , 1979 .

[9]  P. Powell.  Expertise and Decision Support , 2013 .

[10]  Miss A.O. Penney.  (b) , 1974, The New Yale Book of Quotations.

[11]  Eleonora Curlo,et al.  Causal inference as a cognitive strategy , 1993, J. Exp. Theor. Artif. Intell..

[12]  B. Fischhoff,et al.  Judgment under uncertainty: Debiasing , 1982 .

[13]  James Reason,et al.  Human Error , 1990 .

[14]  A. Tversky,et al.  BELIEF IN THE LAW OF SMALL NUMBERS , 1971, Pediatrics.

[15]  Ab McClelland,et al.  The calibration of subjective probabilities: Theories and models 1980-1993 , 1994 .

[16]  H. J. Einhorn.  Expert judgment: Some necessary conditions and an example. , 1974 .

[17]  Baruch Fischhoff,et al.  Risk assessment: Evaluating error in subjective estimates , 1981 .

[18]  Robert L. Glass,et al.  Science and substance: a challenge to software engineers , 1994, IEEE Software.

[19]  Peter Ayton,et al.  On the Competence and Incompetence of Experts , 1992 .

[20]  A. Tversky,et al.  On the psychology of prediction , 1973 .

[21]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[22]  Baruch Fischhoff,et al.  Judgment under uncertainty: For those condemned to study the past: Heuristics and biases in hindsight , 1982 .

[23]  Lorenzo Strigini.  Considerations on current research issues in software safety , 1994 .

[24]  Erik Hollnagel.  The reliability of expert systems , 1989 .

[25]  Baruch Fischhoff,et al.  Judgment under uncertainty: Facts versus fears: Understanding perceived risk , 1982 .

[26]  Ralph L. Keeney,et al.  Eliciting probabilities from experts in complex technical problems , 1991 .

[27]  Stan Kaplan,et al.  ‘Expert information’ versus ‘expert opinions’. Another approach to the problem of eliciting/ combining/using expert knowledge in PRA , 1992 .

[28]  Keung-Chi Ng,et al.  Uncertainty management in expert systems , 1990, IEEE Expert.

[29]  G. Gigerenzer.  Why the distinction between single-event probabilities and frequencies is important for psychology (and vice versa). , 1994 .

[30]  George Wright,et al.  The quality of expert probability judgement: issues and analysis , 1994 .

[31]  B. Fischhoff,et al.  Assessing uncertainty in physical constants , 1986 .

[32]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[33]  Roger M. Cooke,et al.  Expert Opinions as Data Source: Methods and Experiences , 1989 .

[34]  George Apostolakis,et al.  A taxonomy of issues related to the use of expert judgments in probabilistic safety studies , 1992 .

[35]  Hans Jörg Wingender.  Reliability Data Collection and Use in Risk and Availability Assessment , 1986 .

[36]  Victor R. Basili,et al.  Software process evolution at the SEL , 1994, IEEE Software.