Assurance and Assurance Cases

Assurance provides confidence that a system will work as required and not cause harm. Confidence is based on justified beliefs about the system and its environment, and justification can be developed and documented as an assurance case comprised of a structured argument grounded on evidence. For justification to be compelling, the argument must be indefeasible, meaning that we have so thoroughly considered everything that can go wrong (i.e., hazards to the system and defeaters to the argument) that there is no new information that could change our assessment. I show how the obligation for indefeasible justification can guide construction and interpretation of the argument and the evidence in an assurance case and how confidence in the case translates to bounds on the risk posed by the system. Assurance requires predictability in both the system and its environment; I speculate how credible assurance may be provided for recent and forthcoming systems where both kinds of predictability may be lacking.

[1]  Tim Kelly,et al.  Arguing Safety - A Systematic Approach to Managing Safety Cases , 1998 .

[2]  Grigore Rosu,et al.  Synthesizing Monitors for Safety Properties , 2002, TACAS.

[3]  J. Pollock Cognitive Carpentry: A Blueprint for How to Build a Person , 1995 .

[4]  Bev Littlewood,et al.  Conservative Reasoning about the Probability of Failure on Demand of a 1-out-of-2 Software-Based System in Which One Channel Is "Possibly Perfect" , 2013, IEEE Transactions on Software Engineering.

[5]  Seth Ahrenbach,et al.  Reasoning About Safety-Critical Information Flow Between Pilot and Computer , 2017, NFM.

[6]  Sasikumar Punnekkat,et al.  Improving Intelligent Vehicle Dependability by Means of Infrastructure-Induced Tests , 2015, 2015 IEEE International Conference on Dependable Systems and Networks Workshops.

[7]  Daniel Elenius,et al.  Automating Financial Regulatory Compliance Using Ontology+Rules and Sunflower , 2016, SEMANTICS.

[8]  Keith Lehrer,et al.  Knowledge: Undefeated Justified True Belief , 1969 .

[9]  Peter G. Bishop,et al.  A Methodology for Safety Case Development , 2000, SSS.

[10]  Min Wu,et al.  Safety Verification of Deep Neural Networks , 2016, CAV.

[11]  Bev Littlewood,et al.  Conceptual Modeling of Coincident Failures in Multiversion Software , 1989, IEEE Trans. Software Eng..

[12]  Anish Arora,et al.  Detectors and correctors: a theory of fault-tolerance components , 1998, Proceedings. 18th International Conference on Distributed Computing Systems (Cat. No.98CB36183).

[13]  Max Henderson,et al.  What you don't know won't hurt you: Information given to patients about the side-effects of antipsychotic drugs , 2000 .

[14]  David Wright,et al.  Modeling the probability of failure on demand (pfd) of a 1-out-of-2 system in which one channel is "quasi-perfect" , 2017, Reliab. Eng. Syst. Saf..

[15]  John Rushby,et al.  Using model checking to help discover mode confusions and other automation surprises , 2002, Reliab. Eng. Syst. Saf..

[16]  Tomoji Shogenji The degree of epistemic justification and the conjunction fallacy , 2009, Synthese.

[17]  John Rushby,et al.  The Interpretation and Evaluation of Assurance Cases , 2015 .

[18]  John M. Rushby,et al.  Runtime Certification , 2008, RV.

[19]  Bev Littlewood,et al.  Reasoning about the Reliability of Diverse Two-Channel Systems in Which One Channel Is "Possibly Perfect" , 2012, IEEE Transactions on Software Engineering.

[20]  Susmit Jha,et al.  Automated Synthesis of Safe Autonomous Vehicle Control Under Perception Uncertainty , 2016, NFM.

[21]  D. L. Simms,et al.  Normal Accidents: Living with High-Risk Technologies , 1986 .

[22]  Bev Littlewood,et al.  Conservative Bounds for the pfd of a 1-out-of-2 Software-Based System Based on an Assessor's Subjective Probability of "Not Worse Than Independence" , 2013, IEEE Transactions on Software Engineering.

[23]  Richard Hawkins,et al.  A New Approach to creating Clear Safety Arguments , 2011, SSS.

[24]  David Atkinson,et al.  Confirmation and justification. A commentary on Shogenji’s measure , 2009, Synthese.

[25]  Michael Fisher,et al.  Verifying autonomous systems , 2013, CACM.

[26]  Michael Fisher,et al.  Formal Methods for the Certification of Autonomous Unmanned Aircraft Systems , 2011, SAFECOMP.

[27]  Lorenzo Strigini,et al.  Assessing the Risk due to Software Faults: Estimates of Failure Rate versus Evidence of Perfection , 1998, Softw. Test. Verification Reliab..

[28]  Branden Fitelson,et al.  STUDIES IN BAYESIAN CONFIRMATION THEORY , 2001 .

[29]  Carlo Ghezzi,et al.  Self-adaptive software needs quantitative verification at runtime , 2012, CACM.

[30]  Daniel N. Osherson,et al.  Comparison of confirmation measures q,qq , 2007 .

[31]  Xuejun Yang,et al.  Finding and understanding bugs in C compilers , 2011, PLDI '11.

[32]  John A. McDermid,et al.  Safety Case Development: Current Practice, Future Prospects , 1997 .

[33]  Xi Wang,et al.  An Empirical Study on the Correctness of Formally Verified Distributed Systems , 2017, EuroSys.

[34]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[35]  Lorenzo Strigini,et al.  Assessing the risk due to software faults: estimates of failure rate versus evidence of perfection , 1998 .

[36]  B. Littlewood,et al.  The Use of Multi-legged Arguments to Increase Confidence in Safety Claims for Software-based Systems : a Study Based on a BBN Analysis of an Idealised Example , 2005 .

[37]  Koushik Sen,et al.  Rule-Based Runtime Verification , 2004, VMCAI.

[38]  Peter G. Bishop,et al.  A conservative theory for long term reliability growth prediction , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.

[39]  Henry Prakken,et al.  The Carneades model of argument and burden of proof , 2007, Artif. Intell..

[40]  Mahesh Viswanathan,et al.  Runtime Assurance Based On Formal Specifications , 1999, PDPTA.

[41]  Rajeev Alur,et al.  Regular Programming for Quantitative Properties of Data Streams , 2016, ESOP.

[42]  Peter Bishop Does Software Have to Be Ultra Reliable in Safety Critical Systems? , 2013, SAFECOMP.

[43]  Eugene Lavretsky,et al.  Adaptive Control and the NASA X-15-3 Flight Revisited , 2010, IEEE Control Systems.

[44]  Lui Sha,et al.  Using Simplicity to Control Complexity , 2001, IEEE Softw..

[45]  John Rushby Trustworthy Self-Integrating Systems , 2016, ICDCIT.

[46]  R. W. Witty,et al.  Safe programming , 1978 .

[47]  Sasikumar Punnekkat,et al.  Improving Dependability of Vision-Based Advanced Driver Assistance Systems Using Navigation Data and Checkpoint Recognition , 2015, SAFECOMP.

[48]  Lorenzo Strigini,et al.  Software Fault-Freeness and Reliability Predictions , 2013, SAFECOMP.

[49]  John M. Rushby,et al.  New challenges in certification for aircraft software , 2011, 2011 Proceedings of the Ninth ACM International Conference on Embedded Software (EMSOFT).

[50]  Mario Trapp,et al.  Safety Assurance of Open Adaptive Systems - A Survey , 2011, Models@run.time@Dagstuhl.

[51]  David Wright,et al.  The Use of Multilegged Arguments to Increase Confidence in Safety Claims for Software-Based Systems: A Study Based on a BBN Analysis of an Idealized Example , 2007, IEEE Transactions on Software Engineering.

[52]  Edmund L. Gettier Is Justified True Belief Knowledge? , 1963, Arguing About Knowledge.

[53]  John Rushby,et al.  On the Interpretation of Assurance Case Arguments , 2015, JSAI-isAI Workshops.

[54]  Xiaowei Huang,et al.  Reasoning about Cognitive Trust in Stochastic Multiagent Systems , 2017, AAAI.

[55]  Hiroyuki Kido,et al.  A Supplemental Notation of GSN Aiming for Dealing with Changes of Assurance Cases , 2014, 2014 IEEE International Symposium on Software Reliability Engineering Workshops.

[56]  Michael Jackson,et al.  Software requirements & specifications , 1995 .

[57]  L. G. Neuberg,et al.  Bayes or Bust?-A Critical Examination of Bayesian Confirmation Theory. , 1994 .

[58]  Ashish Tiwari,et al.  Safety envelope for security , 2014, HiCoNS.