Assurance cases and prescriptive software safety certification: A comparative study

In safety–critical applications, it is necessary to justify, prior to deployment, why software behaviour is to be trusted. This is normally referred to as software safety assurance. Within certification standards, developers demonstrate this by appealing to the satisfaction of objectives that the safety assurance standards require for compliance. In some standards the objectives can be very detailed in nature, prescribing specific processes and techniques that must be followed. This approach to certification is often described as prescriptive or process-based certification. Other standards set out much more high-level objectives and are less prescriptive about the particular processes and techniques to be used. These standards instead explicitly require the submission of an assurance argument which communicates how evidence, generated during development (for example from testing, analysis and review) satisfies claims concerning the safety of the software. There has been much debate surrounding the relative merits of prescriptive and safety assurance argument approaches to certification. In many ways this debate can lead to confusion. There can in fact be seen to be a role for both approaches in a successful software assurance regime. In this paper, we provide a comparative examination of these two approaches, and seek to identify the relative merits of each. We first introduce the concepts of assurance cases and prescriptive software assurance. We describe how an assurance case could be generated for the software of an aircraft wheel braking system. We then describe how prescriptive certification guidelines could be used in order to gain assurance in the same system. Finally, we compare the results of the two approaches and explain how these approaches may complement each other. This comparison highlights the crucial role that an assurance argument can play in explaining and justifying how the software evidence supports the assurance argument, even when a prescriptive safety standard is being followed.

[1]  Holloway C. Michael,et al.  Streamlining Software Aspects of Certification: Technical Team Report on the First Industry Workshop , 1998 .

[2]  Chris W. Johnson,et al.  The Dangers of Failure Masking in Fault-Tolerant Software: Aspects of a Recent In-Flight Upset Event , 2007 .

[3]  Robyn R. Lutz,et al.  Operational anomalies as a cause of safety-critical requirements evolution , 2003, J. Syst. Softw..

[4]  Ann Miller,et al.  Assurance Cases for Security: The Metrics Challenge , 2007, 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07).

[5]  Geoff Vaughan,et al.  Review of the HSE's safety assessment principles for nuclear facilities. , 2007, Journal of radiological protection : official journal of the Society for Radiological Protection.

[6]  Ibrahim Habli,et al.  A Generic Goal-Based Certification Argument for the Justification of Formal Analysis , 2009, SafeCert@ETAPS.

[7]  Peter G. Bishop,et al.  The Practicalities of Goal-Based Safety Regulation , 2001 .

[8]  John Goodenough,et al.  Arguing Security – Creating Security Assurance Cases , 2014 .

[9]  Tim Kelly,et al.  Arguing Safety - A Systematic Approach to Managing Safety Cases , 1998 .

[10]  T. Kelly,et al.  Concepts and Principles of Compositional Safety Case Construction , 2002 .

[11]  John A. McDermid,et al.  Software Safety: Where's the Evidence? , 2001, SCS.

[12]  Jang-Soo Lee,et al.  Means-ends and whole-part traceability analysis of safety requirements , 2010, J. Syst. Softw..

[13]  G. B. Finelli,et al.  The infeasibility of experimental quantification of life-critical software reliability , 1991, SIGSOFT '91.

[14]  Tim Kelly,et al.  A Model-Driven Approach to Assuring Process Reliability , 2008, 2008 19th International Symposium on Software Reliability Engineering (ISSRE).

[15]  Martyn Thomas,et al.  Software For Dependable Systems , 2008 .

[16]  Richard Hawkins,et al.  Software safety: relating software assurance and software integrity , 2010, Int. J. Crit. Comput. Based Syst..

[17]  Felix Redmill,et al.  System Safety: HAZOP and Software HAZOP , 1999 .

[18]  Nurlida Basir,et al.  Deriving Safety Cases for Hierarchical Structure in Model-Based Development , 2010, SAFECOMP.

[19]  Tim Kelly,et al.  The Who, Where, How, Why And When of Modular and Incremental Certification , 2007 .

[20]  T. Kelly,et al.  A Systematic Approach for Developing Software Safety Arguments , 2009 .

[21]  J. Goodenough,et al.  Towards an Assurance Case Practice for Medical Devices , 2009 .

[22]  Ricky W. Butler,et al.  The infeasibility of experimental quantification of life-critical software reliability , 1991 .

[23]  James Inge Defence Standard 00-56 Issue 4: Safety Management Requirements for Defence Systems , 2007 .

[24]  Peter G. Bishop,et al.  Safety and Assurance Cases: Past, Present and Possible Future - an Adelard Perspective , 2010, SSS.

[25]  J Hayhurst Kelly,et al.  A Practical Tutorial on Modified Condition/Decision Coverage , 2001 .

[26]  John M. Rushby,et al.  Formalism in Safety Cases , 2010, SSS.

[27]  J Hayhurst Kelly,et al.  Streamlining Software Aspects of Certification: Report on the SSAC Survey , 1999 .

[28]  Nurlida Basir,et al.  Constructing a Safety Case for Automatically Generated Code from Formal Program Verification Information , 2008, SAFECOMP.

[29]  J. McDermid,et al.  Software Safety: Why is there no Consensus? , 2002 .

[30]  John C. Knight,et al.  Assurance based development , 2010 .

[31]  Kester Clegg,et al.  Using a Software Safety Argument Pattern Catalogue: Two Case Studies , 2011, SAFECOMP.

[32]  T. Kelly Reviewing Assurance Arguments – A Step-By-Step Approach , 2007 .

[33]  John C. Knight,et al.  A Taxonomy of Fallacies in System Safety Arguments , 2006 .

[34]  John C. Knight,et al.  Assurance Based Development of Critical Systems , 2007, 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07).

[35]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[36]  Robert Andrew Weaver,et al.  The Safety of Software - Constructing and Assuring Arguments , 2003 .

[37]  T. S. E. Maibaum,et al.  Software Certification: Is There a Case against Safety Cases? , 2010, Monterey Workshop.

[38]  Tim Kelly,et al.  Achieving Integrated Process and Product Safety Arguments , 2007, SSS.

[39]  R. Yin Case Study Research: Design and Methods , 1984 .

[40]  Fan Ye,et al.  Justifying the use of COTS components within safety critical applications , 2005 .

[41]  Ewen Denney,et al.  Perspectives on software safety case development for unmanned aircraft , 2012, IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2012).

[42]  Richard Hawkins,et al.  Arguing Conformance , 2012, IEEE Software.

[43]  David Wright,et al.  The Use of Multilegged Arguments to Increase Confidence in Safety Claims for Software-Based Systems: A Study Based on a BBN Analysis of an Idealized Example , 2007, IEEE Transactions on Software Engineering.