Goal-Based Safety Standards: Opportunities and Challenges

Issue 3 of UK Defence Standard (DS) 00-56, published at the end of 2004, is goal-based and replaces an earlier issue of the standard which was much more prescriptive. In civil aerospace the standard for ground-based software (SW01) is also goal-based. This move towards goal-based standards is intended to give system suppliers flexibility in developing systems and in showing that they are safe, and hence to reduce the likelihood that standards become increasingly inappropriate as technology progresses. However this advantage, or opportunity, brings some attendant challenges. The challenges which arise include: • Agreeing on appropriate means, and levels of evidence, for demonstrating safety, especially with new technology; • Contracting for a safety program where the set of safety activities and required evidence may not be determined “up front”. The paper discusses the opportunities and challenges of goal-based safety standards and proposes an approach to managing safety programs which seeks to maximize the opportunities whilst containing the associated risks. The approach is essentially evolutionary, and based on evolving the safety case through the development process. Introduction According to the Oxford English Dictionary (OED) a standard is “an authoritative or recognized exemplar of correctness, perfection or some definite degree of any quality”, thus one would expect standards to be widely accepted and stable. This is broadly true of standards for, say, physical constants, but is not the case in the area of safety. Indeed there has been considerable evolution in safety standards over the last fifteen years. In the last five years a small number of goal-based standards have emerged. This paper investigates this trend and asks whether or not this might lead to establishing authoritative and stable standards. There are many varieties of standards; at minimum we can distinguish process and product standards. In safety, process standards are concerned with how systems are developed and assessed, e.g. methods of safety analysis and criteria for risk assessment; product standards are concerned with particular system attributes, e.g. levels of hardware redundancy or colors of displays. Our focus here is on process standards. Historically many safety process standards have been prescriptive (i.e. tell people what to do) and/or proscriptive (i.e. tell people what to avoid doing). In contrast, goal-based standards tell people what they need to achieve. At its simplest, in safety, this is to establish safety requirements, design to meet the agreed requirements, and to show that the agreed requirements have been met. Traditional standards are generally clear about what needs to be done, but can be difficult to apply in novel circumstances. Goal-based standards offer more flexibility, but raise a number of concerns. Most notably they introduce ambiguity about what is required to achieve compliance. The purpose of this paper is to analyze the challenges and opportunities posed by goal-based standards and to ask whether the adoption of such standards might help in a move towards the “authoritative or recognized exemplar of correctness” which the OED says characterizes a standard. The paper starts by considering why it is difficult to produce safety standards, then considers the evolution of a number of standards, including Defence Standard (DS) 00-56, before considering the challenges, together with responses to those challenges, and opportunities posed by the introduction of goal-based standards. Why are Safety Standards Difficult? There are large numbers of safety standards in use; Hermann [1] identifies many of the software safety standards which were in use at the end of the 1990s. There are many more standards which apply at the system level, or which focus on particular technologies, and there have been several new standards published since. There are many reasons for this diversity in standards; we focus here on issues of risk and technology. Acceptable Risk: Safety is not absolute; a system is deemed safe when the level of risk it poses is acceptable, or tolerable, in its intended context of use. The level of risk which is judged acceptable, or tolerable, depends on many factors including: • Who is put at risk, e.g. adults versus children, or factory workers versus the general public; • The “dreadness” of the risk, e.g. nuclear hazards are generally less acceptable than those due to more “everyday” causes such as road traffic, even where the “objective” level of risk is the same or lower; • Whether or not the risk is taken on voluntarily, e.g. people accept higher risks hang-gliding than they might with domestic goods; • With military systems, whether the risks arise in war, peace, or in operations other than war (OOTW). Thus, determining whether a system is “safe enough” is highly context dependent. For military systems this is exacerbated by the fact that many different classes of individual may be at risk, and it is necessary to deal with a range of operational circumstances including trials and OOTW. For a narrow range of application with little variability in terms of who is put at risk, it may be possible to come up with a clear set of risk acceptance rules. Arguably civil aerospace falls into this class; certainly guidance documents such as DO178B [2] and ARP 4761 [3] are clear about acceptable rates of occurrence for different classes of hazards. In contrast, military standards have to deal with many more situations and tools such as Hazard Risk Indices (HRI), or risk matrices, give only some of the necessary flexibility. Standards such as DS 00-56 Issue 2 say that the HRI table is an example, but it is rare for it to be modified for a particular system – in part due to the lack of guidance on how to adapt the matrices for a given context of use of a system. A safety process standard cannot define acceptable levels of risk if it is to be general (to meet the OED definition). However, it can define the goal of risk acceptance – obtaining agreement between the relevant parties (typically the system supplier and customer, or certification authority). This has sub-goals of agreeing risk levels and agreeing when enough evidence had been produced that the risks were at or below those levels. Technology: Technology changes; in many modern systems “technology refresh” not only occurs during the system life, but it may occur prior to the system entering into service. Standards also change, but generally, they change relatively slowly – for example IEC 61508 [4] took more than ten years to produce, and, after six years of issue, it is already being updated. Prescriptive standards thus run the risk of always being “behind the curve” of technology and this can have several adverse effects. First, the standards may not properly regulate new technology, i.e. new technologies may be employed where there is insufficient evidence to show that they have been employed in a safe manner (with attendant risk of accidents). Second, beneficial new technologies might not be employed because there is no agreed way of assessing them, within the confines of the relevant standard. Third, unnecessary expense may be incurred because inefficient development and assessment techniques are used when there are better technologies – which the relevant standard proscribes, or does not accept. A safety process standard cannot define the technologies to be used, or avoided, if it is to be authoritative, hence stable (to meet the OED definition). However, it could define the goal of providing appropriate and sufficient evidence of safe use of the technology. This would have sub-goals of agreeing the types and combinations of evidence which were appropriate for a given technology, in its context of use. Other Issues: There are many other issues with current standards. One of the most problematic areas is Safety Integrity Levels (SILs), see for example Fowler [5] and Redmill [6] who indicate some of the difficulties in setting “sensible” SIL targets using the rules published in standards. There is a further problem not often remarked upon. In many standards there are four SILs (or DALs in DO178B); these link risk levels or HRI to development processes. The fact that there are four levels constrains the definition of risk levels, e.g. having five levels would be a problem, and thus reduces flexibility in defining and reducing risk levels. For generality, a standard should be silent on matters where there is genuinely no consensus. Thus we would not recommend having a goal for setting a SIL – but the rule of “being silent” includes not proscribing the use of SILs, so they may be employed where it is genuinely helpful to do so. Standards Evolution Over the past decade many safety process standards have been evolving, and continue to do so. For example, MilStd 882 has evolved from Issue C [7] in 1996 to Issue D now [8], with work under way on Issue E. Defence Standard (DS) 00-56 has evolved through three issues in fifteen years (see below for more detail), as have other associated standards In Civil Aerospace, DO178B (strictly a set of guidelines not a standard) has been stable for some years, however annual guidance bulletins have been produced and work is now under way on DO178C (through RTCA Special Committee 205 / EUROCAE Working Group 71). Similarly updates are being made to the safety process standards including ARP 4761 (through SAE S18 / EUROCAE Working Group 63). In a broader context, IEC 61508 is also being updated. All this change leads to a question – why is there such a volume of activity? Generally, there have been few changes in declared criteria for risk acceptance, and this does not seem to be a major driver for change. There have been technological changes and that is, for example, what is driving both DO178B and ARP 4761. However there is an additional factor; that of approaches to regulation. In the USA, the Perry initiative req