Automation of System Safety Analysis : Possibilities and Pitfalls

As systems become more complex it is progressively less tractable to carry out hazard and safety analyses by hand. Both hazard identification/analysis and confirmatory safety analyses, e.g. FMEA and FTA, present significant (although distinct) problems, and the difficulties are perhaps most intense for software, as this is often where much of the complexity is found. To counter these problems there is growing demand for automation of safety analyses. Automation presents many possible benefits, but also some drawbacks. It must also be acknowledged that humans are better at some activities, e.g. reasoning inductively, than computers. The practical issue is how to gain benefit from automation whilst reducing risks which could arise from over-reliance on tools. This paper concentrates on the automation of analyses during system design and development, although many of the observations and conclusions are equally applicable to tools used to manage and analyse safety for in-service systems. The paper discusses some potential risks of, and challenges for, highly automated safety analysis, particularly the effects of removing human skill and experience from the analysis process. It considers: how to design tools to identify design weaknesses, and/or produce derived safety requirements (and knowing when they are more or less effective than humans); how to facilitate human involvement in automated analyses, recognising that improved understanding of a system is an important output of the safety process. Underlying this is the question of where it is possible, and appropriate, to “take the human out of the loop”.