Design for Resilience in Autonomous Systems: Lessons Learned from Controlled Flight into Stall Accidents

Resilience is the means to extend the design reliability of a system beyond the design assurance achieved by the technology. In the case of supervised autonomous systems, resilience can be achieved by the intervention of a human operator when the autonomous system creates an undesired state. This paper describes a detailed analysis of the requirements and the design for intervention in the operation of an autonomous function on a modern airliner. The analysis derives the requirements for the intervention task from an analysis of the six step scenario leading to the Controlled Flight into Stall (CFIS) accidents. Some of the requirements that cannot be performed adequately by the human operator (i.e. monitoring for rare events, complex calculations, and correlation of disparate data) are automated in a stand-alone device on the flight deck. The functional design of this intervention support device, known as the Paranoid Pilot Associate, is described. Limitations and implications of the design are discussed.