The ethics of safety-critical systems

TheEthicsofSafety-CriticalSystemsJonathanP.BowenTheUniversityofReading,DepartmentComputerScienceWhiteknights,POBox225,Reading,BerksRG66AY,UKEmail:J.P.Bowen@reading.ac.ukURL:http://www.cs.reading.ac.uk/people/jpb/May1997AbstractSafety-criticalsystemsrequiretheutmostcareintheirsp eci cationanddesigntoavoiderrorsintheirimplementation,usingstateofthearttechniquesresp on-siblemanner.Todootherwiseisatb estunprofessionalandworstcanleadtodisastrousconsequences.Aninappropriateapproachcouldleadtolossoflife,andwillalmostcertainlyresultin nancialp enaltiesthelongrun,whetherb ecauseoflossofbusinessorb ecausetheimp osition nes.Legislationandstandardsim-p oseexternalpressures,buteducationandethicalconsiderationsshouldhelpprovidemoreself-imp osedguidelinesforallthoseinvolvedthepro ductionofsafety-criticalsystems.Thispap erconsiderssomeoftheissuesinvolved,withp ointerstomate-rialprovidinggreaterdepthinparticularareas,esp eciallywithresp ecttotheuseofformalmetho ds.1Prologue( oo o&)Platoisdeartome,buterstil ltruth.{Aristotle(384{322B.C.)Theuseofcomputersinsafety-criticalsystemsisincreasingrapidly.Computersareprob-ablyusedinatleastanorderofmagnitudemoresafety-criticalapplicationscomparedtoadecadeago.Thedecisionab outtheuseofsoftwareisoftentakenoneconomicratherthansafetygrounds.Forexample,theemploymentofcomputersiny-by-wireaircraftcanmakesigni cantfuelcostsavingssincethecomputerb eprogrammedtoyoptimalroutewithlittlepilotinterventionifallgo eswell.Thesafetyimplicationsaremorediculttoassess.Someinindustryclaimthatin-creaseduseofsoftwareincreasesthesafetysystem[31].Hoever,sinceitissodiculttomeasuresoftwarereliability,thisisjustifyinpractice.Infact,thenumb erofjet-airlinercrashescontinuestorisealarmingly[16].

[1]  W. Wayt Gibbs,et al.  Software's Chronic Crisis , 1994 .

[2]  Jonathan P. Bowen,et al.  High-Integrity System Specification and Design , 1999 .

[3]  Tony Hoare.  Programming is an Engineering Profession , 1983 .

[4]  Jonathan P. Bowen.  Formal methods in safety-critical standards , 1993, Proceedings 1993 Software Engineering Standards Symposium.

[5]  Peter G. Neumann,et al.  Computer-related risks , 1994 .

[6]  Nancy G. Leveson,et al.  An investigation of the Therac-25 accidents , 1993, Computer.

[7]  Jonathan P. Bowen,et al.  Safety-critical systems, formal methods and standards , 1993, Softw. Eng. J..

[8]  Paolo Crivelli,et al.  VII* THE ARGUMENT FROM KNOWING AND NOT KNOWING IN PLATO'S THEAETETUS (187E5-188C8) , 1996 .

[9]  Donald MacKenzie,et al.  The automation of proof: a historical and sociological exploration , 1995, IEEE Ann. Hist. Comput..

[10]  J. L. Lions.  ARIANE 5 Flight 501 Failure: Report by the Enquiry Board , 1996 .

[11]  J. Davenport.  Editor , 1960 .

[12]  Keith W. Miller,et al.  Software engineering code of ethics is approved , 1999, CACM.

[13]  Norman Mooradian,et al.  A Gift of Fire: Social, Legal, and Ethical Issues in Computing , 1998 .

[14]  Luca Cardelli,et al.  The Computer Science and Engineering Handbook , 1997 .

[15]  Michael Thomas.  Formal methods and their role in developing safe systems , 1996 .

[16]  Jonathan P. Bowen,et al.  Ten Commandments of Formal Methods , 1995, Computer.

[17]  Van Doren,et al.  A history of knowledge : past, present, and future , 1991 .

[18]  Jonathan P. Bowen,et al.  Formal Models and the Specification Process , 1997, The Computer Science and Engineering Handbook.

[19]  Ernst-Rüdiger Olderog,et al.  A ProCoS II Project Final Report: ESPRIT Basic Research project 7071 , 1996 .

[20]  Jim Woodcock,et al.  FME '93: Industrial-Strength Formal Methods , 1993, Lecture Notes in Computer Science.

[21]  Jonathan P. Bowen,et al.  ZUM '97: The Z Formal Specification Notation , 1997, Lecture Notes in Computer Science.

[22]  Keith W. Miller,et al.  Ethical Issues for Computer Scientists and Engineers , 1997, The Computer Science and Engineering Handbook.

[23]  R. Spinello.  Case studies in information and computer ethics , 1996 .

[24]  Anthony Hall,et al.  Taking Z Seriously , 1997, ZUM.

[25]  Anthony Hall,et al.  Seven myths of formal methods , 1990, IEEE Software.

[26]  Solomon Schimmel,et al.  Seven Deadly Sins , 1992 .

[27]  Jonathan P. Bowen,et al.  Seven More Myths of Formal Methods , 1995, IEEE Softw..

[28]  Jonathan P. Bowen,et al.  The Industrial Take-up of Formal Methods in Safety-Critical and Other Areas: A Perspective , 1993, FME.

[29]  J. P. Potocki de Montalk.  Computer software in civil aircraft , 1993, Microprocess. Microsystems.

[30]  John A. McDermid,et al.  Software Engineer's Reference Book , 1993 .

[31]  Jean-Raymond Abrial,et al.  The B-book - assigning programs to meanings , 1996 .

[32]  Peter Neumann,et al.  Safeware: System Safety and Computers , 1995, SOEN.

[33]  Carl Sagan,et al.  The demon-haunted world. Science as a candle in the dark , 1995 .