Experience Report: Orthogonal Classification of Safety Critical Issues

Techniques to classify defects have been used for decades, providing relevant information on how to improve systems. Such techniques heavily rely on human experience and have been generalized to cover different types of systems at different maturity levels. However, their application to safety-critical systems development and operation phases neither is very common, or at least not spread publicly, nor disseminated in the industrial and academic worlds. This practical experience report presents the results and conclusions from applying a mature Orthogonal Defect Classification (ODC) to a large set of safety-critical issues. The work is based on the analysis of more than 240 real issues (defects) identified during all the lifecycle phases of 4 safety-critical systems in the aerospace and space domains. The outcomes reveal the challenges in properly classifying this specific type of issues with the broader ODC approach. The difficulties are identified and systematized and specific proposals for improvement are proposed.

[1]  Nuno Silva,et al.  Independent Test Verification: What Metrics Have a Word to Say , 2011, 2011 First International Workshop on Software Certification.

[2]  Dewayne E. Perry,et al.  A case study in root cause defect analysis , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[3]  Harlan D. Mills,et al.  Engineering software under statistical quality control , 1990, IEEE Software.

[4]  Stefan Wagner.  A model and sensitivity analysis of the quality economics of defect-detection techniques , 2006, ISSTA '06.

[5]  Fernanda Grazioli,et al.  A framework to evaluate defect taxonomies , 2009 .

[6]  Robyn R. Lutz,et al.  Empirical analysis of safety-critical anomalies during operations , 2004, IEEE Transactions on Software Engineering.

[7]  Stefan Wagner,et al.  A literature survey of the quality economics of defect-detection techniques , 2006, ISESE '06.

[8]  Michael Jones,et al.  Software engineering: are we getting better at it? , 2005 .

[9]  Nuno Silva,et al.  Independent Assessment of Safety-Critical Systems: We Bring Data! , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering Workshops.

[10]  Dewayne E. Perry,et al.  Classification and evaluation of defects in a project retrospective , 2002, J. Syst. Softw..

[11]  Christian Denger,et al.  An industrial case study of implementing and validating defect classification for process improvement and quality management , 2005, 11th IEEE International Software Metrics Symposium (METRICS'05).

[12]  Nuno Silva,et al.  10 Years of ISVV: What's Next? , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering Workshops.

[13]  Nuno Silva,et al.  Overview of 10 Years of ISVV Findings in Safety-Critical Systems , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering Workshops.

[14]  Raul Moreira Vidal,et al.  Classification of defect types in requirements specifications: Literature review, proposal and assessment , 2011, 6th Iberian Conference on Information Systems and Technologies (CISTI 2011).

[15]  Robyn R. Lutz,et al.  Analyzing software requirements errors in safety-critical, embedded systems , 1993, [1993] Proceedings of the IEEE International Symposium on Requirements Engineering.

[16]  Robyn R. Lutz,et al.  Operational anomalies as a cause of safety-critical requirements evolution , 2003, J. Syst. Softw..

[17]  Forrest Shull,et al.  Defect categorization: making use of a decade of widely varying historical data , 2008, ESEM '08.

[18]  Robyn R. Lutz,et al.  Requirements discovery during the testing of safety-critical software , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[19]  Michael Keating.  The Simple Art of SoC Design , 2011 .

[20]  Nancy G. Leveson,et al.  An analysis of causation in aerospace accidents , 2001, 20th DASC. 20th Digital Avionics Systems Conference (Cat. No.01CH37219).

[21]  Hoyt Lougee,et al.  SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICATION , 2001 .

[22]  Inderpal S. Bhandari,et al.  Orthogonal Defect Classification - A Concept for In-Process Measurements , 1992, IEEE Trans. Software Eng..

[23]  Stefan Wagner,et al.  Defect classification and defect types revisited , 2008, DEFECTS '08.