A field study on root cause analysis of defects in space software

Abstract Critical systems, such as space systems, are developed under strict requirements envisaging high integrity in accordance to specific standards. For such software systems, an independent assessment is put into effect (Independent Software Verification and Validation – ISVV) after the regular development lifecycle and V&V activities, aiming at finding residual faults and raising confidence in the software. However, it has been observed that there is still a significant number of defects remaining at this stage, questioning the effectiveness of the previous engineering processes. This paper presents a root cause analysis of 1070 defects found in four space software projects during ISVV, by applying an improved Orthogonal Defect Classification (ODC) taxonomy and examining the defect types, triggers and impacts, in order to identify why they reached such a later stage in the development. The paper also puts forward proposals for modifications to both the software development (to prevent defects) and the V&V activities (to better detect defects) and an assessment methodology for future works on root cause analysis.

[1]  Fanny Andalia,et al.  Implementation of Analytical Hierarchy Process On Airplane Ticket Booking Application Selection With Software Quality Requirements and Evaluation ISO / IEC 25010 : 2011 , 2018 .

[2]  Nuno Silva,et al.  Towards Making Safety-Critical Systems Safer: Learning from Mistakes , 2014, 2014 IEEE International Symposium on Software Reliability Engineering Workshops.

[3]  Robyn R. Lutz,et al.  Operational anomalies as a cause of safety-critical requirements evolution , 2003, J. Syst. Softw..

[4]  Robyn R. Lutz,et al.  Analyzing software requirements errors in safety-critical, embedded systems , 1993, [1993] Proceedings of the IEEE International Symposium on Requirements Engineering.

[5]  Mustafa Degerli,et al.  Yazilim Muhendisligi ve Sistem Muhendisligi Sureclerinin Harmanlanmasi: ISO/IEC 12207: 2008'in Getirdikleri ve Degisikliklerin Etkilerine Yonelik Degerlendirmeler (Blending Software Engineering and Systems Engineering Processes: Changes Introduced by ISO/IEC 12207: 2008 and Evaluations on Effects of , 2017, UYMS.

[6]  Dewayne E. Perry,et al.  A case study in root cause defect analysis , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[7]  Nancy G. Leveson,et al.  An analysis of causation in aerospace accidents , 2001, 20th DASC. 20th Digital Avionics Systems Conference (Cat. No.01CH37219).

[8]  Nuno Silva,et al.  Assessment of Defect Type influence in Complex and Integrated Space Systems: Analysis Based on ODC and ISVV Issues , 2015, 2015 IEEE International Conference on Dependable Systems and Networks Workshops.

[9]  Ramachandran Baskaran,et al.  Defect Analysis and Prevention for Software Process Quality Improvement , 2010 .

[10]  Steve Easterbrook.  The Role of Independent V&V in Upstream Software Development Processes , 1996 .

[11]  Nuno Silva,et al.  Independent Test Verification: What Metrics Have a Word to Say , 2011, 2011 First International Workshop on Software Certification.

[12]  Claes Wohlin,et al.  Faults-slip-through - a concept for measuring the efficiency of the test process , 2006, Softw. Process. Improv. Pract..

[13]  Forrest Shull,et al.  Defect categorization: making use of a decade of widely varying historical data , 2008, ESEM '08.

[14]  Nuno Silva,et al.  10 Years of ISVV: What's Next? , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering Workshops.

[15]  Nuno Silva,et al.  Independent Assessment of Safety-Critical Systems: We Bring Data! , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering Workshops.

[16]  Dewayne E. Perry,et al.  Classification and evaluation of defects in a project retrospective , 2002, J. Syst. Softw..

[17]  Robyn R. Lutz,et al.  Empirical analysis of safety-critical anomalies during operations , 2004, IEEE Transactions on Software Engineering.

[18]  Nuno Silva,et al.  Software for embedded systems: a quality assessment based on improved ODC taxonomy , 2016, SAC.

[19]  Robert B. Grady,et al.  Practical Software Metrics for Project Management and Process Improvement , 1992 .

[20]  Raul Moreira Vidal,et al.  Classification of defect types in requirements specifications: Literature review, proposal and assessment , 2011, 6th Iberian Conference on Information Systems and Technologies (CISTI 2011).

[21]  Nuno Silva,et al.  Overview of 10 Years of ISVV Findings in Safety-Critical Systems , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering Workshops.

[22]  Nuno Silva,et al.  Results from an Independent View on The Validation of Safety-Critical Space Systems , 2013 .