Preventing Defects: The Impact of Requirements Traceability Completeness on Software Quality

Requirements traceability has long been recognized as an important quality of a well-engineered system. Among stakeholders, traceability is often unpopular due to the unclear benefits. In fact, little evidence exists regarding the expected traceability benefits. There is a need for empirical work that studies the effect of traceability. In this paper, we focus on the four main requirements implementation supporting activities that utilize traceability. For each activity, we propose generalized traceability completeness measures. In a defined process, we selected 24 medium to large-scale open-source projects. For each software project, we quantified the degree to which a studied development activity was enabled by existing traceability with the proposed measures. We analyzed that data in a multi-level Poisson regression analysis. We found that the degree of traceability completeness for three of the studied activities significantly affects software quality, which we quantified as defect rate. Our results provide for the first time empirical evidence that more complete traceability decreases the expected defect rate in the developed software. The strong impact of traceability completeness on the defect rate suggests that traceability is of great practical value for any kind of software development project, even if traceability is not mandated by a standard or regulation.

[1]  Jae-on Kim,et al.  Standardization in Causal Analysis , 1981 .

[2]  David A. Belsley,et al.  Regression Analysis and its Application: A Data-Oriented Approach.@@@Applied Linear Regression.@@@Regression Diagnostics: Identifying Influential Data and Sources of Collinearity , 1981 .

[3]  Lyle E. Bourne,et al.  Text-Based Decisions: Changes in the Availability of Facts Due to Instructions and the Passage of Time. , 1982 .

[4]  Shawn A. Bohner,et al.  A framework for software maintenance metrics , 1990, Proceedings. Conference on Software Maintenance 1990.

[5]  Olly Gotel,et al.  An analysis of the requirements traceability problem , 1994, Proceedings of IEEE International Conference on Requirements Engineering.

[6]  Balasubramaniam Ramesh,et al.  Implementing requirements traceability: a case study , 1995, Proceedings of 1995 IEEE International Symposium on Requirements Engineering (RE'95).

[7]  Harry M. Sneed Estimating the costs of software maintenance tasks , 1995, Proceedings of International Conference on Software Maintenance.

[8]  Dar-Biau Liu,et al.  Metrics for requirements engineering , 1995, J. Syst. Softw..

[9]  Shawn A. Bohner,et al.  Impact analysis in the software change process: a year 2000 perspective , 1996, 1996 Proceedings of International Conference on Software Maintenance.

[10]  Klaus Pohl,et al.  PRO-ART: enabling requirements pre-traceability , 1996, Proceedings of the Second International Conference on Requirements Engineering.

[11]  Mikael Lindvall,et al.  Practical implications of traceability , 1996 .

[12]  Shari Lawrence Pfleeger,et al.  Software Quality: The Elusive Target , 1996, IEEE Softw..

[13]  Richard Stevens,et al.  Requirements Traceability , 1997, RE.

[14]  Klaus Pohl,et al.  Adapting traceability environments to project-specific needs , 1998, CACM.

[15]  Tim Kelly,et al.  Arguing Safety - A Systematic Approach to Managing Safety Cases , 1998 .

[16]  Balasubramaniam Ramesh,et al.  Factors influencing requirements traceability practice , 1998, CACM.

[17]  Michael Jackson,et al.  A Reference Model for Requirements and Specifications , 2000, IEEE Softw..

[18]  Michael Jackson,et al.  A reference model for requirements and specifications - extended abstract , 2000, Proceedings Fourth International Conference on Requirements Engineering. ICRE 2000. (Cat. No.98TB100219).

[19]  Keith H. Bennett,et al.  A Staged Model for the Software Life Cycle , 2000, Computer.

[20]  Harvey P. Siy,et al.  Predicting Fault Incidence Using Software Change History , 2000, IEEE Trans. Software Eng..

[21]  Lionel C. Briand,et al.  Exploring the relationships between design measures and software quality in object-oriented systems , 2000, J. Syst. Softw..

[22]  Hoyt Lougee,et al.  SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICATION , 2001 .

[23]  Barbara Paech,et al.  Systematic requirements recycling through abstraction and traceability , 2002, Proceedings IEEE Joint International Conference on Requirements Engineering.

[24]  J. Dick Rich Traceability , 2002 .

[25]  Ramanath Subramanyam,et al.  Empirical Analysis of CK Metrics for Object-Oriented Design Complexity: Implications for Software Defects , 2003, IEEE Trans. Software Eng..

[26]  Hyunsoo Kim,et al.  The software maintenance project effort estimation model based on function points , 2003, J. Softw. Maintenance Res. Pract..

[27]  Carl K. Chang,et al.  Event-Based Traceability for Managing Evolutionary Change , 2003, IEEE Trans. Software Eng..

[28]  Andrian Marcus,et al.  Recovering documentation-to-source-code traceability links using latent semantic indexing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[29]  Jane Cleland-Huang,et al.  A heterogeneous solution for improving the return on investment of requirements traceability , 2004, Proceedings. 12th IEEE International Requirements Engineering Conference, 2004..

[30]  B. Flyvbjerg Five Misunderstandings About Case-Study Research , 2006, 1304.1186.

[31]  Steve Riddle,et al.  Overcoming the traceability benefit problem , 2005, 13th IEEE International Conference on Requirements Engineering (RE'05).

[32]  Stefan Biffl,et al.  A case study on value-based requirements tracing , 2005, ESEC/FSE-13.

[33]  James D. Herbsleb,et al.  Global software development at siemens: experience from nine projects , 2005, ICSE.

[34]  Andreas Zeller,et al.  Mining metrics to predict component failures , 2006, ICSE.

[35]  Stephen Clark,et al.  Best Practices for Automated Traceability , 2007, Computer.

[36]  Michael J. Crawley,et al.  The R book , 2022 .

[37]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[38]  Victor R. Basili,et al.  The influence of organizational structure on software quality , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[39]  A. De Lucia,et al.  Traceability management for impact analysis , 2008, 2008 Frontiers of Software Maintenance.

[40]  Ilka Philippow,et al.  Rule-Based Maintenance of Post-Requirements Traceability Relations , 2008, 2008 16th IEEE International Requirements Engineering Conference.

[41]  Ilka Philippow,et al.  Getting back to basics: Promoting the use of a traceability information model in practice , 2009, 2009 ICSE Workshop on Traceability in Emerging Forms of Software Engineering.

[42]  Ilka Philippow,et al.  Motivation Matters in the Traceability Trenches , 2009, 2009 17th IEEE International Requirements Engineering Conference.

[43]  Klaus Pohl,et al.  Requirements Engineering - Fundamentals, Principles, and Techniques , 2010 .

[44]  Nan Niu,et al.  Using Semantics-Enabled Information Retrieval in Requirements Tracing: An Ongoing Experimental Investigation , 2010, 2010 IEEE 34th Annual Computer Software and Applications Conference.

[45]  Robert A. Greevy Data Analysis and Graphics Using R: An Example-Based Approach , 2010 .

[46]  John H. Maindonald,et al.  Data Analysis and Graphics Using R: An Example-Based Approach , 2010 .

[47]  Mehrdad Sabetzadeh,et al.  Characterizing the Chain of Evidence for Software Safety Cases: A Conceptual Model Based on the IEC 61508 Standard , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[48]  Yann-Gaël Guéhéneuc,et al.  Design evolution metrics for defect prediction in object oriented systems , 2010, Empirical Software Engineering.

[49]  Mehrdad Sabetzadeh,et al.  SafeSlice: a model slicing and design safety inspection tool for SysML , 2011, ESEC/FSE '11.

[50]  Jeremy Dick,et al.  Requirements Engineering, Third Edition , 2011 .

[51]  Andrea De Lucia,et al.  Improving Source Code Lexicon via Traceability and Information Retrieval , 2011, IEEE Transactions on Software Engineering.

[52]  Nan Niu,et al.  TraCter: A tool for candidate traceability link clustering , 2011, 2011 IEEE 19th International Requirements Engineering Conference.

[53]  Mehrdad Sabetzadeh,et al.  A Model-Driven Engineering Approach to Support the Verification of Compliance to Safety Standards , 2011, 2011 IEEE 22nd International Symposium on Software Reliability Engineering.

[54]  James D. Herbsleb,et al.  Factors leading to integration failures in global feature-oriented development: an empirical analysis , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[55]  Anthony Finkelstein,et al.  Early failure prediction in feature request management systems: an extended study , 2011, Requirements Engineering.

[56]  Andrea De Lucia,et al.  On integrating orthogonal information retrieval methods to improve traceability recovery , 2011, 2011 27th IEEE International Conference on Software Maintenance (ICSM).

[57]  Giuliano Antoniol,et al.  Traceability Fundamentals , 2012, Software and Systems Traceability.

[58]  Michael J. Crawley The R Book: Crawley/The R Book , 2012 .

[59]  Mehrdad Sabetzadeh,et al.  A SysML-based approach to traceability management and design slicing in support of safety certification: Framework, tool support, and case studies , 2012, Inf. Softw. Technol..

[60]  Charles R. Severance The Apache Software Foundation: Brian Behlendorf , 2012, Computer.

[61]  Evan Moritz,et al.  TraceLab: An experimental workbench for equipping researchers to innovate, synthesize, and comparatively evaluate traceability solutions , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[62]  Claire Ingram,et al.  Cost-Benefits of Traceability , 2012, Software and Systems Traceability.

[63]  Gilbert Regan,et al.  The Barriers to Traceability and their Potential Solutions: Towards a Reference Framework , 2012, 2012 38th Euromicro Conference on Software Engineering and Advanced Applications.

[64]  Alexander Egyed,et al.  Assessing the effect of requirements traceability for software maintenance , 2012, 2012 28th IEEE International Conference on Software Maintenance (ICSM).

[65]  Ilka Philippow,et al.  Requirements Traceability across Organizational Boundaries - A Survey and Taxonomy , 2013, REFSQ.

[66]  Yi Zhang,et al.  Strategic Traceability for Safety-Critical Projects , 2013, IEEE Software.

[67]  Bogdan Dit,et al.  Enhancing Software Traceability by Automatically Expanding Corpora with Relevant Documentation , 2013, 2013 IEEE International Conference on Software Maintenance.

[68]  Andrea De Lucia,et al.  Configuring topic models for software engineering tasks in TraceLab , 2013, 2013 7th International Workshop on Traceability in Emerging Forms of Software Engineering (TEFSE).

[69]  Chang Liu,et al.  A Study on the Effect of Traceability Links in Software Maintenance , 2013, IEEE Access.

[70]  Collin McMillan,et al.  When and How Using Structural Information to Improve IR-Based Traceability Recovery , 2013, 2013 17th European Conference on Software Maintenance and Reengineering.

[71]  Yann-Gaël Guéhéneuc,et al.  Trustrace: Mining Software Repositories to Improve the Accuracy of Requirement Traceability Links , 2013, IEEE Transactions on Software Engineering.

[72]  Patrick Mäder,et al.  An empirical study on project-specific traceability strategies , 2013, 2013 21st IEEE International Requirements Engineering Conference (RE).

[73]  Patrick Mäder,et al.  Towards feature-aware retrieval of refinement traces , 2013, 2013 7th International Workshop on Traceability in Emerging Forms of Software Engineering (TEFSE).

[74]  Alexander Egyed,et al.  Recovering traceability between features and code in product variants , 2013, SPLC '13.

[75]  Leanna Rierson,et al.  Developing Safety-Critical Software: A Practical Guide for Aviation Software and DO-178C Compliance , 2013 .

[76]  Ilka Philippow,et al.  A Survey on Usage Scenarios for Requirements Traceability in Practice , 2013, REFSQ.

[77]  Patrick Mäder,et al.  Software traceability: trends and future directions , 2014, FOSE.

[78]  Patrick Mäder,et al.  Mind the gap: assessing the conformance of software traceability to relevant guidelines , 2014, ICSE.

[79]  Zhendong Niu,et al.  Traceability-enabled refactoring for managing just-in-time requirements , 2014, 2014 IEEE 22nd International Requirements Engineering Conference (RE).

[80]  Gilbert Regan,et al.  A Traceability Process Assessment Model for the Medical Device Domain , 2014, EuroSPI.

[81]  Alexander Egyed,et al.  Do developers benefit from requirements traceability when evolving and maintaining a software system? , 2014, Empirical Software Engineering.

[82]  Alexander Egyed,et al.  Using Traceability for Incremental Construction and Evolution of Software Product Portfolios , 2015, 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability.

[83]  Patrick Mäder,et al.  Estimating the Implementation Risk of Requirements in Agile Software Development Projects with Traceability Metrics , 2015, REFSQ.

[84]  LiGuo Huang,et al.  Can method data dependencies support the assessment of traceability between requirements and source code? , 2015, J. Softw. Evol. Process..

[85]  Patrick Mäder,et al.  A quality model for the systematic assessment of requirements traceability , 2015, 2015 IEEE 23rd International Requirements Engineering Conference (RE).

[86]  Barbara Paech,et al.  Do Information Retrieval Algorithms for Automated Traceability Perform Effectively on Issue Tracking System Data? , 2016, REFSQ.

[87]  Alejandra Ruiz,et al.  Model-based specification of safety compliance needs for critical systems: A holistic generic metamodel , 2016, Inf. Softw. Technol..

[88]  Björn Regnell,et al.  How Firms Adapt and Interact in Open Source Ecosystems: Analyzing Stakeholder Influence and Collaboration Patterns , 2016, REFSQ.