Achievements and Challenges in State-of-the-Art Software Traceability Between Test and Code Artifacts

Testing is a key activity of software development and maintenance that determines the level of reliability. Traceability is the ability to describe and follow the life of software artifacts, and has been promoted as a means for supporting various activities, most importantly testing. Traceability information facilitates the testing and debugging of complex software by modeling the dependencies between code and tests. Actively supplementing traceability to testing enables rectifying defects more reliably and efficiently. Despite its importance, the development of test-to-code traceability has not been sufficiently addressed in the literature, and even worse there is currently no organized review of traceability studies in this field. In this work, we have investigated the main conferences, workshops, and journals of the requirements engineering, testing, and reliability, and identified those contributions that refer to traceability topics. From that starting point, we characterized and analyzed the chosen contributions against three research questions by utilizing a comparative framework including nine criteria. As a result, our study arrives to some interesting points, and outlines a number of potential research directions. This, in turn, can pave the way for facilitating and empowering traceability research in this domain to assist software engineers and testers in test management.

[1]  Ladan Tahvildari,et al.  Reconstructing Traceability between Bugs and Test Cases: An Experimental Study , 2011, 2011 18th Working Conference on Reverse Engineering.

[2]  Michael R. Lyu,et al.  Framework for modeling software reliability, using various testing-efforts and fault-detection rates , 2001, IEEE Trans. Reliab..

[3]  Gabriele Bavota,et al.  Evaluating test‐to‐code traceability recovery methods through controlled experiments , 2013, J. Softw. Evol. Process..

[4]  Andrea De Lucia,et al.  Recovering traceability links between unit tests and classes under test: An improved method , 2010, 2010 IEEE International Conference on Software Maintenance.

[5]  Jens Krinke,et al.  EzUnit: A Framework for Associating Failed Unit Tests with Potential Programming Errors , 2007, XP.

[6]  Arda Goknil,et al.  Survey of Traceability Approaches in Model-Driven Engineering , 2007, 11th IEEE International Enterprise Distributed Object Computing Conference (EDOC 2007).

[7]  Dr. Hossein Saiedian,et al.  The Importance of Traceability Why Software Requirements Traceability Remains a Challenge , .

[8]  Gerard Meszaros,et al.  xUnit Test Patterns: Refactoring Test Code , 2007 .

[9]  Stephen Clark,et al.  Best Practices for Automated Traceability , 2007, Computer.

[10]  Abdallah Qusef Recovering Test-to-Code Traceability Via Slicing and Conceptual Coupling , 2011, 2011 18th Working Conference on Reverse Engineering.

[11]  Juan Garbajosa,et al.  A study to support agile methods more effectively through traceability , 2011, Innovations in Systems and Software Engineering.

[12]  Mordechai Nisenson,et al.  A Traceability Technique for Specifications , 2008, 2008 16th IEEE International Conference on Program Comprehension.

[13]  Harvey S. Koch,et al.  Managing Test-Procedures to Achieve Reliable Software , 1983, IEEE Transactions on Reliability.

[14]  Mark Neal,et al.  Why and how of requirements tracing , 1994, IEEE Software.

[15]  Maria-Eugenia Iacob,et al.  Requirements Traceability and Transformation Conformance in Model-Driven Development , 2006, 2006 10th IEEE International Enterprise Distributed Object Computing Conference (EDOC'06).

[16]  James M. Bieman,et al.  Software reliability growth with test coverage , 2002, IEEE Trans. Reliab..

[17]  Yonggang Zhang,et al.  Text mining and software engineering: an integrated source code and document analysis approach , 2008, IET Softw..

[18]  Jens von Pilgrim,et al.  A survey of traceability in requirements engineering and model-driven development , 2010, Software & Systems Modeling.

[19]  Andrea De Lucia,et al.  Traceability Recovery Using Numerical Analysis , 2009, 2009 16th Working Conference on Reverse Engineering.

[20]  Richard N. Taylor,et al.  Software traceability with topic modeling , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[21]  Andrea Zisman,et al.  Rule-based generation of requirements traceability relations , 2004, J. Syst. Softw..

[22]  Arie van Deursen,et al.  Refactoring test code , 2001 .

[23]  John C. Grundy,et al.  Improving automated documentation to code traceability by combining retrieval techniques , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[24]  Saeed Maghsoodloo,et al.  A reliability and cost analysis of an automatic prototype generator test paradigm , 1992 .

[25]  Serge Demeyer,et al.  Establishing Traceability Links between Unit Test Cases and Units under Test , 2009, 2009 13th European Conference on Software Maintenance and Reengineering.

[26]  Gabriele Bavota,et al.  SCOTCH: Test-to-code traceability using slicing and conceptual coupling , 2011, 2011 27th IEEE International Conference on Software Maintenance (ICSM).

[27]  Xin Yao,et al.  Multi-Objective Approaches to Optimal Testing Resource Allocation in Modular Software Systems , 2010, IEEE Transactions on Reliability.

[28]  Naftali Tishby,et al.  Sufficient Dimensionality Reduction , 2003, J. Mach. Learn. Res..

[29]  Richard Stevens,et al.  Requirements Traceability , 1997, RE.

[30]  Tom Mens,et al.  Towards a taxonomy of software change , 2005, J. Softw. Maintenance Res. Pract..

[31]  Alexander Egyed,et al.  A Scenario-Driven Approach to Trace Dependency Analysis , 2003, IEEE Trans. Software Eng..

[32]  Jane Cleland-Huang,et al.  Utilizing supporting evidence to improve dynamic requirements traceability , 2005, 13th IEEE International Conference on Requirements Engineering (RE'05).

[33]  Kishor S. Trivedi,et al.  Accelerated Degradation Tests Applied to Software Aging Experiments , 2010, IEEE Transactions on Reliability.

[34]  Gerard Salton,et al.  A vector space model for automatic indexing , 1975, CACM.

[35]  Per Runeson,et al.  Evaluation of traceability recovery in context: A taxonomy for information retrieval tools , 2012, EASE.

[36]  Giuliano Antoniol,et al.  Recovering Traceability Links between Code and Documentation , 2002, IEEE Trans. Software Eng..

[37]  T. Landauer,et al.  Indexing by Latent Semantic Analysis , 1990 .

[38]  Andrea Zisman,et al.  XTraQue: traceability for product line systems , 2009, Software & Systems Modeling.

[39]  Antje von Knethen,et al.  Change-Oriented Requirements Traceability: Support for Evolution of Embedded Systems , 2002, ICSM.

[40]  Genny Tortora,et al.  Assessing IR-based traceability recovery tools through controlled experiments , 2009, Empirical Software Engineering.

[41]  Thomas Hofmann,et al.  Probabilistic latent semantic indexing , 1999, SIGIR '99.

[42]  Genny Tortora,et al.  Recovering traceability links in software artifact management systems using information retrieval methods , 2007, TSEM.

[43]  Chao Liu,et al.  Recovering Relationships between Documentation and Source Code based on the Characteristics of Software Engineering , 2009, Electron. Notes Theor. Comput. Sci..

[44]  Soo Dong Kim,et al.  Traceability-centric model-driven object-oriented engineering , 2010, Inf. Softw. Technol..

[45]  Andrian Marcus,et al.  Recovering documentation-to-source-code traceability links using latent semantic indexing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[46]  Tony Gorschek,et al.  Requirements Traceability: a Systematic Review and Industry Case Study , 2012, Int. J. Softw. Eng. Knowl. Eng..

[47]  Cristina V. Lopes,et al.  Aspect-oriented programming , 1999, ECOOP Workshops.

[48]  Kent Beck,et al.  Test-infected: programmers love writing tests , 2000 .

[49]  Michael R. Lyu,et al.  Optimal testing resource allocation, and sensitivity analysis in software development , 2005, IEEE Transactions on Reliability.

[50]  Harry M. Sneed Reverse engineering of test cases for selective regression testing , 2004, Eighth European Conference on Software Maintenance and Reengineering, 2004. CSMR 2004. Proceedings..

[51]  Arie van Deursen,et al.  Can LSI help reconstructing requirements traceability in design and test? , 2006, Conference on Software Maintenance and Reengineering (CSMR'06).

[52]  Domenico Cotroneo,et al.  Combining Operational and Debug Testing for Improving Reliability , 2013, IEEE Transactions on Reliability.

[53]  Babu Zachariah Analysis of Software Testing Strategies Through Attained Failure Size , 2012, IEEE Transactions on Reliability.

[54]  R. N. Rattihalli,et al.  Failure Size Proportional Models and an Analysis of Failure Detection Abilities of Software Testing Strategies , 2007, IEEE Transactions on Reliability.

[55]  Klaus Pohl,et al.  Adapting traceability environments to project-specific needs , 1998, CACM.

[56]  Henry Muccini,et al.  A scoped approach to traceability management , 2009, J. Syst. Softw..

[57]  Chin-Yu Huang,et al.  Analysis of incorporating logistic testing-effort function into software reliability modeling , 2002, IEEE Trans. Reliab..

[58]  Arbi Ghazarian A Research Agenda for Software Reliability , 2009 .

[59]  Ingolf Krüger,et al.  Feature Location Using Data Mining on Existing Test-Cases , 2012, 2012 19th Working Conference on Reverse Engineering.

[60]  Olly Gotel,et al.  An analysis of the requirements traceability problem , 1994, Proceedings of IEEE International Conference on Requirements Engineering.

[61]  Barbara G. Ryder,et al.  Constructing the Call Graph of a Program , 1979, IEEE Transactions on Software Engineering.

[62]  Letha H. Etzkorn,et al.  Recovering traceability links between source code and fixed bugs via patch analysis , 2011, TEFSE '11.

[63]  D. M. Hutton,et al.  Software Test Automation: Effective Use of Test Execution Tools , 2000 .

[64]  Wan M. N. Wan-Kadir,et al.  An Evaluation of Traceability Approaches to Support Software Evolution , 2007, International Conference on Software Engineering Advances (ICSEA 2007).

[65]  Julia Rubin,et al.  Model traceability , 2006, IBM Syst. J..

[66]  Don Batory,et al.  Scaling step-wise refinement , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[67]  Romain Robbes,et al.  Linking e-mails and source code artifacts , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[68]  Vahid Garousi,et al.  Tool Support for Automated Traceability of Test/Code Artifacts in Embedded Software Systems , 2011, 2011IEEE 10th International Conference on Trust, Security and Privacy in Computing and Communications.

[69]  Shigeru Yamada,et al.  Optimal allocation and control problems for software-testing resources , 1990 .

[70]  Janusz W. Laski,et al.  Dynamic slicing of computer programs , 1990, J. Syst. Softw..

[71]  Jane Huffman Hayes,et al.  Assessing traceability of software engineering artifacts , 2010, Requirements Engineering.

[72]  Gabriele Bavota,et al.  Recovering test-to-code traceability using slicing and textual analysis , 2014, J. Syst. Softw..

[73]  Tony Gorschek,et al.  A method for evaluating rigor and industrial relevance of technology evaluations , 2011, Empirical Software Engineering.