Evaluating test‐to‐code traceability recovery methods through controlled experiments

Recently, different methods and tools have been proposed to automate or semi‐automate test‐to‐code traceability recovery. Among these, Slicing and Coupling based Test to Code trace Hunter (SCOTCH) exploits slicing and conceptual coupling to identify the classes tested by a JUnit test. However, until now the evaluation of test‐to‐code traceability recovery methods has been limited to experiments assessing their tracing accuracy rather than the actual support these methods provide to a software engineer during traceability recovery tasks. Indeed, a research method or tool has a better chance of being transferred to practitioners if it is supported by empirical evidence. In this paper, we present the results of two controlled experiments carried out to evaluate the support given by SCOTCH during traceability recovery, when compared with other traceability recovery methods. The results show that SCOTCH is able to suggest a higher number of correct links with higher accuracy, thus sensibly improving the performances of software engineers during test‐to‐code traceability recovery tasks. Copyright © 2012 John Wiley & Sons, Ltd.

[1]  Harald C. Gall,et al.  CVS release history data for detecting logical couplings , 2003, Sixth International Workshop on Principles of Software Evolution, 2003. Proceedings..

[2]  Diane K. Michelson,et al.  Applied Statistics for Engineers and Scientists , 2001, Technometrics.

[3]  Giuliano Antoniol,et al.  Recovering Traceability Links between Code and Documentation , 2002, IEEE Trans. Software Eng..

[4]  Tibor Gyimóthy,et al.  Using information retrieval based coupling measures for impact analysis , 2009, Empirical Software Engineering.

[5]  David J. Groggel,et al.  Practical Nonparametric Statistics , 2000, Technometrics.

[6]  Richard N. Taylor,et al.  Software traceability with topic modeling , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[7]  Arie van Deursen,et al.  Refactoring test code , 2001 .

[8]  Alexander Egyed,et al.  Automating requirements traceability: Beyond the record & replay paradigm , 2002, Proceedings 17th IEEE International Conference on Automated Software Engineering,.

[9]  Andrian Marcus,et al.  Recovering documentation-to-source-code traceability links using latent semantic indexing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[10]  Genny Tortora,et al.  IR-Based Traceability Recovery Processes: An Empirical Comparison of "One-Shot" and Incremental Processes , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[11]  Alexander Egyed,et al.  A scenario-driven approach to traceability , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[12]  Alexander Egyed,et al.  A Scenario-Driven Approach to Trace Dependency Analysis , 2003, IEEE Trans. Software Eng..

[13]  Genny Tortora,et al.  ADAMS: an Artefact-based Process Support System , 2004, SEKE.

[14]  Gabriele Bavota,et al.  SCOTCH: Test-to-code traceability using slicing and conceptual coupling , 2011, 2011 27th IEEE International Conference on Software Maintenance (ICSM).

[15]  Andrea De Lucia,et al.  Recovering traceability links between unit tests and classes under test: An improved method , 2010, 2010 IEEE International Conference on Software Maintenance.

[16]  Fabrizio Sebastiani,et al.  Machine learning in automated text categorization , 2001, CSUR.

[17]  Leon Moonen,et al.  Assessing the value of coding standards: An empirical study , 2008, 2008 IEEE International Conference on Software Maintenance.

[18]  Winifred Menezes,et al.  Marketing Technology to Software Practitioners , 2000, IEEE Softw..

[19]  SIWUW A. Raghvan.  Diffusing Software-Engineering Methods , 2001 .

[20]  Janusz W. Laski,et al.  Dynamic Program Slicing , 1988, Inf. Process. Lett..

[21]  Harald C. Gall,et al.  Detection of logical coupling based on product release history , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[22]  Andrea Zisman.  Using Rules for Traceability Creation , 2012, Software and Systems Traceability.

[23]  Gabriele Bavota,et al.  An empirical analysis of the distribution of unit test smells and their impact on software maintenance , 2012, 2012 28th IEEE International Conference on Software Maintenance (ICSM).

[24]  Arie van Deursen,et al.  The Video Store Revisited – Thoughts on Refactoring and Testing , 2002 .

[25]  Jens Krinke,et al.  EzUnit: A Framework for Associating Failed Unit Tests with Potential Programming Errors , 2007, XP.

[26]  Ingolf Krüger,et al.  Tracing requirements to tests with high precision and recall , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[27]  Genny Tortora,et al.  Assessing IR-based traceability recovery tools through controlled experiments , 2009, Empirical Software Engineering.

[28]  Genny Tortora,et al.  Recovering traceability links in software artifact management systems using information retrieval methods , 2007, TSEM.

[29]  William E. Riddle,et al.  Software technology maturation , 1985, ICSE '85.

[30]  Carl K. Chang,et al.  Event-Based Traceability for Managing Evolutionary Change , 2003, IEEE Trans. Software Eng..

[31]  Janusz W. Laski,et al.  Dynamic slicing of computer programs , 1990, J. Syst. Softw..

[32]  Marco Torchiano,et al.  How Developers' Experience and Ability Influence Web Application Comprehension Tasks Supported by UML Stereotypes: A Series of Four Experiments , 2010, IEEE Transactions on Software Engineering.

[33]  Joseph Robert Horgan,et al.  Dynamic program slicing , 1990, PLDI '90.

[34]  Marvin V. Zelkowitz,et al.  Experimental Validation of New Software Technology , 2003, Lecture Notes on Empirical Software Engineering.

[35]  Giuliano Antoniol,et al.  Tracing object-oriented code into functional requirements , 2000, Proceedings IWPC 2000. 8th International Workshop on Program Comprehension.

[36]  B. Kitchenham,et al.  Case Studies for Method and Tool Evaluation , 1995, IEEE Softw..

[37]  Kent Beck,et al.  Test-infected: programmers love writing tests , 2000 .

[38]  Harry M. Sneed.  Reverse engineering of test cases for selective regression testing , 2004, Eighth European Conference on Software Maintenance and Reengineering, 2004. CSMR 2004. Proceedings..

[39]  Peter M. Chisnall,et al.  Questionnaire Design, Interviewing and Attitude Measurement , 1993 .

[40]  Richard A. Harshman,et al.  Indexing by Latent Semantic Analysis , 1990, J. Am. Soc. Inf. Sci..

[41]  Oussama Ben Khadra,et al.  Goal-centric traceability for managing non-functional requirements , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[42]  Serge Demeyer,et al.  Establishing Traceability Links between Unit Test Cases and Units under Test , 2009, 2009 13th European Conference on Software Maintenance and Reengineering.

[43]  David S. Janzen,et al.  Towards traceable test-driven development , 2009, 2009 ICSE Workshop on Traceability in Emerging Forms of Software Engineering.

[44]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[45]  Andrea De Lucia,et al.  Information Retrieval Methods for Automated Traceability Recovery , 2012, Software and Systems Traceability.

[46]  Barbara G. Ryder,et al.  Constructing the Call Graph of a Program , 1979, IEEE Transactions on Software Engineering.