Prioritizing the creation of unit tests in legacy software systems

Test‐driven development (TDD) is a software development practice that prescribes writing unit tests before writing implementation code. Recent studies have shown that TDD practices can significantly reduce the number of pre‐release defects. However, most TDD research thus far has focused on new development. We investigate the adaptation of TDD‐like practices for already‐implemented code, in particular legacy systems. We call such an adaptation ‘Test‐driven maintenance’ (TDM). In this paper, we present a TDM approach that assists software development and testing managers to use the limited resources they have for testing legacy systems efficiently. The approach leverages the development history of a project to generate a prioritized list of functions that managers should focus their unit test writing resources on. The list is updated dynamically as the development of the legacy system progresses. We evaluate our approach on two large software systems: a large commercial system and the Eclipse Open Source Software system. For both systems, our findings suggest that heuristics based on the function size, modification frequency and bug fixing frequency should be used to prioritize the unit test writing efforts for legacy systems. Copyright © 2011 John Wiley & Sons, Ltd.

[1]  Per Runeson,et al.  A survey of unit testing practices , 2006, IEEE Software.

[2]  Philomena Jurey.  Fair and Balanced , 2005 .

[3]  Bing Wu,et al.  Legacy Information Systems: Issues and Directions , 1999, IEEE Softw..

[4]  A. Zeller,et al.  Predicting Defects for Eclipse , 2007, Third International Workshop on Predictor Models in Software Engineering (PROMISE'07: ICSE Workshops 2007).

[5]  Elaine J. Weyuker,et al.  Predicting the location and number of faults in large software systems , 2005, IEEE Transactions on Software Engineering.

[6]  Rainer Koschke,et al.  Revisiting the evaluation of defect prediction models , 2009, PROMISE '09.

[7]  Laurie A. Williams,et al.  Test-driven development as a defect-reduction practice , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[8]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[9]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[10]  Elliot Soloway,et al.  Where the bugs are , 1985, CHI '85.

[11]  Raj Jain,et al.  The art of computer systems performance analysis - techniques for experimental design, measurement, simulation, and modeling , 1991, Wiley professional computing.

[12]  Nachiappan Nagappan,et al.  Using Software Dependencies and Churn Metrics to Predict Field Failures: An Empirical Case Study , 2007, First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007).

[13]  Lionel C. Briand,et al.  Data Mining Techniques for Building Fault-proneness Models in Telecom Java Software , 2007, The 18th IEEE International Symposium on Software Reliability (ISSRE '07).

[14]  Walter F. Tichy,et al.  Rcs — a system for version control , 1985, Softw. Pract. Exp..

[15]  Premkumar T. Devanbu,et al.  Fair and balanced?: bias in bug-fix datasets , 2009, ESEC/FSE '09.

[16]  Gregg Rothermel,et al.  Empirical studies of test case prioritization in a JUnit testing environment , 2004, 15th International Symposium on Software Reliability Engineering.

[17]  K. K. Aggarwal,et al.  Code coverage based technique for prioritizing test cases for regression testing , 2004, SOEN.

[18]  Ahmed E. Hassan,et al.  Prioritizing Unit Test Creation for Test-Driven Maintenance of Legacy Systems , 2010, 2010 10th International Conference on Quality Software.

[19]  Ahmed E. Hassan,et al.  A Case Study of Bias in Bug-Fix Datasets , 2010, 2010 17th Working Conference on Reverse Engineering.

[20]  Ahmed E. Hassan,et al.  Predicting faults using the complexity of code changes , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[21]  Hausi A. Müller,et al.  Understanding software systems using reverse engineering technology perspectives from the Rigi project , 1993, CASCON.

[22]  Richard C. Holt,et al.  The top ten list: dynamic fault prediction , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[23]  Marc J. Rochkind,et al.  The source code control system , 1975, IEEE Transactions on Software Engineering.

[24]  N. Nagappan,et al.  Use of relative code churn measures to predict system defect density , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[25]  Hausi A. Müller,et al.  Understanding Software Systems Using Reverse Engineering Technology , 1994, COODBSE.

[26]  L. Erlikh,et al.  Leveraging legacy system dollars for e-business , 2000 .

[27]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[28]  Martin Fowler,et al.  Planning Extreme Programming , 2000 .

[29]  Andreas Zeller,et al.  Predicting faults from cached history , 2008, ISEC '08.

[30]  Harvey P. Siy,et al.  Predicting Fault Incidence Using Software Change History , 2000, IEEE Trans. Software Eng..

[31]  Boby George,et al.  A structured experiment of test-driven development , 2004, Inf. Softw. Technol..

[32]  Elaine J. Weyuker,et al.  Looking for bugs in all the right places , 2006, ISSTA '06.

[33]  Boby George,et al.  An initial investigation of test driven development in industry , 2003, SAC '03.

[34]  Tze-Jie Yu,et al.  An Analysis of Several Software Defect Models , 1988, IEEE Trans. Software Eng..

[35]  Walter F. Tichy,et al.  Case study: extreme programming in a university environment , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[36]  Andreas Zeller,et al.  Predicting defects in SAP Java code: An experience report , 2009, 2009 31st International Conference on Software Engineering - Companion Volume.

[37]  Jesús M. González-Barahona,et al.  Towards a Theoretical Model for Software Growth , 2007, Fourth International Workshop on Mining Software Repositories (MSR'07:ICSE Workshops 2007).

[38]  Taghi M. Khoshgoftaar,et al.  Data Mining for Predictors of Software Quality , 1999, Int. J. Softw. Eng. Knowl. Eng..

[39]  Dewayne E. Perry,et al.  Classification and evaluation of defects in a project retrospective , 2002, J. Syst. Softw..

[40]  Keith H. Bennett,et al.  Legacy Systems: Coping with Success , 1995, IEEE Softw..

[41]  Laurie A. Williams,et al.  Realizing quality improvement through test driven development: results and experiences of four industrial teams , 2008, Empirical Software Engineering.

[42]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[43]  Andreas Zeller,et al.  Mining Version Histories to Guide Software Changes , 2004 .

[44]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[45]  Audris Mockus,et al.  Identifying reasons for software changes using historic databases , 2000, Proceedings 2000 International Conference on Software Maintenance.

[46]  U. Wagener,et al.  Maintaining a Competitive Edge , 1993 .

[47]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[48]  Marco Torchiano,et al.  On the effectiveness of the test-first approach to programming , 2005, IEEE Transactions on Software Engineering.

[49]  Lionel C. Briand,et al.  Predicting fault-prone components in a java legacy system , 2006, ISESE '06.

[50]  Gregg Rothermel,et al.  Selecting a Cost-Effective Test Case Prioritization Technique , 2004, Software Quality Journal.