A Critical Analysis of Empirical Research in Software Testing

In the foreseeable future, software testing will remain one of the best tools we have at our disposal to ensure software dependability. Empirical studies are crucial to software testing research in order to compare and improve software testing techniques and practices. In fact, there is no other way to assess the cost-effectiveness of testing techniques, since all of them are, to various extents, based on heuristics and simplifying assumptions. However, when empirically studying the cost and fault- detection rates of a testing technique, a number of validity issues arise. Further, there are many ways in which empirical studies can be performed, ranging from simulations to controlled experiments with human subjects. What are the strengths and drawbacks of the various approaches? What is the best option under which circumstances? This paper presents a critical analysis of empirical research in software testing and will attempt to highlight and clarify the issues above in a structured and practical manner.

[1]  Keith Phalp,et al.  An investigation of machine learning based prediction systems , 2000, J. Syst. Softw..

[2]  Douglas Fisher,et al.  Machine Learning Approaches to Estimating Software Development Effort , 1995, IEEE Trans. Software Eng..

[3]  Building a software cost estimation model based on categorical data , 2001, Proceedings Seventh International Software Metrics Symposium.

[4]  Amrit L. Goel,et al.  Empirical Data Modeling in Software Engineering Using Radical Basis Functions , 2000, IEEE Trans. Software Eng..

[5]  Mohammad Alshayeb,et al.  An Empirical Validation of Object-Oriented Metrics in Two Different Iterative Software Processes , 2003, IEEE Trans. Software Eng..

[6]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[7]  Pekka Abrahamsson,et al.  Empirical Evaluation of Agile Software Development: The Controlled Case Study Approach , 2004, PROFES.

[8]  Lionel C. Briand,et al.  Automated, contract-based user testing of commercial-off-the-shelf components , 2006, ICSE '06.

[9]  Alberto Sillitti,et al.  Collecting, integrating and analyzing software metrics and personal software process data , 2003, 2003 Proceedings 29th Euromicro Conference.

[10]  Mohamed E. Fayad,et al.  Software Engineering in the Small , 2000 .

[11]  Mark Harman,et al.  The Current State and Future of Search Based Software Engineering , 2007, Future of Software Engineering (FOSE '07).

[12]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[13]  Massimiliano Di Penta,et al.  Assessing and improving state-based class testing: a series of experiments , 2004, IEEE Transactions on Software Engineering.

[14]  Barbara A. Kitchenham,et al.  A Simulation Study of the Model Evaluation Criterion MMRE , 2003, IEEE Trans. Software Eng..

[15]  Stephen G. MacDonell,et al.  Using prior-phase effort records for re-estimation during software projects , 2003, Proceedings. 5th International Workshop on Enterprise Networking and Computing in Healthcare Industry (IEEE Cat. No.03EX717).

[16]  R. Tibshirani,et al.  An introduction to the bootstrap , 1993 .

[17]  Aditya P. Mathur,et al.  Integration testing using interface mutation , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.

[18]  Lionel C. Briand,et al.  Using Machine Learning to Refine Black-Box Test Specifications and Test Suites , 2008, 2008 The Eighth International Conference on Quality Software.

[19]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[20]  Natalia Juristo Juzgado,et al.  Basics of Software Engineering Experimentation , 2010, Springer US.

[21]  Martin J. Shepperd,et al.  Estimating Software Project Effort Using Analogies , 1997, IEEE Trans. Software Eng..

[22]  Aamer Nadeem,et al.  A state-based approach to integration testing based on UML models , 2007, Inf. Softw. Technol..

[23]  Gregg Rothermel,et al.  An empirical study of regression testing techniques incorporating context and lifetime factors and improved cost-benefit models , 2006, SIGSOFT '06/FSE-14.

[24]  Jason Crampton,et al.  The interpretation and utility of three cohesion metrics for object-oriented design , 2006, TSEM.

[25]  Mary Jean Harrold,et al.  Empirical evaluation of the tarantula automatic fault-localization technique , 2005, ASE.

[26]  Tsun S. Chow,et al.  Testing Software Design Modeled by Finite-State Machines , 1978, IEEE Transactions on Software Engineering.

[27]  Lionel C. Briand,et al.  Assessing, Comparing, and Combining Statechart- based testing and Structural testing: An Experiment , 2007, ESEM 2007.

[28]  Hong Sun,et al.  Investigating the use of analysis contracts to improve the testability of object‐oriented code , 2003, Softw. Pract. Exp..

[29]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[30]  Maya Daneva.  Approaching the ERP Project Cost Estimation Problem: an Experiment , 2007, ESEM 2007.

[31]  Barry W. Boehm,et al.  Cost models for future software life cycle processes: COCOMO 2.0 , 1995, Ann. Softw. Eng..

[32]  Gary D. Boetticher,et al.  Using Machine Learning to Predict Project Effort: Empirical Case Studies in Data-Starved Domains , 2001 .

[33]  Ingunn Myrtveit,et al.  Reliability and validity in comparative studies of software prediction models , 2005, IEEE Transactions on Software Engineering.

[34]  Kenji Yokoyama,et al.  Development of a hybrid cost estimation model in an iterative manner , 2006, ICSE.

[35]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[36]  Y. Miyazaki,et al.  Robust regression for developing software estimation models , 1994, J. Syst. Softw..

[37]  Lionel C. Briand,et al.  Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria , 2006, IEEE Transactions on Software Engineering.

[38]  Lionel C. Briand,et al.  Using genetic algorithms for early schedulability analysis and stress testing in real-time systems , 2006, Genetic Programming and Evolvable Machines.

[39]  Aamer Nadeem,et al.  A state-based approach to integration testing for object-oriented programs , 2005 .

[40]  Lionel C. Briand,et al.  Using simulation to empirically investigate test coverage criteria based on statechart , 2004, Proceedings. 26th International Conference on Software Engineering.

[41]  Barry W. Boehm,et al.  Software development cost estimation approaches — A survey , 2000, Ann. Softw. Eng..

[42]  A. Jefferson Offutt,et al.  A fault model for subtype inheritance and polymorphism , 2001, Proceedings 12th International Symposium on Software Reliability Engineering.

[43]  Parag C. Pendharkar,et al.  A probabilistic model for predicting software development effort , 2003, IEEE Transactions on Software Engineering.

[44]  Siba N. Mohanty,et al.  Software cost estimation: Present and future , 1981, Softw. Pract. Exp..

[45]  Isabella Wieczorek,et al.  Resource Estimation in Software Engineering , 2002 .

[46]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[47]  Chris F. Kemerer,et al.  A Metrics Suite for Object Oriented Design , 2015, IEEE Trans. Software Eng..