A Comprehensive Survey of Recent Developments in Software Testing Methodologies

Testing is one of the phases of all process models of software engineering. It remains the most indispensable part of software quality assurance. High reliability of software is expected in the real world as it, otherwise, becomes obsolete. It is more so with complex and machine critical applications. In this paper we provide a comprehensive survey of recent developments in software testing methodologies. Various approaches discussed in this paper include automatic generation of test cases, search based techniques, Just-InTime quality assurance, static analysis, bad smell detection, early detection of concurrency problems, random testing, integration testing, combinatorial testing, model-based testing, test-driven approaches, dependency-based test case prioritization, state-based testing, adaptive testing and so on. This paper throws light into dependency structures for test case prioritization and test suite generation with minimum size test suites maximum coverage with discussion on empirical studies. The recent methodologies in software testing are focused in this paper besides finding potential gaps for future work.

[1]  Günter Rudolph,et al.  Convergence analysis of canonical genetic algorithms , 1994, IEEE Trans. Neural Networks.

[2]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[3]  Alessandro Orso.  Integration Testing of Object-Oriented Software , 1998 .

[4]  Martin Glinz,et al.  Using Dependency Charts to Improve Scenario-Based Testing , 2000 .

[5]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[6]  Prioritizing test cases for regression testing , 2000, ISSTA.

[7]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[8]  Eda Marchetti,et al.  The Cow_Suite Approach to Planning and Deriving Test Suites in UML Projects , 2002, UML.

[9]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[10]  Yannis Smaragdakis,et al.  JCrasher: an automatic robustness tester for Java , 2004, Softw. Pract. Exp..

[11]  Paolo Tonella,et al.  Evolutionary testing of classes , 2004, ISSTA '04.

[12]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[13]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[14]  J. Jenny Li,et al.  Prioritize code for testing to improve code coverage of complex software , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[15]  Laurie A. Williams,et al.  System test case prioritization of new and regression test cases , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[16]  Jean-Marc Jézéquel,et al.  Automatic test case optimization: a bacteriologic algorithm , 2005, IEEE Software.

[17]  Paolo Tonella,et al.  Using the Case-Based Ranking Methodology for Test Case Prioritization , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[18]  Myra B. Cohen,et al.  Combinatorial Interaction Regression Testing: A Study of Test Case Generation and Prioritization , 2007, 2007 IEEE International Conference on Software Maintenance.

[19]  Bertrand Meyer,et al.  Automatic Testing of Object-Oriented Software , 2007, SOFSEM.

[20]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[21]  Tao Xie,et al.  Improving Structural Testing of Object-Oriented Programs via Integrating Evolutionary Testing and Symbolic Execution , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[22]  Jianjun Zhao,et al.  Test Case Prioritization Based on Analysis of Program Structure , 2008, 2008 15th Asia-Pacific Software Engineering Conference.

[23]  Neelam Gupta,et al.  Experiments with test case prioritization using relevant slices , 2008, J. Syst. Softw..

[24]  T. H. Tse,et al.  Adaptive Random Test Case Prioritization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[25]  Rajib Mall,et al.  System testing for object‐oriented systems with test case prioritization , 2009, Softw. Test. Verification Reliab..

[26]  Kiran Lakhotia,et al.  Search{Based Testing , 2009 .

[27]  Kishor S. Trivedi,et al.  Software Reliability and Testing Time Allocation: An Architecture-Based Approach , 2010, IEEE Transactions on Software Engineering.

[28]  Tim Menzies,et al.  Genetic Algorithms for Randomized Unit Testing , 2011, IEEE Transactions on Software Engineering.

[29]  Paolo Tonella,et al.  Symbolic search-based testing , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[30]  Gordon Fraser,et al.  Combining search-based and constraint-based testing , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[31]  Ravishankar K. Iyer,et al.  Automated Derivation of Application-Aware Error Detectors Using Static Analysis: The Trusted Illiac Approach , 2011, IEEE Transactions on Dependable and Secure Computing.

[32]  Lionel C. Briand,et al.  Random Testing: Theoretical Results and Practical Implications , 2012, IEEE Transactions on Software Engineering.

[33]  Arie van Deursen,et al.  Invariant-Based Automatic Testing of Modern Web Applications , 2012, IEEE Transactions on Software Engineering.

[34]  Claire Le Goues,et al.  GenProg: A Generic Method for Automatic Software Repair , 2012, IEEE Transactions on Software Engineering.

[35]  Andrea Arcuri,et al.  A Theoretical and Empirical Analysis of the Role of Test Sequence Length in Software Testing for Structural Coverage , 2012, IEEE Transactions on Software Engineering.

[36]  Andreas Zeller,et al.  Mutation-Driven Generation of Unit Tests and Oracles , 2010, IEEE Transactions on Software Engineering.

[37]  Jay F. Nunamaker,et al.  Comparing the Defect Reduction Benefits of Code Inspection and Test-Driven Development , 2012, IEEE Transactions on Software Engineering.

[38]  Lionel C. Briand,et al.  A UML/MARTE Model Analysis Method for Uncovering Scenarios Leading to Starvation and Deadlocks in Concurrent Systems , 2012, IEEE Transactions on Software Engineering.

[39]  Zhendong Niu,et al.  Schedule of Bad Smell Detection and Resolution: A New Way to Save Effort , 2012, IEEE Transactions on Software Engineering.

[40]  Frank Tip,et al.  A Comprehensive Approach to Naming and Accessibility in Refactoring Java Programs , 2012, IEEE Transactions on Software Engineering.

[41]  Audris Mockus,et al.  A large-scale empirical study of just-in-time quality assurance , 2013, IEEE Transactions on Software Engineering.

[42]  Kai-Yuan Cai,et al.  Enhancing software reliability estimates using modified adaptive testing , 2013, Inf. Softw. Technol..

[43]  Myra B. Cohen,et al.  An orchestrated survey of methodologies for automated software test case generation , 2013, J. Syst. Softw..

[44]  Cemal Yilmaz.  Test Case-Aware Combinatorial Interaction Testing , 2013, IEEE Transactions on Software Engineering.

[45]  Gordon Fraser,et al.  Whole Test Suite Generation , 2013, IEEE Transactions on Software Engineering.

[46]  Ioannis Stamelos,et al.  Layer assessment of object-oriented software: A metric facilitating white-box reuse , 2013, J. Syst. Softw..

[47]  Vojislav B. Misic,et al.  The Effects of Test-Driven Development on External Quality and Productivity: A Meta-Analysis , 2013, IEEE Transactions on Software Engineering.

[48]  Ibrahim Habli,et al.  An Empirical Evaluation of Mutation Testing for Improving the Test Quality of Safety-Critical Software , 2013, IEEE Transactions on Software Engineering.

[49]  Tim Miller,et al.  Using Dependency Structures for Prioritization of Functional Test Suites , 2013, IEEE Transactions on Software Engineering.

[50]  Daniela Cruzes,et al.  A study of cyclic dependencies on defect profile of software components , 2013, J. Syst. Softw..

[51]  Márcio Eduardo Delamaro,et al.  A scoping study on the 25 years of research into software testing in Brazil and an outlook on the future of the area , 2013, J. Syst. Softw..

[52]  Malte Lochau,et al.  Delta-oriented model-based integration testing of large-scale systems , 2014, J. Syst. Softw..

[53]  Richard Torkar,et al.  Empirical evaluations on the cost-effectiveness of state-based testing: An industrial case study , 2014, Inf. Softw. Technol..

[54]  P. Govindarajulu,et al.  Automatic Discovery of Dependency Structures for Test Case Prioritization , 2015 .