Intrusive Test Automation with Failed Test Case Clustering

Regression testing is an indispensable process in software development, which ensures stable features have not been adversely broken by new changes. When GUI plays an important role in an application, a popular choice to automate the tests is applying GUI Capture/Replay tools. Unfortunately, in many applications which render images as output, the correctness of a replay run can no longer be straightforwardly verified. In this paper, we propose a test automation solution, called Intrusive Test Automation, which uses program instrumentation technique to collect the runtime internal information of a program. As a result, the correctness of a test run can be verified by the runtime traces. In addition, when large number of failed test cases are reported by the test automation system, recommending some representative test cases as a start for debugging can be helpful to programmers. This paper proposes a clustering technique based on the information collected from the instrumented code. In principle, fixing bugs in one representative test case can fix its related failed test cases as well. A case study is presented to demonstrate the effectiveness of the approach.

[1]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[2]  W. Masri,et al.  An empirical evaluation of test case filtering techniques based on exercising complex information flows , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[3]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[4]  John Steven,et al.  jRapture: A Capture/Replay tool for observation-based testing , 2000, ISSTA '00.

[5]  Christus,et al.  A General Method Applicable to the Search for Similarities in the Amino Acid Sequence of Two Proteins , 2022 .

[6]  Andrea Clematis,et al.  An approach to application software fault tolerance , 1990, EW 4.

[7]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[8]  Hrushikesha Mohanty,et al.  Clustering test cases to achieve effective test selection , 2010, A2CWiC '10.

[9]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[10]  Mary Jean Harrold,et al.  An empirical study of the effects of test-suite reduction on fault localization , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[11]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[12]  Mary Jean Harrold,et al.  Empirical evaluation of the tarantula automatic fault-localization technique , 2005, ASE.

[13]  Baowen Xu,et al.  An Improved Regression Test Selection Technique by Clustering Execution Profiles , 2010, 2010 10th International Conference on Quality Software.