Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software

Software is increasingly being developed/maintained by multiple, often geographically distributed developers working concurrently. Consequently, rapid-feedback-based quality assurance mechanisms such as daily builds and smoke regression tests, which help to detect and eliminate defects early during software development and maintenance, have become important. This paper addresses a major weakness of current smoke regression testing techniques, i.e., their inability to automatically (re)test graphical user interfaces (GUIs). Several contributions are made to the area of GUI smoke testing. First, the requirements for GUI smoke testing are identified and a GUI smoke test is formally defined as a specialized sequence of events. Second, a GUI smoke regression testing process called daily automated regression tester (DART) that automates GUI smoke testing is presented. Third, the interplay between several characteristics of GUI smoke test suites including their size, fault detection ability, and test oracles is empirically studied. The results show that: 1) the entire smoke testing process is feasible in terms of execution time, storage space, and manual effort, 2) smoke tests cannot cover certain parts of the application code, 3) having comprehensive test oracles may make up for not having long smoke test cases, and 4) using certain oracles can make up for not having large smoke test suites.

[1]  Paul R. Ritter,et al.  Experience in testing the Motif interface , 1991, IEEE Software.

[2]  Atif M. Memon,et al.  GUI ripping: reverse engineering of graphical user interfaces for testing , 2003, 10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings..

[3]  Daniel A. Menascé,et al.  Load Testing of Web Sites , 2002, IEEE Internet Comput..

[4]  Atif M. Memon,et al.  Automating regression testing for evolving GUI software , 2005, J. Softw. Maintenance Res. Pract..

[5]  Malte Finsterwalder.  Automating Acceptance Tests for GUI Applications in an Extreme Programming Environment , 2001 .

[6]  Lee J. White,et al.  User-based testing of GUI sequences and their interactions , 2001, Proceedings 12th International Symposium on Software Reliability Engineering.

[7]  Mary Lou Soffa,et al.  Hierarchical GUI Test Case Generation Using Automated Planning , 2001, IEEE Trans. Software Eng..

[8]  John Robbins,et al.  DeBugging Applications with Cdrom , 2000 .

[9]  Heinz Züllighoven,et al.  The Need for Speed : Automating Acceptance Testing in an eXtreme Programming Environment – , 2002 .

[10]  Jim Weirich.  Using Perl to Check Web Links , 1997 .

[11]  Brian Marick,et al.  When Should a Test Be Automated , 1998 .

[12]  Peter A. Vogel,et al.  An integrated general purpose automated test environment , 1993, ISSTA '93.

[13]  Wayne Zachary,et al.  A Tool for Automatically Generating Transcripts of Human-Computer Interaction , 1993 .

[14]  Atif M. Memon,et al.  Empirical evaluation of the fault-detection effectiveness of smoke regression test cases for GUI-based software , 2004, 20th IEEE International Conference on Software Maintenance, 2004. Proceedings..

[15]  H. Sahai,et al.  The Analysis of Variance: Fixed, Random and Mixed Models , 2000 .

[16]  Steve McConnell,et al.  Daily Build and Smoke Test , 1996, IEEE Software.

[17]  Hal Berghel.  Using the WWW Test Pattern to check HTML Client Compliance , 1995, Computer.

[18]  G. J. G. Upton,et al.  Applied Multivariate Data Analysis, Volume 1: Regression and Experimental Design , 1994, The Mathematical Gazette.

[19]  Atif M. Memon,et al.  Automating regression testing for evolving GUI software: Research Articles , 2005 .

[20]  Atif M. Memon,et al.  What test oracle should I use for effective GUI testing? , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[21]  Mary Lou Soffa,et al.  A comprehensive framework for testing graphical user interfaces , 2001 .

[22]  Even-André Karlsson,et al.  Daily build and feature development in large distributed projects , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.