DART: a framework for regression testing "nightly/daily builds" of GUI applications

"Nightly/daily building and smoke testing" have become widespread since they often reveal bugs early in the software development process. During these builds, software is compiled, linked, and (re)tested with the goal of validating its basic functionality. Although successful for conventional software, smoke tests are difficult to develop and automatically rerun for software that has a graphical user interface (GUI). In this paper, we describe a framework called DART (daily automated regression tester) that addresses the needs of frequent and automated re-testing of GUI software. The key to our success is automation: DART automates everything from structural GUI analysis; test case generation; test oracle creation; to code instrumentation; test execution; coverage evaluation; regeneration of test cases; and their re-execution. Together with the operating system's task scheduler, DART can execute frequently with little input from the developer/tester to retest the GUI software. We provide results of experiments showing the time taken and memory required for GUI analysis, test case and test oracle generation, and test execution. We also empirically compare the relative costs of employing different levels of detail in the GUI test cases.

[1]  Charles J. Poole,et al.  Using Extreme Programming in a Maintenance Environment , 2001, IEEE Softw..

[2]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[3]  David W. Binkley,et al.  Semantics Guided Regression Test Cost Reduction , 1997, IEEE Trans. Software Eng..

[4]  Mary Lou Soffa,et al.  A comprehensive framework for testing graphical user interfaces , 2001 .

[5]  K. Beck,et al.  Extreme Programming Explained , 2002 .

[6]  Peter Schuh.  Recovery, Redemption, and Extreme Programming , 2001, IEEE Softw..

[7]  Mary Lou Soffa,et al.  Hierarchical GUI Test Case Generation Using Automated Planning , 2001, IEEE Trans. Software Eng..

[8]  Mary Lou Soffa,et al.  Using a goal-driven approach to generate test cases for GUIs , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[9]  Steve McConnell,et al.  Daily Build and Smoke Test , 1996, IEEE Software.

[10]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .

[11]  Mary Lou Soffa,et al.  Coverage criteria for GUI testing , 2001, ESEC/FSE-9.

[12]  Mary Lou Soffa,et al.  Automated test oracles for GUIs , 2000, SIGSOFT '00/FSE-8.

[13]  Lee J. White,et al.  User-based testing of GUI sequences and their interactions , 2001, Proceedings 12th International Symposium on Software Reliability Engineering.

[14]  Malte Finsterwalder.  Automating Acceptance Tests for GUI Applications in an Extreme Programming Environment , 2001 .

[15]  James Grenning,et al.  Launching Extreme Programming at a Process-Intensive Company , 2001, IEEE Softw..

[16]  Even-André Karlsson,et al.  Daily build and feature development in large distributed projects , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[17]  Heinz Züllighoven,et al.  The Need for Speed : Automating Acceptance Testing in an eXtreme Programming Environment – , 2002 .

[18]  Brian Marick,et al.  When Should a Test Be Automated , 1998 .

[19]  Wayne Zachary,et al.  A Tool for Automatically Generating Transcripts of Human-Computer Interaction , 1993 .

[20]  ATIF M. MEMON.  Advances in GUI testing , 2003, Adv. Comput..

[21]  Marilyn C. Salzman,et al.  Smoke and mirrors: setting the stage for a successful usability test , 1994, Behav. Inf. Technol..

[22]  David S. Rosenblum,et al.  Using Coverage Information to Predict the Cost-Effectiveness of Regression Testing Strategies , 1997, IEEE Trans. Software Eng..

[23]  Ron Jeffries,et al.  Extreme Programming Installed , 2000 .

[24]  Brad A. Myers,et al.  User interface software tools , 1995, TCHI.

[25]  Cem Kaner.  Improving the maintainability of automated test suites , 1997 .

[26]  Atif M. Memon.  GUI Testing: Pitfalls and Process , 2002, Computer.