Automated GUI performance testing

A significant body of prior work has devised approaches for automating the functional testing of interactive applications. However, little work exists for automatically testing their performance. Performance testing imposes additional requirements upon GUI test automation tools: the tools have to be able to replay complex interactive sessions, and they have to avoid perturbing the application’s performance. We study the feasibility of using five Java GUI capture and replay tools for GUI performance test automation. Besides confirming the severity of the previously known GUI element identification problem, we also describe a related problem, the temporal synchronization problem, which is of increasing importance for GUI applications that use timer-driven activity. We find that most of the tools we study have severe limitations when used for recording and replaying realistic sessions of real-world Java applications and that all of them suffer from the temporal synchronization problem. However, we find that the most reliable tool, Pounder, causes only limited perturbation and thus can be used to automate performance testing. Based on an investigation of Pounder’s approach, we further improve its robustness and reduce its perturbation. Finally, we demonstrate in a set of case studies that the conclusions about perceptible performance drawn from manual tests still hold when using automated tests driven by Pounder. Besides the significance of our findings to GUI performance testing, the results are also relevant to capture and replay-based functional GUI test automation approaches.

[1]  Bin Mu,et al.  Design and Implementation of GUI Automated Testing Framework Based on XML , 2009, 2009 WRI World Congress on Software Engineering.

[2]  Fevzi Belli,et al.  Finite state testing and analysis of graphical user interfaces , 2001, Proceedings 12th International Symposium on Software Reliability Engineering.

[3]  Myra B. Cohen,et al.  Towards Dynamic Adaptive Automated Test Generation for Graphical User Interfaces , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[4]  Mary Lou Soffa,et al.  Hierarchical GUI Test Case Generation Using Automated Planning , 2001, IEEE Trans. Software Eng..

[5]  William C. Chu,et al.  An object-oriented architecture supporting Web application testing , 1999, Proceedings. Twenty-Third Annual International Computer Software and Applications Conference (Cat. No.99CB37032).

[6]  Paolo Tonella,et al.  A case study-based comparison of web testing techniques applied to AJAX web applications , 2008, International Journal on Software Tools for Technology Transfer.

[7]  Satish Narayanasamy,et al.  BugNet: continuously recording program execution for deterministic replay debugging , 2005, 32nd International Symposium on Computer Architecture (ISCA'05).

[8]  Atif M. Memon,et al.  Relationships between Test Suites, Faults, and Fault Detection in GUI Testing , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[9]  Matthias Hauswirth,et al.  Producing wrong data without doing anything obviously wrong! , 2009, ASPLOS.

[10]  Atif M. Memon,et al.  An Initial Characterization of Industrial Graphical User Interface Systems , 2009, 2009 International Conference on Software Testing Verification and Validation.

[11]  Matthias Hauswirth,et al.  Evaluating the accuracy of Java profilers , 2010, PLDI '10.

[12]  Atif M. Memon,et al.  Designing and comparing automated test oracles for GUI-based software applications , 2007, TSEM.

[13]  Sreedevi Sampath.  Towards defining and exploiting similarities in Web application use cases through user session analysis , 2004 .

[14]  João Saraiva,et al.  A generic library for GUI reasoning and testing , 2009, SAC '09.

[15]  Forrest Shull,et al.  Experimenting with software testbeds for evaluating new technologies , 2007, Empirical Software Engineering.

[16]  James Miller,et al.  A practical approach to testing GUI systems , 2006, Empirical Software Engineering.

[17]  David J. Kasik,et al.  Toward automatic generation of novice user test scripts , 1996, CHI.

[18]  Gregg Rothermel,et al.  Improving web application testing with user session data , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[19]  A. Ruiz,et al.  GUI Testing Made Easy , 2008, Testing: Academic & Industrial Conference - Practice and Research Techniques (taic part 2008).

[20]  Hai Jin,et al.  Live migration of virtual machine based on full system trace and replay , 2009, HPDC '09.

[21]  Edward L. Jones,et al.  Specification-driven automated testing of GUI-based Java programs , 2004, ACM-SE 42.

[22]  Atif M. Memon,et al.  Using a pilot study to derive a GUI model for automated testing , 2008, TSEM.

[23]  Arie van Deursen,et al.  Research Issues in the Automated Testing of Ajax Applications , 2010, SOFSEM.

[24]  Atif M. Memon,et al.  Call-Stack Coverage for GUI Test Suite Reduction , 2008, IEEE Trans. Software Eng..

[25]  Matthias Hauswirth,et al.  Automating performance testing of interactive Java applications , 2010, AST '10.

[26]  Atif M. Memon,et al.  Test case generator for GUITAR , 2008, ICSE Companion '08.

[27]  James F. Power,et al.  An approach to quantifying the run-time behaviour of Java GUI applications , 2004 .

[28]  Amer Diwan,et al.  The DaCapo benchmarks: java benchmarking development and analysis , 2006, OOPSLA '06.

[29]  Chen Fu,et al.  Maintaining and evolving GUI-directed test scripts , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[30]  Matthias Hauswirth,et al.  Trevis: a context tree visualization & analysis framework and its use for classifying performance failure reports , 2010, SOFTVIS '10.

[31]  Lee J. White,et al.  Generating test cases for GUI responsibilities using complete interaction sequences , 2000, Proceedings 11th International Symposium on Software Reliability Engineering. ISSRE 2000.

[32]  Sergiu M. Dascalu,et al.  Unit-level test adequacy criteria for visual dataflow languages and a testing methodology , 2008, TSEM.

[33]  Dianxiang Xu,et al.  A Systematic Capture and Replay Strategy for Testing Complex GUI Based Java Applications , 2010, 2010 Seventh International Conference on Information Technology: New Generations.

[34]  Atif M. Memon,et al.  Automated gui testing guided by usage profiles , 2007, ASE.

[35]  Insup Lee,et al.  Model-Based Testing of GUI-Driven Applications , 2009, SEUS.

[36]  Gregg Rothermel,et al.  Leveraging user-session data to support Web application testing , 2005, IEEE Transactions on Software Engineering.

[37]  Josh Fischman,et al.  Testing made easy. , 2003, U.S. news & world report.

[38]  Atif M. Memon,et al.  Automatically repairing event sequence-based GUI test suites for regression testing , 2008, TSEM.

[39]  Atif M. Memon,et al.  An Extensible Heuristic-Based Framework for GUI Test Case Maintenance , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[40]  Paul A. Strooper,et al.  Model-based testing of multiple GUI variants using the GUI test generator , 2010, AST '10.

[41]  Alex Ruiz,et al.  Test-Driven GUI Development with TestNG and Abbot , 2007, IEEE Software.

[42]  David Chenho Kung,et al.  Object-based data flow testing of web applications , 2000, Proceedings First Asia-Pacific Conference on Quality Software.

[43]  Atif M. Memon,et al.  Automating regression testing for evolving GUI software: Research Articles , 2005 .

[44]  Arie van Deursen,et al.  Invariant-based automatic testing of AJAX user interfaces , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[45]  Koen De Bosschere,et al.  Record/replay for nondeterministic program executions , 2003, CACM.

[46]  Izzat Alsmadi.  The Utilization of User Sessions in Testing , 2008, Seventh IEEE/ACIS International Conference on Computer and Information Science (icis 2008).

[47]  Koen De Bosschere,et al.  RecPlay: a fully integrated practical record/replay system , 1999, TOCS.

[48]  Qing Xie,et al.  Developing cost-effective model-based techniques for GUI testing , 2006, ICSE.

[49]  Charles Lowell,et al.  Successful Automation of GUI Driven Acceptance Testing , 2003, XP.

[50]  John Steven,et al.  jRapture: A Capture/Replay tool for observation-based testing , 2000, ISSTA '00.

[51]  Daniel P. Siewiorek,et al.  A method to automate user interface testing using variable finite state machines , 1997, Proceedings of IEEE 27th International Symposium on Fault Tolerant Computing.

[52]  Zhendong Su,et al.  ExecRecorder: VM-based full-system replay for attack analysis and system recovery , 2006, ASID '06.

[53]  Atif M. Memon,et al.  Generating Event Sequence-Based Test Cases Using GUI Runtime State Feedback , 2010, IEEE Transactions on Software Engineering.

[54]  Lieven Eeckhout,et al.  Statistically rigorous java performance evaluation , 2007, OOPSLA.

[55]  Matthias Hauswirth,et al.  Measuring the performance of interactive applications with listener latency profiling , 2008, PPPJ '08.

[56]  Atif M. Memon,et al.  Using GUI Run-Time State as Feedback to Generate Test Cases , 2007, 29th International Conference on Software Engineering (ICSE'07).

[57]  Mengqi Wu,et al.  Effective GUI Testing Automation: Developing an Automated GUI Testing Tool , 2004 .

[58]  Rob Miller,et al.  GUI testing using computer vision , 2010, CHI.

[59]  Atif M. Memon,et al.  Automating regression testing for evolving GUI software , 2005, J. Softw. Maintenance Res. Pract..

[60]  Gerard Meszaros,et al.  Agile regression testing using record & playback , 2003, OOPSLA '03.

[61]  Atif M. Memon,et al.  Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software , 2005, IEEE Transactions on Software Engineering.

[62]  Matthias Hauswirth,et al.  Characterizing the design and performance of interactive java applications , 2010, 2010 IEEE International Symposium on Performance Analysis of Systems & Software (ISPASS).

[63]  Giuseppe A. Di Lucca,et al.  Testing Web applications , 2002, International Conference on Software Maintenance, 2002. Proceedings..

[64]  Paolo Tonella,et al.  Analysis and testing of Web applications , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[65]  Paolo Tonella,et al.  State-Based Testing of Ajax Web Applications , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[66]  David Chenho Kung,et al.  Structural testing of Web applications , 2000, Proceedings 11th International Symposium on Software Reliability Engineering. ISSRE 2000.