On the Industrial Applicability of Visual GUI Testing

There is constant pressure on the software development industry to improve software quality and deliver new and innovative software faster and more efficiently. These demands affect all aspects of software development, from requirements engineering to testing. Today, much testing is performed with costly, tedious and error-prone manual practices, especially on higher levels of system abstraction such as in system and acceptance testing. Increased test automation has been proposed as a key solution to help alleviate these test-related problems and has found increasing use. Automated tests typically consist of scripts that can be executed to give frequent feedback on the system's quality, ensure that previously correct functionality has not been negatively affected by changes, and also relieve resources e.g. human testers. However, as a system evolves, changes to the requirements also require maintenance of the tests, which can be costly. Furthermore, most automated test techniques approach testing from a lower level of system abstraction, e.g. testing the low-level components and functions, but their use for system testing has been questioned. Test techniques have been proposed that automate the interaction with the graphical user interface (GUI) of a system. For example, record and replay tools can be used to automate system testing by emulating a user interacting with the system. Existing literature have pointed out several shortcomings of these techniques, for example they are sensitive to GUI layout and code changes, require access to source code, are dependent on a specific platform or operating system access etc. In this thesis we evaluate if Visual GUI Testing (VGT), a novel technique that combines test scripts with image recognition, can be used to automate high-level, software testing. The strength of VGT lies in the use of image recognition that makes the technique robust in instances where previous GUIbased test techniques had limitations. Yet, VGT is only sparsely applied in industry and the academic body of knowledge on the technique is limited; there is a lack of empirical evaluation and experience reports from industrial use. To fill this gap in knowledge we have conducted three empirical case studies applying different VGT tools in industrial software projects at two Swedish companies. We have also conducted an industrial survey with participants from multiple companies. These studies provide evidence that VGT can be successfully applied in industrial software development projects, but also detail challenges, problems and limitations that need to be overcome for more widespread industrial adoption. Future work should focus on evaluating the long-term maintenance costs of VGT test scripts, since our studies only present initial evidence that these costs are manageable in relation to the benefits of the technique.

[1]  Juha Itkonen,et al.  Exploratory testing: a multiple case study , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[2]  Lars Pareto,et al.  Costs and Benefits of Structure-aware Capture/Replay tools , 2006 .

[3]  S. K. Lo,et al.  Reliability and Validity , 2020, International Encyclopedia of Human Geography.

[4]  Atif M. Memon,et al.  An event‐flow model of GUI‐based applications for testing , 2007, Softw. Test. Verification Reliab..

[5]  Chen Fu,et al.  Experimental assessment of manual versus tool-based maintenance of GUI-directed test scripts , 2009, 2009 IEEE International Conference on Software Maintenance.

[6]  David Chenho Kung,et al.  Software requirements and acceptance testing , 1997, Ann. Softw. Eng..

[7]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .

[8]  Laurie A. Williams,et al.  On the Effectiveness of Unit Test Automation at Microsoft , 2009, 2009 20th International Symposium on Software Reliability Engineering.

[9]  Robert V. Binder,et al.  Testing Object-Oriented Systems: Models, Patterns, and Tools , 1999 .

[10]  Gary T. Leavens,et al.  A Simple and Practical Approach to Unit Testing: The JML and JUnit Way , 2002, ECOOP.

[11]  Jocelyn Armarego,et al.  CASE Tools: Constructivism and its Application to Learning and Usability of Software Engineering Tools , 2001, Comput. Sci. Educ..

[12]  J. Kitzinger,et al.  Developing Focus Group Research: Politics, Theory and Practice , 1998 .

[13]  J. Jacoby,et al.  Is There an Optimal Number of Alternatives for Likert Scale Items? Study I: Reliability and Validity , 1971 .

[14]  Mary Lou Soffa,et al.  Hierarchical GUI Test Case Generation Using Automated Planning , 2001, IEEE Trans. Software Eng..

[15]  Luke S. Zettlemoyer,et al.  A visual medium for programmatic control of interactive applications , 1999, CHI '99.

[16]  Jin Chen,et al.  When a GUI Regression Test Failed, What Should be Blamed? , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[17]  Sarfraz Khurshid,et al.  Symbolic execution for software testing in practice: preliminary assessment , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[18]  Kari Känsälä,et al.  Software Product Family Evaluation , 2003, SPLC.

[19]  Tathagat Varma,et al.  Automated software testing: introduction, management and performance , 2000, SOEN.

[20]  Mary Lou Soffa,et al.  Regression testing of GUIs , 2003, ESEC/FSE-11.

[21]  Wei-Tek Tsai,et al.  Regression testing in an industrial environment , 1998, CACM.

[22]  Kari Känsälä,et al.  Software Product Family Evaluation , 2004, SPLC.

[23]  Joseph L. Hellerstein,et al.  Reducing the Cost of IT Operations - Is Automation Always the Answer? , 2005, HotOS.

[24]  Sigrid Eldh,et al.  Analysis of Mistakes as a Method to Improve Test Case Design , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[25]  Per Runeson,et al.  Combining Scenario-based Requirements with Static Verification and Dynamic Testing , 1998, REFSQ.

[26]  B. Paech,et al.  Criteria for Software Testing Tool Evaluation – A Task Oriented View , 2005 .

[27]  Robert Feldt,et al.  Automated System Testing Using Visual GUI Testing Tools: A Comparative Study in Industry , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[28]  Atif M. Memon GUI Testing: Pitfalls and Process , 2002, Computer.

[29]  Helena Holmström Olsson,et al.  JAutomate: A Tool for System- and Acceptance-test Automation , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[30]  Gregg Rothermel,et al.  An empirical study of regression test selection techniques , 2001, ACM Trans. Softw. Eng. Methodol..

[31]  M. Patton,et al.  Enhancing the quality and credibility of qualitative analysis. , 1999, Health services research.

[32]  S. Berner,et al.  Observations and lessons learned from automated testing , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[33]  Johan Andersson,et al.  The Video Store Revisited Yet Again: Adventures in GUI Acceptance Testing , 2004, XP.

[34]  Kai Petersen,et al.  Benefits and limitations of automated software testing: Systematic literature review and practitioner survey , 2012, 2012 7th International Workshop on Automation of Software Test (AST).

[35]  M. Young Test Oracles , 2001 .

[36]  L. Cronbach Coefficient alpha and the internal structure of tests , 1951 .

[37]  Chen Fu,et al.  Creating GUI Testing Tools Using Accessibility Technologies , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[38]  Lefteris Angelis,et al.  Links between the personalities, views and attitudes of software engineers , 2010, Inf. Softw. Technol..

[39]  Johannes RyserÊÊÊÊÊÊÊÊÊÊÊÊÊÊMartin Glinz,et al.  A Scenario-Based Approach to Validating and Testing Software Systems Using Statecharts , 1999 .

[40]  David Chenho Kung,et al.  Behavior-based acceptance testing of software systems: a formal scenario approach , 1994, Proceedings Eighteenth Annual International Computer Software and Applications Conference (COMPSAC 94).

[41]  Marc Kellogg,et al.  Automating functional tests using Selenium , 2006, AGILE 2006 (AGILE'06).

[42]  Atif M. Memon,et al.  Test case generator for GUITAR , 2008, ICSE Companion '08.

[43]  Helen Sharp,et al.  Models of motivation in software engineering , 2009, Inf. Softw. Technol..

[44]  Matthias Hauswirth,et al.  Automated GUI performance testing , 2011, Software Quality Journal.

[45]  James Miller,et al.  A practical approach to testing GUI systems , 2006, Empirical Software Engineering.

[46]  Rajesh Subramanyan,et al.  A survey on model-based testing approaches: a systematic review , 2007, WEASELTech '07.

[47]  Albert L. Lederer,et al.  The validation of a political model of information systems development cost estimating , 1991, SIGCPR '91.

[48]  Atif M. Memon,et al.  Automated gui testing guided by usage profiles , 2007, ASE.

[49]  James A. Whittaker,et al.  Model‐Based Software Testing , 2002 .

[50]  Rhys William Andrews,et al.  External Constraints on Local Service Standards: The Case of Comprehensive Performance Assessment in English Local Government , 2005 .

[51]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[52]  Peter Fröhlich,et al.  Automated Test Case Generation from Dynamic Models , 2000, ECOOP.

[53]  Christof Ebert,et al.  The impacts of software product management , 2007, J. Syst. Softw..

[54]  Michael Olan,et al.  Unit testing: test early, test often , 2003 .

[55]  Mika Mäntylä,et al.  The Role of the Tester's Knowledge in Exploratory Software Testing , 2013, IEEE Transactions on Software Engineering.

[56]  Dave Astels,et al.  The RSpec Book: Behaviour Driven Development with RSpec, Cucumber, and Friends , 2010 .

[57]  Andy Hopper,et al.  Virtual Network Computing , 1998, IEEE Internet Comput..

[58]  Mengqi Wu,et al.  Effective GUI Testing Automation: Developing an Automated GUI Testing Tool , 2004 .

[59]  Kent L. Beck,et al.  Test-driven Development - by example , 2002, The Addison-Wesley signature series.

[60]  Atif M. Memon,et al.  GUI ripping: reverse engineering of graphical user interfaces for testing , 2003, 10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings..

[61]  Jesús Torres Valderrama,et al.  Generation of test cases from functional requirements. A survey , 2006 .

[62]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[63]  John E. G. Bateson,et al.  Exploring Complex Decision Making Units: A New Approach , 1982 .

[64]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[65]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[66]  Woei-Kae Chen,et al.  Integration of specification-based and CR-based approaches for GUI testing , 2005, 19th International Conference on Advanced Information Networking and Applications (AINA'05) Volume 1 (AINA papers).

[67]  Emil Börjesson Multi-Perspective Analysis of Software Development: a method and an Industrial Case Study , 2010 .

[68]  A. Pretschner,et al.  Model-based testing for real , 2004, International Journal on Software Tools for Technology Transfer.

[69]  Matthias Hauswirth,et al.  Automating performance testing of interactive Java applications , 2010, AST '10.

[70]  Brian Marick,et al.  When Should a Test Be Automated , 1998 .

[71]  Mary Lou Soffa,et al.  Automated test oracles for GUIs , 2000, SIGSOFT '00/FSE-8.

[72]  Woei-Kae Chen,et al.  Bad Smells and Refactoring Methods for GUI Test Scripts , 2012, 2012 13th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing.

[73]  Gérard Lachapelle,et al.  Testing GPS L5 Acquisition and Tracking Algorithms Using a Hardware Simulator , 2006 .

[74]  Sigrid Eldh Software Testing Techniques , 2007 .

[75]  Tore Dybå,et al.  A systematic review of quasi-experiments in software engineering , 2009, Inf. Softw. Technol..

[76]  Joseph A. Maxwell,et al.  Qualitative Research Design: An Interactive Approach , 1996 .

[77]  Robert C. Martin Professionalism and Test-Driven Development , 2007, IEEE Software.

[78]  Debra J. Richardson,et al.  Specification-based test oracles for reactive systems , 1992, International Conference on Software Engineering.

[79]  Colin Robson,et al.  Real World Research: A Resource for Social Scientists and Practitioner-Researchers , 1993 .

[80]  Frank van der Linden,et al.  Family Evaluation Framework overview & introduction , 2005 .

[81]  Zongxiang Zhang,et al.  Using graphical representation of user interfaces as visual references , 2012 .

[82]  Ellis Horowitz,et al.  Graphical User Interface Testing , 2012 .

[83]  Fadi A. Zaraket,et al.  GUICOP: Specification-Based GUI Testing , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[84]  Bertrand Meyer,et al.  Reconciling Manual and Automated Testing: The AutoTest Experience , 2007, 2007 40th Annual Hawaii International Conference on System Sciences (HICSS'07).

[85]  Robert Feldt,et al.  Transitioning Manual System Test Suites to Automated Testing: An Industrial Case Study , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[86]  Edgars Diebelis,et al.  Self-Testing Approach and Testing Tools , 2012 .

[87]  Richard Potter,et al.  Triggers: guiding automation with pixels to achieve data access , 1993 .

[88]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[89]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[90]  Malte Finsterwalder Automating Acceptance Tests for GUI Applications in an Extreme Programming Environment , 2001 .

[91]  Elaine J. Weyuker,et al.  Testing Component-Based Software: A Cautionary Tale , 1998, IEEE Softw..

[92]  Chen Fu,et al.  Maintaining and evolving GUI-directed test scripts , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[93]  D R Shearer,et al.  Acceptance testing. , 1998, Radiology management.

[94]  A. Bryman The debate about quantitative and qualitative research: A question of method or epistemology? , 1984 .

[95]  Tony Gorschek,et al.  A Model for Technology Transfer in Practice , 2006, IEEE Software.

[96]  M. F.,et al.  Bibliography , 1985, Experimental Gerontology.

[97]  Charles Lowell,et al.  Successful Automation of GUI Driven Acceptance Testing , 2003, XP.

[98]  Rob Miller,et al.  GUI testing using computer vision , 2010, CHI.

[99]  Ken Schwaber,et al.  SCRUM Development Process , 1997 .

[100]  Carl E. Rasmussen,et al.  The Need for Open Source Software in Machine Learning , 2007, J. Mach. Learn. Res..

[101]  Qian Yang,et al.  A survey of coverage based testing tools , 2006, AST '06.

[102]  Rob Miller,et al.  Sikuli: using GUI screenshots for search and automation , 2009, UIST '09.