A comprehensive framework for testing graphical user interfaces

The widespread recognition of the usefulness of graphical user interfaces (GUIs) has established their importance as critical components of today's software. Although the use of GUIs continues to grow, GUI testing has remained a neglected research area. Since GUIs have characteristics that are different from those of conventional software, such as user events for input and graphical output, techniques developed to test conventional software cannot be directly applied to test GUIs. This thesis develops a unified solution to the GUI testing problem with the particular goals of automation and integration of tools and techniques used in various phases of GUI testing. These goals are accomplished by developing a GUI testing framework with a GUI model as its central component. For efficiency and scalability, a GUI is represented as a hierarchy of components, each used as a basic unit of testing. The framework also includes a test coverage evaluator, test case generator, test oracle, test executor, and regression tester. The test coverage evaluator employs hierarchical, event-based coverage criteria to automatically specify what to test in a GUI and to determine whether the test suite has adequately tested the GUI. The test case generator employs plan generation techniques from artificial intelligence to automatically generate a test suite. A test executor automatically executes all the test cases on the GUI. As test cases are being executed, a test oracle automatically determines the correctness of the GUI. The test oracle employs a model of the expected state of the GUI in terms of its constituent objects and their properties. After changes are made to a GUI, a regression tester partitions the original GUI test suite into valid test cases that represent correct input/output for the modified GUI and invalid test cases that no longer represent correct input/output. The regression tester employs a new technique to reuse some of the invalid test cases by repairing them. A cursory exploration of extending the framework to handle the new testing requirements of web-user interfaces (WUIs) is also done. The framework has been implemented and experiments have demonstrated that the developed techniques are both practical and useful.

[1]  James D. Foley,et al.  Controlling user interface objects through pre- and postconditions , 1992, CHI.

[2]  Boris Beizer,et al.  Software testing techniques (2. ed.) , 1990 .

[3]  Lydie du Bousquet,et al.  Lutess: a specification-driven testing environment for synchronous software , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[4]  B GoodenoughJohn,et al.  Toward a theory of test data selection , 1975 .

[5]  David W. Binkley,et al.  Reducing the cost of regression testing by semantics guided test case selection , 1995, Proceedings of International Conference on Software Maintenance.

[6]  Amedeo Cesta,et al.  Recent Advances in AI Planning , 1997, Lecture Notes in Computer Science.

[7]  Laura K. Dillon,et al.  Generating oracles from your favorite temporal logic specifications , 1996, SIGSOFT '96.

[8]  Brad A. Myers,et al.  State of the Art in User Interface Software Tools , 1992 .

[9]  Gregg Rothermel,et al.  A Comparative Study of Regression Test Selection Techniques , 1997 .

[10]  William E. Perry,et al.  Effective methods for software testing , 1995 .

[11]  Thomas J. Ostrand,et al.  A visual test development environment for GUI systems , 1998, ISSTA.

[12]  Richard Fikes,et al.  STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving , 1971, IJCAI.

[13]  James A. Hendler,et al.  HTN Planning: Complexity and Expressivity , 1994, AAAI.

[14]  Bernhard Nebel,et al.  Extending Planning Graphs to an ADL Subset , 1997, ECP.

[15]  Guy L. Steele,et al.  Java Language Specification, Second Edition: The Java Series , 2000 .

[16]  Henry A. Kautz,et al.  BLACKBOX: A New Approach to the Application of Theorem Proving to Problem Solving , 1998 .

[17]  Jeffrey J. Hendrickson,et al.  Integrated data capture and analysis tools for research and testing on graphical user interfaces , 1992, CHI.

[18]  Engin Kirda,et al.  Web engineering device independent Web services , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[19]  Daniel S. Weld.  An Introduction to Least Commitment Planning , 1994, AI Mag..

[20]  B. Selman.  The Role of Domain-speciic Knowledge in the Planning as Satissability Framework , 1998 .

[21]  Brad A. Myers,et al.  User interface tools , 1994, CHI '94.

[22]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[23]  Peter A. Vogel,et al.  An integrated general purpose automated test environment , 1993, ISSTA '93.

[24]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[25]  Vladimir Lifschitz,et al.  ON THE SEMANTICS OF STRIPS , 1987 .

[26]  J WeyukerElaine,et al.  Using Coverage Information to Predict the Cost-Effectiveness of Regression Testing Strategies , 1997 .

[27]  David W. Binkley,et al.  Semantics Guided Regression Test Cost Reduction , 1997, IEEE Trans. Software Eng..

[28]  Aniello Cimitile,et al.  Post-maintenance testing based on path change analysis , 1988, Proceedings. Conference on Software Maintenance, 1988..

[29]  John S. Gourlay.  A Mathematical Framework for the Investigation of Testing , 1983, IEEE Transactions on Software Engineering.

[30]  Mary Lou Soffa,et al.  Comparing Causal-Link and Propositional Planners : Tradeoffs between Plan Length and Domain Size , 2022 .

[31]  Jim Clarke.  Automated test generation from a behavioral model , 1998 .

[32]  Bart Selman,et al.  Planning as Satisfiability , 1992, ECAI.

[33]  Ben Shneiderman,et al.  Visual and Textual Consistency Checking Tools for Graphical User Interfaces , 1997, IEEE Trans. Software Eng..

[34]  Elaine J. Weyuker.  Translatability and Decidability Questions for Restricted Classes of Program Schemas , 1979, SIAM J. Comput..

[35]  David J. Kasik,et al.  Toward automatic generation of novice user test scripts , 1996, CHI.

[36]  Tsun S. Chow,et al.  Testing Software Design Modeled by Finite-State Machines , 1978, IEEE Transactions on Software Engineering.

[37]  Manuela M. Veloso,et al.  FLECS: Planning with a Flexible Commitment Strategy , 1995, J. Artif. Intell. Res..

[38]  Daniel P. Siewiorek,et al.  A method to automate user interface testing using variable finite state machines , 1997, Proceedings of IEEE 27th International Symposium on Fault Tolerant Computing.

[39]  David Holmes,et al.  The Java Programming Language, Third Edition , 2000 .

[40]  Brad A. Myers,et al.  User interface software tools , 1995, TCHI.

[41]  Fabio Somenzi,et al.  Redundancy identification/removal and test generation for sequential circuits using implicit state enumeration , 1993, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[42]  Qing Yu,et al.  Oracles for checking temporal properties of concurrent systems , 1994, SIGSOFT '94.

[43]  Daniel A. Menascé,et al.  Load Testing of Web Sites , 2002, IEEE Internet Comput..

[44]  James D. Hollan,et al.  Strategic directions in human-computer interaction , 1996, CSUR.

[45]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[46]  David S. Rosenblum,et al.  Predicting the cost-effectiveness of regression testing strategies , 1996, SIGSOFT '96.

[47]  Avrim Blum,et al.  Fast Planning Through Planning Graph Analysis , 1995, IJCAI.

[48]  Gregg Rothermel,et al.  Empirical Studies of a Safe Regression Test Selection Technique , 1998, IEEE Trans. Software Eng..

[49]  B. Eng,et al.  GENERATING A TEST ORACLE FROM PROGRAM DOCUMENTATION , 1995 .

[50]  Adam A. Porter,et al.  Specification-based Testing of Reactive Software: Tools and Experiments Experience Report , 1997, Proceedings of the (19th) International Conference on Software Engineering.

[51]  Gregg Rothermel,et al.  A safe, efficient algorithm for regression test selection , 1993, 1993 Conference on Software Maintenance.

[52]  E. Pednault.  Toward a mathematical theory of plan synthesis , 1987 .

[53]  Johanna D. Moore,et al.  Decomposition and Causality in Partial-order Planning , 1994, AIPS.

[54]  Matthew L. Ginsberg,et al.  Procedural Reasoning in Constraint Satisfaction , 1998, KR.

[55]  J. McCarthy.  Situations, Actions, and Causal Laws , 1963 .

[56]  Lee J. White.  Regression testing of GUI event interactions , 1996, 1996 Proceedings of International Conference on Software Maintenance.

[57]  Elaine J. Weyuker,et al.  The Automatic Generation of Load Test Suites and the Assessment of the Resulting Software , 1995, IEEE Trans. Software Eng..

[58]  Andrzej Pelc,et al.  The complexity of data mining on the Web , 1996, PODC '96.

[59]  Ankur R. Hajare,et al.  Testing the Human Computer Interface for the Telerobotic Assembly of the Space Station , 1993, HCI.

[60]  Roger S. Pressman,et al.  Software Engineering: A Practitioner's Approach , 1982 .

[61]  David S. Rosenblum,et al.  Using Coverage Information to Predict the Cost-Effectiveness of Regression Testing Strategies , 1997, IEEE Trans. Software Eng..

[62]  Brad A. Myers,et al.  Why are Human-Computer interfaces Difficult to Design and Implement? , 1993 .

[63]  Debra J. Richardson,et al.  Specification-based test oracles for reactive systems , 1992, International Conference on Software Engineering.

[64]  Philip J. Bernhard,et al.  A reduced test suite for protocol conformance testing , 1994, TSEM.

[65]  Lee J. White,et al.  Generating test cases for GUI responsibilities using complete interaction sequences , 2000, Proceedings 11th International Symposium on Software Reliability Engineering. ISSRE 2000.

[66]  Lawrence S. Kroll.  Mathematica--A System for Doing Mathematics by Computer. , 1989 .

[67]  Lawrence J. Fogel,et al.  Artificial Intelligence through Simulated Evolution , 1966 .

[68]  S. L. Gerhart,et al.  Toward a theory of test data selection , 1975, IEEE Transactions on Software Engineering.

[69]  Thomas Ball,et al.  On the limit of control flow analysis for regression test selection , 1998, ISSTA '98.

[70]  James A. Hendler,et al.  Toward a general framework for hierarchical task-network planning , 1993 .

[71]  Michael R. Donat.  Automating Formal Speci cation-Based Testing To Appear: TAPSOFT '97 , 1997 .

[72]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[73]  Daniel S. Weld,et al.  UCPOP: A Sound, Complete, Partial Order Planner for ADL , 1992, KR.

[74]  Jo W. Tombaugh,et al.  Usability testing: is the whole test greater than the sum of its parts? , 1992, CHI '92.

[75]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[76]  David Chenho Kung,et al.  Or Regression Testing of Object-Oriented Programs , 1996, J. Syst. Softw..

[77]  Joseph Robert Horgan,et al.  Incremental regression testing , 1993, 1993 Conference on Software Maintenance.

[78]  Michal Young,et al.  Residual test coverage monitoring , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[79]  Martha E. Pollack,et al.  Flaw Selection Strategies for Partial-Order Planning , 1997, J. Artif. Intell. Res..

[80]  Bart Selman,et al.  Pushing the Envelope: Planning, Propositional Logic and Stochastic Search , 1996, AAAI/IAAI, Vol. 2.

[81]  D. Richardson.  Testing with Analysis and Oracle Support Taos Test Development Test Execution Test Measurement Parallel Test Executor Behavior Verifier Coverage Analyzer Test Criterion Deriver Test Case Editor Test Suite Editor Test Oracle Editor , 2007 .

[82]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[83]  Debra J. Richardson,et al.  TAOS: Testing with Analysis and Oracle Support , 1994, ISSTA '94.

[84]  Edwin P. D. Pednault,et al.  ADL: Exploring the Middle Ground Between STRIPS and the Situation Calculus , 1989, KR.

[85]  Mary Lou Soffa,et al.  Interprocedual Data Flow Testing , 1989, Symposium on Testing, Analysis, and Verification.

[86]  Bart Selman,et al.  The Role of Domain-Specific Knowledge in the Planning as Satisfiability Framework , 1998, AIPS.

[87]  Paul R. Ritter,et al.  Experience in testing the Motif interface , 1991, IEEE Software.

[88]  BallThomas.  On the limit of control flow analysis for regression test selection , 1998 .

[89]  Phyllis G. Frankl,et al.  A framework for testing database applications , 2000, ISSTA '00.

[90]  Michael R. Donat.  Automating Formal Specification-Based Testing , 1997, TAPSOFT.

[91]  J WeyukerElaine,et al.  Predicting the cost-effectiveness of regression testing strategies , 1996 .

[92]  James Gosling.  The Java Language Specification - Second Edition , 2000 .

[93]  Bart Selman,et al.  Randomization in Backtrack Search: Exploiting Heavy-Tailed Profiles for Solving Hard Scheduling Problems , 1998, AIPS.

[94]  Hong Zhu,et al.  Test data adequacy measurement , 1993, Softw. Eng. J..