Generating Event Sequence-Based Test Cases Using GUI Runtime State Feedback

This paper presents a fully automatic model-driven technique to generate test cases for graphical user interfaces (GUIs)-based applications. The technique uses feedback from the execution of a ¿seed test suite,¿ which is generated automatically using an existing structural event interaction graph model of the GUI. During its execution, the runtime effect of each GUI event on all other events pinpoints event semantic interaction (ESI) relationships, which are used to automatically generate new test cases. Two studies on eight applications demonstrate that the feedback-based technique 1) is able to significantly improve existing techniques and helps identify serious problems in the software and 2) the ESI relationships captured via GUI state yield test suites that most often detect more faults than their code, event, and event-interaction-coverage equivalent counterparts.

[1]  Mark R. Blackburn,et al.  Mars Polar Lander fault identification using model-based testing , 2002, Eighth IEEE International Conference on Engineering of Complex Computer Systems, 2002. Proceedings..

[2]  Marinus J. Plasmeijer,et al.  Model-Based Testing of Thin-Client Web Applications , 2006, FATES/RV.

[3]  David Notkin,et al.  Mutually Enhancing Test Generation and Specification Inference , 2003, FATES.

[4]  Behçet Sarikaya.  Conformance Testing: Architectures and Test Sequences , 1989, Comput. Networks.

[5]  Atif M. Memon,et al.  Designing and comparing automated test oracles for GUI-based software applications , 2007, TSEM.

[6]  L. Apfelbaum,et al.  Automated functional test generation , 1995, Conference Record AUTOTESTCON '95. 'Systems Readiness: Test Technology for the 21st Century'.

[7]  XieQing,et al.  Automating regression testing for evolving GUI software , 2005 .

[8]  Fevzi Belli,et al.  Test minimization for human-computer interaction , 2006, Applied Intelligence.

[9]  Alan Hartman,et al.  Using a model-based test generator to test for standard conformance , 2002, IBM Syst. J..

[10]  James Bret Michael,et al.  Environment behavior models for scenario generation and testing automation , 2005, ACM SIGSOFT Softw. Eng. Notes.

[11]  Yong Rae Kwon,et al.  Testing of object-oriented programs based on finite state machines , 1995, Proceedings 1995 Asia Pacific Software Engineering Conference.

[12]  Gary McGraw,et al.  Generating Software Test Data by Evolution , 2001, IEEE Trans. Software Eng..

[13]  Atif M. Memon,et al.  An event‐flow model of GUI‐based applications for testing , 2007, Softw. Test. Verification Reliab..

[14]  M. Scheetz,et al.  Generating test cases from an OO model with an AI planning system , 1999 .

[15]  Anneliese Amschler Andrews,et al.  Automated testing support for a robot tape library , 1993, Proceedings of 1993 IEEE International Symposium on Software Reliability Engineering.

[16]  Mike Holcombe,et al.  Specification and testing using generalized machines: a presentation and a case study , 1998 .

[17]  Levi Lucio,et al.  A Methodology and a Framework for Model-Based Testing , 2004, RISE.

[18]  Gregg Rothermel,et al.  On test suite composition and cost-effective regression testing , 2004, TSEM.

[19]  Florentin Ipate,et al.  Complete Testing from a Stream X-Machine Specification , 2005, Fundam. Informaticae.

[20]  Anneliese Amschler Andrews,et al.  Domain based regression testing , 1994, Proceedings 1994 International Conference on Software Maintenance.

[21]  David L. Spooner,et al.  Automatic Generation of Floating-Point Test Data , 1976, IEEE Transactions on Software Engineering.

[22]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[23]  Peter M. Maurer,et al.  Generating test data with enhanced context-free grammars , 1990, IEEE Software.

[24]  James A. Whittaker.  Markov Chain Techniques for Software Testing and Reliability Analysis, PhD Dissertation , 1992 .

[25]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[26]  Lori A. Clarke,et al.  An information flow model of fault detection , 1993, ISSTA '93.

[27]  Tathagat Varma,et al.  Automated software testing: introduction, management and performance , 2000, SOEN.

[28]  Atif M. Memon,et al.  Call-Stack Coverage for GUI Test Suite Reduction , 2008, IEEE Trans. Software Eng..

[29]  Daniel P. Siewiorek,et al.  A method to automate user interface testing using variable finite state machines , 1997, Proceedings of IEEE 27th International Symposium on Fault Tolerant Computing.

[30]  Atif M. Memon,et al.  Using GUI Run-Time State as Feedback for Test Automation , 2009 .

[31]  Wen-Huei Chen,et al.  An Optimization Technique for Protocol Conformance Testing Using Multiple Uio Sequences , 1990, Inf. Process. Lett..

[32]  Jim Clarke.  Automated test generation from a behavioral model , 1998 .

[33]  Robert B. France,et al.  Generating test cases from an OO model with an AI planning system , 1999, Proceedings 10th International Symposium on Software Reliability Engineering (Cat. No.PR00443).

[34]  James Bret Michael,et al.  Environment behavior models for scenario generation and testing automation , 2005, A-MOST.

[35]  V. Lakshmi Narasimhan,et al.  ADTEST: A Test Data Generation Suite for Ada Software Systems , 1997, IEEE Trans. Software Eng..

[36]  Frances E. Allen,et al.  Control-flow analysis , 2022 .

[37]  Mary Lou Soffa,et al.  Automated test data generation using an iterative relaxation method , 1998, SIGSOFT '98/FSE-6.

[38]  István Majzik,et al.  Model-based Automatic Test Generation for Event-Driven Embedded Systems using Model Checkers , 2006, 2006 International Conference on Dependability of Computer Systems.

[39]  Emin Gün Sirer,et al.  Using production grammars in software testing , 1999, DSL '99.

[40]  Mary Lou Soffa,et al.  Hierarchical GUI Test Case Generation Using Automated Planning , 2001, IEEE Trans. Software Eng..

[41]  Debra J. Richardson,et al.  An Analysis of Test Data Selection Criteria Using the RELAY Model of Fault Detection , 1993, IEEE Trans. Software Eng..

[42]  M MemonAtif,et al.  Designing and comparing automated test oracles for GUI-based software applications , 2007 .

[43]  Margus Veanes,et al.  Towards a Tool Environment for Model-Based Testing with AsmL , 2003, FATES.

[44]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[45]  Denise M. Woit.  Conditional-event usage testing , 1998, CASCON.

[46]  Atif M. Memon,et al.  Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software , 2005, IEEE Transactions on Software Engineering.

[47]  Tsun S. Chow,et al.  Testing Software Design Modeled by Finite-State Machines , 1978, IEEE Transactions on Software Engineering.

[48]  Marcelo d'Amorim,et al.  An Empirical Comparison of Automated Generation and Classification Techniques for Object-Oriented Unit Testing , 2006, 21st IEEE/ACM International Conference on Automated Software Engineering (ASE'06).

[49]  James A. Whittaker.  Stochastic software testing , 1997, Ann. Softw. Eng..

[50]  David J. Kasik,et al.  Toward automatic generation of novice user test scripts , 1996, CHI.

[51]  Richard J. Linn,et al.  Conformance testing methodologies and architectures for OSI protocols , 1995 .

[52]  Denise M. Woit.  Specifying operational profiles for modules , 1993, ISSTA '93.

[53]  Atif M. Memon,et al.  Studying the Characteristics of a "Good" GUI Test Suite , 2006, 2006 17th International Symposium on Software Reliability Engineering.

[54]  Lee J. White,et al.  Event-based modelling, analysis and testing of user interactions: approach and case study: Research Articles , 2006 .

[55]  Paolo Tonella,et al.  State-Based Testing of Ajax Web Applications , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[56]  Atanas Rountev,et al.  Evaluating the imprecision of static analysis , 2004, PASTE '04.

[57]  Sung Deok Cha,et al.  Generating test sequences from a set of MSCs , 2003, Comput. Networks.

[58]  Fevzi Belli,et al.  Finite state testing and analysis of graphical user interfaces , 2001, Proceedings 12th International Symposium on Software Reliability Engineering.

[59]  Philip J. Bernhard,et al.  A reduced test suite for protocol conformance testing , 1994, TSEM.

[60]  Lee J. White,et al.  Generating test cases for GUI responsibilities using complete interaction sequences , 2000, Proceedings 11th International Symposium on Software Reliability Engineering. ISSRE 2000.

[61]  Fevzi Belli,et al.  Event‐based modelling, analysis and testing of user interactions: approach and case study , 2006, Softw. Test. Verification Reliab..

[62]  David Notkin,et al.  Tool-assisted unit-test generation and selection based on operational abstractions , 2006, Automated Software Engineering.

[63]  Barry K. Rosen.  Data Flow Analysis for Procedural Languages , 1979, JACM.

[64]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[65]  David Hovemeyer,et al.  Finding bugs is easy , 2004, SIGP.

[66]  Yi Sun,et al.  Automated generation of test programs from closed specifications of classes and test cases , 2004, Proceedings. 26th International Conference on Software Engineering.

[67]  Lionel C. Briand,et al.  Using simulation to empirically investigate test coverage criteria based on statechart , 2004, Proceedings. 26th International Conference on Software Engineering.

[68]  Atif M. Memon,et al.  Using GUI Run-Time State as Feedback to Generate Test Cases , 2007, 29th International Conference on Software Engineering (ICSE'07).

[69]  Adele E. Howe,et al.  Test Case Generation as an AI Planning Problem , 2004, Automated Software Engineering.

[70]  Bogdan Korel,et al.  The chaining approach for software test data generation , 1996, TSEM.

[71]  Margus Veanes,et al.  Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer , 2008, Formal Methods and Testing.

[72]  Mary Beth Rosson,et al.  Survey on user interface programming , 1992, CHI.