An Initial Characterization of Industrial Graphical User Interface Systems

To date we have developed and applied numerous model-based GUI testing techniques; however, we are unable to provide definitive improvement schemes to real-world GUI test planners, as our data was derived from open source applications, small compared to industrial systems. This paper presents a study of three industrial GUI-based software systems developed at ABB, including data on classified defects detected during late-phase testing and customer usage, test suites, and source code change metrics. The results show that (1) 50% of the defects found through the GUI are categorized as data access and handling, control flow and sequencing, correctness, and processing defects, (2) system crashes exposed defects 12-19% of the time, and (3) GUI and non-GUI components are constructed differently, in terms of source code metrics.

[1]  Elaine J. Weyuker,et al.  Using operational distributions to judge testing progress , 2003, SAC '03.

[2]  Brian Robinson,et al.  A defect-driven process for software quality improvement , 2008, ESEM '08.

[3]  Bruce Elliott,et al.  Testing Computer Software , 1994 .

[4]  V. Basili.  Software modeling and measurement: the Goal/Question/Metric paradigm , 1992 .

[5]  Robert V. Binder,et al.  Testing Object-Oriented Systems: Models, Patterns, and Tools , 1999 .

[6]  Jim Clarke.  Automated test generation from a behavioral model , 1998 .

[7]  James A. Whittaker,et al.  A Markov Chain Model for Statistical Software Testing , 1994, IEEE Trans. Software Eng..

[8]  Man F. Lau,et al.  An extended fault class hierarchy for specification-based testing , 2005, TSEM.

[9]  Donald E. Knuth,et al.  The errors of tex , 1989, Softw. Pract. Exp..

[10]  Atif M. Memon,et al.  Automated gui testing guided by usage profiles , 2007, ASE.

[11]  Wei-Tek Tsai,et al.  An experimental study of fault detection in user requirements documents , 1992, TSEM.

[12]  Alfred Bröckers,et al.  The role of software process modeling in planning industrial measurement programs , 1996, Proceedings of the 3rd International Software Metrics Symposium.

[13]  Sigrid Eldh.  Software Testing Techniques , 2007 .

[14]  Elaine J. Weyuker,et al.  Collecting and categorizing software error data in an industrial environment , 2018, J. Syst. Softw..

[15]  Alexander Pretschner,et al.  Model-Based Testing in Practice , 2005, FM.

[16]  Cem Kaner,et al.  Testing computer software (2. ed.) , 1993 .

[17]  Atif M. Memon,et al.  Using GUI Run-Time State as Feedback to Generate Test Cases , 2007, 29th International Conference on Software Engineering (ICSE'07).

[18]  Brad A. Myers,et al.  User interface software tools , 1995, TCHI.

[19]  Tatsuhiro Tsuchiya,et al.  On fault classes and error detection capability of specification-based testing , 2002, TSEM.

[20]  Atif M. Memon,et al.  Relationships between Test Suites, Faults, and Fault Detection in GUI Testing , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[21]  Ray Turner.  Software engineering methodology , 1984 .

[22]  Atif M. Memon,et al.  Designing and comparing automated test oracles for GUI-based software applications , 2007, TSEM.

[23]  Mary Lou Soffa,et al.  Coverage criteria for GUI testing , 2001, ESEC/FSE-9.

[24]  Ram Chillarege,et al.  Orthogonal defect classification , 1996 .

[25]  Siddhartha R. Dalal,et al.  Model-based testing in practice , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[26]  Victor R. Basili,et al.  Identifying domain-specific defect classes using inspections and change history , 2006, ISESE '06.

[27]  Atif M. Memon,et al.  Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software , 2005, IEEE Transactions on Software Engineering.

[28]  Atif M. Memon,et al.  GUI ripping: reverse engineering of graphical user interfaces for testing , 2003, 10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings..

[29]  Mark Butcher,et al.  Improving software testing via ODC: Three case studies , 2002, IBM Syst. J..

[30]  Inderpal S. Bhandari,et al.  Orthogonal Defect Classification - A Concept for In-Process Measurements , 1992, IEEE Trans. Software Eng..

[31]  Mary Lou Soffa,et al.  Using a goal-driven approach to generate test cases for GUIs , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[32]  Norman F. Schneidewind,et al.  An Experiment in Software Error Data Collection and Analysis , 1979, IEEE Transactions on Software Engineering.