Scalable Model-based Robustness Testing: Novel Methodologies and Industrial Application

Metaheuristic search techniques have been extensively used to automate the process of generating test cases and thus providing solutions for a more cost-effective testing process. This approach to test automation, often coined as “Search-based Software Testing” (SBST), has been used for a wide variety of test case generation purposes. Since SBST techniques are heuristic by nature, they must be empirically investigated in terms of how costly and effective they are at reaching their test objectives and whether they scale up to realistic development artifacts. However, approaches to empirically study SBST techniques have shown wide variation in the literature. This paper presents the results of a systematic, comprehensive review that aims at characterizing how empirical studies have been designed to investigate SBST cost-effectiveness and what empirical evidence is available in the literature regarding SBST cost-effectiveness and scalability. We also provide a framework that drives the data collection process of this systematic review and can be the starting point of guidelines on how SBST techniques can be empirically assessed. The intent is to aid future researchers doing empirical studies in SBST by providing an unbiased view of the body of empirical evidence and by guiding them in performing well designed and executed empirical studies references.

[1]  Raghu Yedduladoddi Aspect Oriented Software Development , 2009 .

[2]  Robert J. Walker,et al.  An initial assessment of aspect-oriented programming , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[3]  Lionel C. Briand,et al.  A Search-Based OCL Constraint Solver for Model-Based Test Data Generation , 2011, 2011 11th International Conference on Quality Software.

[4]  Hassan Gomaa,et al.  Designing concurrent, distributed, and real-time applications with UML , 2000, ICSE.

[5]  Dianxiang Xu,et al.  A State-Based Approach to Testing Aspect-Oriented Programs , 2005, SEKE.

[6]  Doron Drusinsky,et al.  Modeling and verification using UML statecharts - a working guide to reactive system design, runtime monitoring and execution-based model checking , 2006 .

[7]  Tzilla Elrad,et al.  Aspect-Oriented Software Development , 2004 .

[8]  Rachel Harrison,et al.  An exploratory study of the effect of aspect-oriented programming on maintainability , 2008, Software Quality Journal.

[9]  Claes Wohlin,et al.  Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment , 2000, Empirical Software Engineering.

[10]  Alexander Knapp,et al.  Enhancing UML state machines with aspects , 2007, MODELS'07.

[11]  Dag I. K. Sjøberg,et al.  Evaluating the effect of a delegated versus centralized control style on the maintainability of object-oriented software , 2004, IEEE Transactions on Software Engineering.

[12]  István Majzik,et al.  Modeling and Analysis of Exception Handling by Using UML Statecharts , 2004, FIDJI.

[13]  Lionel C. Briand,et al.  Modeling robustness behavior using aspect-oriented modeling to support robustness testing of industrial systems , 2011, Software & Systems Modeling.

[14]  M A Sinclair,et al.  Questionnaire design. , 1975, Applied ergonomics.

[15]  Alexander Knapp,et al.  Executing Underspecified OCL Operation Contracts with a SAT Solver , 2008, Electron. Commun. Eur. Assoc. Softw. Sci. Technol..

[16]  Gefei Zhang,et al.  HiLA: high-level aspects for UML state machines , 2009, MODELS'09.

[17]  David J. Sheskin,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 1997 .

[18]  Lionel C. Briand,et al.  An enhanced test case selection approach for model-based testing: an industrial case study , 2010, FSE '10.

[19]  Len Thomas,et al.  Retrospective Power Analysis , 1997 .

[20]  Clémentine Nebut,et al.  Automatic test generation: a use case driven approach , 2006, IEEE Transactions on Software Engineering.

[21]  Ramnivas Laddad,et al.  Aspectj in Action: Practical Aspect-Oriented Programming , 2003 .

[22]  Jordi Cabot,et al.  Verification of UML/OCL Class Diagrams using Constraint Programming , 1899, 2008 IEEE International Conference on Software Testing Verification and Validation Workshop.

[23]  Sebastian Kleinschmager,et al.  Does aspect-oriented programming increase the development speed for crosscutting code? An empirical study , 2009, ESEM 2009.

[24]  Siobhán Clarke,et al.  Model-Driven Theme/UML , 2009, LNCS Trans. Aspect Oriented Softw. Dev..

[25]  D. Boehm-Davis,et al.  Mental representations of programs for student and professional programmers , 1987 .

[26]  Gordon Fraser,et al.  On Parameter Tuning in Search Based Software Engineering , 2011, SSBSE.

[27]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[28]  Wasif Afzal,et al.  A systematic review of search-based testing for non-functional system properties , 2009, Inf. Softw. Technol..

[29]  Gefei Zhang Towards Aspect-Oriented State Machines Gefei , 2006 .

[30]  Thomas Weigert,et al.  Specifying Telecommunications Systems with UML , 2003, UML for Real.

[31]  Bran Selic,et al.  Modeling and Analysis of Real-Time and Embedded Systems , 2005, MoDELS.

[32]  João Araújo,et al.  An expressive aspect composition language for UML state diagrams , 2007, MODELS'07.

[33]  Tom Pender UML Bible , 2003 .

[34]  Bin Lei,et al.  State Based Robustness Testing for Components , 2010, Electron. Notes Theor. Comput. Sci..

[35]  Janice Singer,et al.  Guide to Advanced Empirical Software Engineering , 2007 .

[36]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[37]  Robert V. Binder,et al.  Testing Object-Oriented Systems: Models, Patterns, and Tools , 1999 .

[38]  Mehmet Aksit,et al.  A Controlled Experiment for the Assessment of Aspects - Tracing in an Industrial Context , 2008 .