Industrial Case Studies for Evaluating Search Based Structural Testing

Evolutionary structural testing has been researched and promising results have been presented. However, it has hardly been applied to real-world complex systems and as such, little is known about the scalability, applicability and acceptability of it in an industrial setting. The European project EvoTest (IST-33472) team has been working from 2006 till 2009 to improve this situation and this paper informs about the results. We start with an overview of tools and techniques which we have developed for automated evolutionary structural testing. Subsequently, we describe the empirical setup used to study the applicability of evolutionary structural testing in industry through two case studies. The test objects used for the studies are selected functions (handwritten and generated) from production systems at Daimler and Berner & Mattner Systemtechnik (BMS) like, for example, Rear Window Defroster, Global Powertrain Engine Controller, Window Lift Control System, etc. The results of the case studies are described and research questions are assessed based on the obtained results. In summary, the results indicate that evolutionary structural testing in an industrial setting is worthwhile and profitable. Hardly any detailed knowledge of evolutionary computation is required to search for interesting test data. The case studies also research the benefits of using techniques like automated parameter tuning and search space smoothing.

[1]  Tanja E. J. Vos,et al.  Evolutionary White-Box Software Test with the EvoTest Framework: A Progress Report , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[2]  Jacob Cohen Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[3]  Susan Elliott Sim,et al.  Case studies for software engineers , 2006, ICSE '06.

[4]  Andrea Arcuri,et al.  It Does Matter How You Normalise the Branch Distance in Search Based Software Testing , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[5]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.

[6]  Thierry Rolina Past, Present, and Future of Real-Time Embedded Automotive Software: A Close Look at Basic Concepts of AUTOSAR , 2006 .

[7]  Joachim Wegener,et al.  Automatic Testing of an Autonomous Parking System Using Evolutionary Computation , 2004 .

[8]  Matthias Grochtmann,et al.  Systematic Testing of Real-Time Systems , 2000 .

[9]  Adam A. Porter,et al.  Empirical studies of software engineering: a roadmap , 2000, ICSE '00.

[10]  Mark Harman,et al.  Evolutionary testing in the presence of loop-assigned flags: a testability transformation approach , 2004, ISSTA '04.

[11]  Tanja E. J. Vos,et al.  Industrial Scaled Automated Structural Testing with the Evolutionary Testing Tool , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[12]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..

[13]  Marc Schoenauer,et al.  Bringing evolutionary computation to industrial applications with guide , 2009, GECCO.

[14]  Joachim Wegener,et al.  Automatic Test Data Generation For Structural Testing Of Embedded Software Systems By Evolutionary Testing , 2002, GECCO.

[15]  Marc Roper Computer aided software testing using genetic algorithms , 1997 .

[16]  Mark Harman,et al.  The impact of input domain reduction on search-based test data generation , 2007, ESEC-FSE '07.

[17]  Mark Harman,et al.  Testability transformation , 2004, IEEE Transactions on Software Engineering.

[18]  Natalia Juristo Juzgado,et al.  Reviewing 25 Years of Testing Technique Experiments , 2004, Empirical Software Engineering.

[19]  Mark Harman,et al.  A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search , 2010, IEEE Transactions on Software Engineering.

[20]  Dorothy Graham,et al.  Software test automation: effective use of test execution tools , 1999 .

[21]  Janice Singer,et al.  Studying Software Engineers: Data Collection Techniques for Software Field Studies , 2005, Empirical Software Engineering.

[22]  Forrest Shull,et al.  Building Knowledge through Families of Experiments , 1999, IEEE Trans. Software Eng..

[23]  Zbigniew Michalewicz,et al.  Parameter Setting in Evolutionary Algorithms , 2007, Studies in Computational Intelligence.

[24]  Mark Harman,et al.  Empirical evaluation of a nesting testability transformation for evolutionary testing , 2009, TSEM.

[25]  André Baresel,et al.  Structural and Functional Sequence Test of Dynamic and State-Based Software with Evolutionary Algorithms , 2003, GECCO.

[26]  Zbigniew Michalewicz,et al.  Parameter Control in Evolutionary Algorithms , 2007, Parameter Setting in Evolutionary Algorithms.

[27]  Mark Harman,et al.  A multi-objective approach to search-based test data generation , 2007, GECCO '07.

[28]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[29]  Mark Harman,et al.  Handling dynamic data structures in search based testing , 2008, GECCO '08.

[30]  John A. Clark,et al.  Multi-objective Improvement of Software Using Co-evolution and Smart Seeding , 2008, SEAL.

[31]  L. C. Briand A Critical Analysis of Empirical Research in Software Testing , 2007, ESEM 2007.

[32]  R. Bell,et al.  IEC 61508: functional safety of electrical/electronic/ programme electronic safety-related systems: overview , 1999 .

[33]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..