ICST 2008 Special Issue

This special issue contains extended versions of three papers from the first IEEE International Conference on Software Testing Verification and Validation (ICST 2008). These three papers were selected based on reviews from members of the program committee and subsequently subjected to another round of review and revision. All three papers focus on automated test generation and include significant empirical evaluation. However, they apply very different approaches to automation: model based, random, and white-box. The first paper by Ciupa, Pretschner, Oriol, Leitner, and Meyer offers an empirical investigation of a variant of random testing applied to object-oriented software. The nature of object-oriented systems complicates random testing since there is a need to introduce objects as well as to randomly generate input. The authors applied an approach in which an object was produced using a constructor call followed by routines that changed the object’s state. This has the advantage of only generating valid objects. The (Eiffel) software to which random testing was applied contained contracts and these provided the oracle. The authors performed multiple runs of the same duration and examined the faults found. Interestingly, they found that the actual number of faults found did not vary too much but that different runs found different faults. The authors also classified faults and investigated the faults found by manual testing, random testing, and those in user incident reports. They discovered that these processes found different types of faults and hence are complementary. The second paper, by Bardin andHerrmann, investigates automated testing based onmachine code rather than, for example, source code. One of the main motivations for this work is that often source code isnotavailable,especially ifpartsofdevelopmenthavebeenoutsourced.Anadditionalmotivation is that the machine code is closer, than the source code, to the physical system that actually runs. The authors observe thatmachine code ismore difficult to analyse than source code in structured programming languages since, for example, the controlflow isnot explicit.However, they show that path-based approaches can be adapted and describe a tool (OSMOSE) that achieves this. The approach essentially can be seen as finding the path predicate (precondition) for a suitable path through the machine code and then solving this to find test input. The tool combines static and dynamic analysis in a manner inspired by concolic execution. Finally, they report on the results of experiments using OSMOSE. Our final paper by Grieskamp, Kicillof, Stobie, and Braberman discusses the development of interoperability documentation for Windows protocols within Microsoft and the use of test-driven approaches to verify that the documents accurately represent the protocols. The approaches used not only include manual testing but also a significant amount of model-based testing (MBT) using Microsoft’s tool Spec Explorer. Vendors in India and China carried out the testing. A major focus of the paper is the use of MBT in this project. There are two main positive messages regarding MBT. First, it was found that the testers were able to apply MBT after an initial training course. Second, MBT was found to scale and to be significantly more efficient than manual testing. On a final note, we would like to express our gratitude to the many able and hard working people who were involved in the organization of ICST 2008 and this special issue. The steering committee members provided valuable advice and we were assisted by industry chairs Robert Eschbach, Per Runeson, and Clay Williams, student papers chairs Henry Muccini and Tao Xie, and by the general chair Lionel Briand. The success of ICST 2008 and also this special issue is testament to the efforts of these individuals as well as those who reviewed the papers for ICST 2008 and the special issue in a thorough and timely manner. We are also indebted to the ICST 2008 publicity chairs Harita Bhaskar, Yvan Labiche, Yves Le Traon, and T.H. Tse and the web chair Sudipto Ghosh. Finally, we thank all the authors who spent valuable time in preparing papers for ICST 2008 and this special issue.