(Quasi-)Experimental Studies in Industrial Settings

Software engineering research primarily deals with technologies that promise software organisations to deliver high quality products on time and within budget. However, in many cases researchers do not investigate the validity of the promises and, therefore, information about the relative strengths and weaknesses of those technologies in comparison with the already existing ones is often missing. Although experimentation and experimental software engineering have been suggested to address this issue and significant progress has been made throughout the last couple of years in this area, there is still a lack of experimental work in industrial settings. The reasons for this poor situation range from practical constraints, such as, the costs associated with a study and the benefits for a single company, to more methodological ones, such as the level of control that can be imposed on the different treatment conditions in an industrial setting. In this chapter we present an practical approach that helps overcome most of the objections. The approach represents a balance between the benefits for practitioners and methodological rigor. In essence, it uses training situations to set up and run empirical studies. While this procedure disqualifies the study as a pure controlled experiment, the characteristics of a quasi-experiment can often be preserved. The chapter explains the principle of the approach, differences between controlled experiments and quasi-experiments and finally, presents an example of a quasi-experiment in an industrial setting.