Using design of experiments, sensitivity analysis, and hybrid simulation to evaluate changes to a software development process: a case study

Hybrid simulation models combine the high-level project issues of System Dynamics models with the process detail of discrete event simulation models. Hybrid models not only capture the best of both of these simulation paradigms, but they also are able to address new issues that are important in managing complex real-world development projects that neither the System Dynamics nor Discrete Event simulation paradigms are able to address alone. In order to reap the full benefits from a simulation model, a structured approach for analyzing model results is necessary. This article applies Design of Experiments (DOE) and broad range sensitivity analysis (BRSA) to a hybrid system dynamics and discrete event simulation model of a software development process. DOE is used to analyse the interaction effects, such as the degree to which the impact of the process change depends on worker motivation, schedule pressure and other project environmental variables. The sensitivity of the model to parameter changes over a broad range of plausible values is used to analyse the non-linear aspects of the model. The end result is a deeper insight into the conditions under which the process change will succeed, and improved recommendations for process change design and implementation. In this particular study, significant interactions and non-linearities were revealed, supporting the hypothesis that consideration of these complex effects is essential for insightful interpretation of model results and effective decision-making. Copyright © 2004 John Wiley & Sons, Ltd.