Experimental Evaluation of a Simulation Environment for Information Systems Design

Abstract This article presents an experiment assessing the decision support value of a simulation environment for the information systems (IS) design process. We have implemented a prototype simulation environment that uses data flow diagrams (DFDs) augmented with the performance rates of system components to specify the structure and dynamics of IS designs. The DFD-based representation is automatically mapped to a stochastic queuing network simulation. Knowledge-based help supports formulation of simulation run parameters and interpretation of output. We measure the prototype's impact on system dynamics assessment via the accuracy of IS professionals' responses to questions about the dynamics of four IS cases. The prototype's simulation capability has a significant positive effect on accuracy scores for questions involving waiting times, system times, and queue lengths of jobs or customers. Subjects choosing to conduct more and longer simulation runs provide significantly more accurate assessments than less-active users of simulation. The findings suggest that IS professionals can make use of simulation technology within a compact time frame if the proper supports are provided; on-line help or other methods should be used to encourage users to conduct sufficiently long simulation runs; and embedding the prototype's capabilities in a computer-aided software engineering workbench would result in better designed information systems.