ICIS Paper: Methodological Issues in Experimental IS Research: Experiences and Recommendations

Within the last ten years the use of experimental methodology in information systems (IS) research has substantially increased. However, despite our experience with experimentation, studies continue to suffer from methodological problems. These problems have led to an accumulation of conflicting results in several areas of IS research. Moreover, future research studies will keep producing contradictory results unless researchers begin to answer questions of task and measurement validity before reporting their experimental findings. This article discusses common methodological problems in experimental IS studies and, through a description of a series of graphics experiments at the University of Minnesota, illustrates the particularly acute problem of low internal validity. Suggestions are offered to experimental IS researchers on how some of these common problems can be alleviated or even avoided, particularly in studies on the use of managerial graphics.