Design Science Research Evaluation

The consensus view is that the rigorous evaluation of design science (DS) artifacts is essential. There are many types of DS artifacts and many forms of evaluation; what is missing is guidance for how to perform the evaluation, more specifically, what evaluation methods to use with specific DS research outputs. Here we find and review 148 DS research articles published in a selected set of information systems (IS), computer science (CS) and engineering journals. We analyze the articles to develop taxonomies of DS artifact types and artifact evaluation methods; we apply these taxonomies to determine which evaluation methods are associated in the literature with particular artifacts. We show that there are several popular "artifact - evaluation method" combinations in the literature. The results inform DS researchers of usual and customary combinations of research artifacts and evaluation methods, potentially providing them with rationale and justification for an evaluation method selection.

[1]  Jay F. Nunamaker,et al.  Systems Development in Information Systems Research , 1990, J. Manag. Inf. Syst..

[2]  Sandeep Purao,et al.  Action Design Research , 2011, MIS Q..

[3]  James R. Marsden,et al.  A Query-Driven Approach to the Design and Management of Flexible Database Systems , 2002, J. Manag. Inf. Syst..

[4]  Anne Cleven,et al.  Design alternatives for the evaluation of design science research artifacts , 2009, DESRIST.

[5]  N.F.M. Roozenburg,et al.  A methodological comparison of the structures of scientific research and engineering design: their similarities and differences , 1991 .

[6]  Richard Baskerville,et al.  Diversity in information systems action research methods , 1998 .

[7]  Charles A. Snyder,et al.  Technology and the Decentralization of Information Systems , 2003, Inf. Syst. Manag..

[8]  Samir Chatterjee,et al.  A Design Science Research Methodology for Information Systems Research , 2008 .

[9]  Sandeep Purao,et al.  Proceedings of the 4th International Conference on Design Science Research in Information Systems and Technology , 2009 .

[10]  Tuure Tuunanen,et al.  A Contigency Model for Requirements Development , 2007, J. Assoc. Inf. Syst..

[11]  Omar El Sawy,et al.  Building an Information System Design Theory for Vigilant EIS , 1992, Inf. Syst. Res..

[12]  G. Kelly.  The Psychology of Personal Constructs , 2020 .

[13]  Omar A. El Sawy,et al.  Assessing Information System Design Theory in Perspective: How Useful Was our 1992 Initial Rendition? , 2004 .

[14]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[15]  Pradeep Kumar Ray,et al.  Designing an evaluation framework for IT service management , 2010, Inf. Manag..

[16]  Paul Veerkamp,et al.  Modeling Design Process , 1990, AI Mag..

[17]  Tuure Tuunanen,et al.  Extending Critical Success Factors Methodology to Facilitate Broadly Participative Information Systems Planning , 2003, J. Manag. Inf. Syst..