Artifact Evaluation in Information Systems Design-Science Research - a Holistic View

Design science in Information Systems (IS) research pertains to the creation of artifacts to solve reallife problems. Research on IS artifact evaluation remains at an early stage. In the design-science research literature, evaluation criteria are presented in a fragmented or incomplete manner. This paper addresses the following research questions: which criteria are proposed in the literature to evaluate IS artifacts? Which ones are actually used in published research? How can we structure these criteria? Finally, which evaluation methods emerge as generic means to assess IS artifacts? The artifact resulting from our research comprises three main components: a hierarchy of evaluation criteria for IS artifacts organized according to the dimensions of a system (goal, environment, structure, activity, and evolution), a model providing a high-level abstraction of evaluation methods, and finally, a set of generic evaluation methods which are instantiations of this model. These methods result from an inductive study of twenty-six recently published papers.

[1]  Kam-Fai Wong,et al.  Web 2.0 Environmental Scanning and Adaptive Decision Support for Business Mergers and Acquisitions , 2012, MIS Q..

[2]  Guy G. Gable,et al.  The Design Science Research Roadmap: In Progress Evaluation , 2013, PACIS.

[3]  Kyong-Ho Lee,et al.  The Impact of the Human Element in the Information Systems Quality for Decision Making and User Satisfaction , 2016 .

[4]  Jan vom Brocke,et al.  Evaluations in the Science of the Artificial - Reconsidering the Build-Evaluate Pattern in Design Science Research , 2012, DESRIST.

[5]  Jan Pries-Heje,et al.  A Comprehensive Framework for Evaluation in Design Science Research 1 , 2022 .

[6]  Shouhong Wang,et al.  Ontology-Based Data Summarization Engine: A Design Methodology , 2012, J. Comput. Inf. Syst..

[7]  Sheng-Tun Li,et al.  Design and Evaluation of a Layered Thematic Knowledge Map System , 2008, J. Comput. Inf. Syst..

[8]  Sabine Matook,et al.  Conceptualizing the IT Artifact for MIS Research , 2008, ICIS.

[9]  Keng Siau,et al.  Evaluation techniques for systems analysis and design modelling methods – a review and comparative analysis , 2011, Inf. Syst. J..

[10]  Shirley Gregor,et al.  Designing for Mutability in Information Systems Artifacts , 2007 .

[11]  Ivo Vondrák,et al.  Business Process Modeling , 2007, Encyclopedia of Database Systems.

[12]  Anne Cleven,et al.  Design alternatives for the evaluation of design science research artifacts , 2009, DESRIST.

[13]  Anindya Datta,et al.  A Cost-Based Database Request Distribution Technique for Online e-Commerce Applications , 2012, MIS Q..

[14]  Hongwei Chris Yang,et al.  Bon Appétit for Apps: Young American Consumers' Acceptance of Mobile Applications , 2013, J. Comput. Inf. Syst..

[15]  Jan Pries-Heje,et al.  Strategies for Design Science Research Evaluation , 2008, ECIS.

[16]  Christian Fischer,et al.  The Information Systems Design Science Research Body Of Knowledge - A Citation Analysis In Recent Top-Journal Publications , 2011, PACIS.

[17]  Wm. Arthur Conklin,et al.  Re-examining the Information Systems Security Problem from a Systems Theory Perspective , 2010, AMCIS.

[18]  Andreas Abecker,et al.  Managing Kowledge at Multiple Organizational Levels Using Faceted Ontologies , 2008, J. Comput. Inf. Syst..

[19]  Salvatore T. March,et al.  Design and natural science research on information technology , 1995, Decis. Support Syst..

[20]  Herbert A. Simon,et al.  The Sciences of the Artificial - 3rd Edition , 1981 .

[21]  Hsinchun Chen,et al.  CyberGate: A Design Framework and System for Text Analysis of Computer-Mediated Communication , 2008, MIS Q..

[22]  Pei-Lee Teh,et al.  Knowledge Sharing in is Personnel: Organizational Behavior's Perspective , 2011, J. Comput. Inf. Syst..

[23]  Shi-Ming Huang,et al.  Developing A Continuous Auditing Assistance System Based on Information Process Models , 2007, J. Comput. Inf. Syst..

[24]  Daniel L. Sherrell,et al.  Communications of the Association for Information Systems , 1999 .

[25]  Sheng-Tun Li,et al.  Concept-Guided Query Expansion for Knowledge Management with Semi-Automatic Knowledge Capturing , 2009, J. Comput. Inf. Syst..

[26]  Jan Muntermann,et al.  A method for taxonomy development and its application in information systems , 2013, Eur. J. Inf. Syst..

[27]  Param Vir Singh,et al.  A Hidden Markov Model for Collaborative Filtering , 2010, MIS Q..

[28]  Shirley Gregor,et al.  The Anatomy of a Design Theory , 2007, J. Assoc. Inf. Syst..

[29]  Yair Wand,et al.  Using Cognitive Principles to Guide Classification in Information Systems Modeling , 2008, MIS Q..

[30]  Alan R. Hevner,et al.  POSITIONING AND PRESENTING DESIGN SCIENCE RESEARCH FOR MAXIMUM IMPACT 1 , 2013 .

[31]  Les Gasser,et al.  A Design Theory for Systems That Support Emergent Knowledge Processes , 2002, MIS Q..

[32]  Tuure Tuunanen,et al.  Design Science Research Evaluation , 2012, DESRIST.

[33]  Jean-Louis Le Moigne.  Modeling For Reasoning Socio-Economic Behaviors , 2006, Cybern. Hum. Knowing.

[34]  Dongsong Zhang,et al.  The Effects of Tree-View Based Presentation Adaptation on Model Web Browsing , 2011, MIS Q..

[35]  Shouhong Wang,et al.  Towards Innovative Design Research in Information Systems , 2010, J. Comput. Inf. Syst..

[36]  Robert J. Kauffman,et al.  Making Sense of Technology Trends in the Information Technology Landscape: A Design Science Approach , 2008, MIS Q..

[37]  Jeffrey J. Johnson,et al.  A Rapid Mapping Conversion Methodology to a Commercial-off-the-Shelf System , 2010, J. Comput. Inf. Syst..

[38]  Lars Skyttner,et al.  General Systems Theory: Perspectives, Problems, Practice , 2006 .

[39]  Anurag Agarwal,et al.  Scheduling Online Advertisements to Maximize Revenue under Non-Linear Pricing , 2012, J. Comput. Inf. Syst..

[40]  John R. Venable,et al.  Design Science Research Post Hevner et al.: Criteria, Standards, Guidelines, and Expectations , 2010, DESRIST.

[41]  Detmar W. Straub,et al.  Validation Guidelines for IS Positivist Research , 2004, Commun. Assoc. Inf. Syst..

[42]  Robert Winter,et al.  Design science research in Europe , 2008 .

[43]  Kyong-Ho Lee,et al.  A Sophisticated Approach to Semantic Web Services Discovery , 2008, J. Comput. Inf. Syst..

[44]  Jiang-Liang Hou,et al.  A Model for Document Validation Using Concurrent Authentication Processes , 2008, J. Comput. Inf. Syst..

[45]  David R. Firth,et al.  Communications of the Association for Information Systems , 2011 .

[46]  Pertti Järvinen.  On Reviewing of Results in Design Research , 2007, ECIS.

[47]  Magali Roux-Rouquié,et al.  The systemic paradigm and its relevance to the modelling of biological functions. , 2002, Comptes rendus biologies.

[48]  Shirley Gregor,et al.  Information Systems Foundations: The Role of Design Science , 2011 .

[49]  L. Bertalanffy.  General system theory : foundations, development, applications , 1977 .

[50]  Jan Pries-Heje,et al.  The Design Theory Nexus , 2008, MIS Q..

[51]  Omar El Sawy,et al.  Building an Information System Design Theory for Vigilant EIS , 1992, Inf. Syst. Res..

[52]  Stephan Aier,et al.  Criteria of progress for information systems design theories , 2011, Inf. Syst. E Bus. Manag..

[53]  Yolande E. Chan,et al.  A Multilevel Model for Measuring Fit Between a Firm's Competitive Strategies and Information Systems Capabilities , 2011, MIS Q..

[54]  Ken Orr,et al.  Data quality and systems theory , 1998, CACM.

[55]  Ahmed Abbasi,et al.  MetaFraud: A Meta-Learning Framework for Detecting Financial Fraud , 2012, MIS Q..

[56]  J. Lopreato,et al.  General system theory : foundations, development, applications , 1970 .

[57]  R. Hirschheim,et al.  Analysing information systems evaluation: another look at an old problem , 1998 .

[58]  Michael D. Myers,et al.  A set of ethical principles for design science research in information systems , 2014, Inf. Manag..

[59]  Ron Weber,et al.  On the ontological expressiveness of information systems analysis and design grammars , 1993, Inf. Syst. J..

[60]  Sumali Conlon,et al.  A Rule-Based System to Extract Financial Information , 2012, J. Comput. Inf. Syst..

[61]  Jun-Woo Kim,et al.  Fuzzy-Semantic Information Management System for Dispersed Information , 2011, J. Comput. Inf. Syst..

[62]  Herbert A. Simon,et al.  The Sciences of the Artificial , 1970 .

[63]  Yair Wand,et al.  Guidelines for Designing Visual Ontologies to Support Knowledge Identification , 2011, MIS Q..

[64]  Alan R. Hevner,et al.  Design Science Research , 2014, Computing Handbook, 3rd ed..

[65]  Brian T. Pentland,et al.  Process Grammar as a Tool for Business Process Design , 2008, MIS Q..

[66]  Shirley Gregor,et al.  Building theory in a practical science , 2010 .

[67]  Jay F. Nunamaker,et al.  Detecting Fake Websites: The Contribution of Statistical Learning Theory , 2010, MIS Q..

[68]  Martin Bichler,et al.  Design science in information systems research , 2006, Wirtschaftsinf..

[69]  Larry Laudan,et al.  Science and Values , 1986 .

[70]  Samir Chatterjee,et al.  A Design Science Research Methodology for Information Systems Research , 2008 .

[71]  Richard Lai,et al.  Modeling Business Goal for Business/it Alignment Using Requirements Engineering , 2011, J. Comput. Inf. Syst..

[72]  Marta Indulska,et al.  Business Process Modeling- A Comparative Analysis , 2009, J. Assoc. Inf. Syst..

[73]  José Antônio Valle Antunes,et al.  Design Science Research , 2015 .

[74]  Yong Seog Kim,et al.  Multi-Objective Clustering with Data - and Human-Driven Metrics , 2011, J. Comput. Inf. Syst..

[75]  M. Meehan.  General System Theory: Foundations, Development, Applications , 1969 .