Towards an understanding of how and why Design Science Research scholars evaluate

Evaluation activities in Design Science Research not only verify utility but also scientific rigour and the truth-like value of prescriptive knowledge contributions. Assuming a lack of guidance, evaluation frameworks like the one of Sonnenberg and vom Brocke have been proposed prior to evaluating their actual utility to scholars. This research now aims at evaluating how scholars actually apply the framework in practice and their reasoning for doing so. The research-in-progress paper at hand presents preliminary results from a citation analysis. While we find an increasing awareness of the relevance of ex-ante evaluations, there is still a lack of comprehensive detail on ex-ante evaluation activities and an emphasis on ex-post evaluations in artificial settings. The call to accumulate incremental prescriptive knowledge has mostly been ignored and artifact changes are rarely disclosed. Therefore, we question the missing guidance as the sole reason that scholars emphasise building rather than evaluation. We propose to conduct a case study that investigates the reasons behind scholars’ evaluation decisions.

[1]  Dirk Metzger,et al.  Designing mHealth Applications for Developing Countries , 2016, ECIS.

[2]  Samir Chatterjee,et al.  A Design Science Research Methodology for Information Systems Research , 2008 .

[3]  Jan vom Brocke,et al.  Living IT infrastructures - An ontology-based approach to aligning IT infrastructure capacity and business needs , 2014, Int. J. Account. Inf. Syst..

[4]  Alan R. Hevner,et al.  The Three Cycle View of Design Science , 2007, Scand. J. Inf. Syst..

[5]  Patrick Afflerbach,et al.  An economic decision model for determining the appropriate level of business process standardization , 2016 .

[6]  Manuel Bolsinger,et al.  Bringing value-based business process management to the operational process level , 2015, Inf. Syst. E Bus. Manag..

[7]  Dirk Metzger,et al.  Shaping the future of mobile service support systems - ex-ante evaluation of smart glasses in technical customer service processes , 2016, GI-Jahrestagung.

[8]  Sandeep Purao,et al.  Action Design Research , 2011, MIS Q..

[9]  Gerhard Schwabe,et al.  Mobiles Lernen für Industrie 4.0: Probleme, Ziele, Lernarrangements , 2017, Wirtschaftsinformatik.

[10]  Tony Cornford,et al.  A critical approach to evaluation , 2005, Eur. J. Inf. Syst..

[11]  Susanne Leist,et al.  Applying Evaluations While Building the Artifact -- Experiences from the Development of Process Model Complexity Metrics , 2016, 2016 49th Hawaii International Conference on System Sciences (HICSS).

[12]  Stephan Aier,et al.  Design Principles for Business-Model-based Management Methods - A Service-Dominant Logic Perspective , 2017, DESRIST.

[13]  Boris Otto,et al.  Proposing a Capability Perspective on Digital Business Models , 2015, 2015 IEEE 17th Conference on Business Informatics.

[14]  Alan R. Hevner,et al.  POSITIONING AND PRESENTING DESIGN SCIENCE RESEARCH FOR MAXIMUM IMPACT 1 , 2013 .

[15]  Martin Lehnert,et al.  Prioritization of Interconnected Processes , 2018, Bus. Inf. Syst. Eng..

[16]  Dirk Metzger,et al.  Sind Smart Glasses die Zukunft der Digitalisierung von Arbeitsprozessen? Explorative Fallstudien zukünftiger Einsatzszenarien in der Logistik , 2017, Wirtschaftsinformatik.

[17]  Jörg Becker,et al.  Designing a Framework for the Development of Domain-Specific Process Modelling Languages , 2017, DESRIST.

[18]  Stefano Pezzuto,et al.  This is an Open Access article distributed under the terms of the Creative Commons Attribution-Noncommercial License, which permits unrestricted use, distribution, and reproduction in any noncommercial medium, provided the original work is properly cited. , 2009 .

[19]  Ralf Abraham,et al.  Development of Design Principles for Boundary Objects in Enterprise Transformation , 2015 .

[20]  Tuure Tuunanen,et al.  Continuous Requirements Risk Profiling in Information Systems Development , 2015, 2015 48th Hawaii International Conference on System Sciences.

[21]  Michael D. Myers,et al.  A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems , 1999, MIS Q..

[22]  Constantinos J. Stefanou,et al.  A framework for the ex-ante evaluation of ERP software , 2001, Eur. J. Inf. Syst..

[23]  Hans-Georg Fill,et al.  Meta Modeling for Business Process Improvement , 2017, Bus. Inf. Syst. Eng..

[24]  Jan Pries-Heje,et al.  Strategies for Design Science Research Evaluation , 2008, ECIS.

[25]  Jan vom Brocke,et al.  The missing link between BPM and accounting: Using event data for accounting in process-oriented organizations , 2014, Bus. Process. Manag. J..

[26]  Christian Leyh,et al.  Enabling Business Domain-Specific e-Collaboration: Developing Artifacts to Integrate e-Collaboration into Product Costing , 2017, DESRIST.

[27]  Martin Bichler,et al.  Design science in information systems research , 2006, Wirtschaftsinf..

[28]  Salvatore T. March,et al.  Design and natural science research on information technology , 1995, Decis. Support Syst..

[29]  Jan Pries-Heje,et al.  FEDS: a Framework for Evaluation in Design Science Research , 2016, Eur. J. Inf. Syst..

[30]  Walter Brenner,et al.  Digital Product Innovation in Manufacturing Industries - Towards a Taxonomy for Feedback-driven Product Development Scenarios , 2017, HICSS.

[31]  Daniel Miehle,et al.  Development of Dynamic Key Figures for the identification of Critical Components in Smart Factory Information Networks , 2017, ECIS.

[32]  Olivera Marjanovic,et al.  Designing Innovative Education through Action Design Research: Method and Application for Teaching Design Activities in Large Lecture Environments , 2016, J. Inf. Technol. Theory Appl..

[33]  Robert Winter,et al.  Fail Early, Fail Often: Towards Coherent Feedback Loops in Design Science Research Evaluation , 2014, ICIS.

[34]  Cornè J van Staden,et al.  Using a user experience evaluation framework for eModeration , 2017, 2017 Conference on Information Communication Technology and Society (ICTAS).

[35]  Jan vom Brocke,et al.  Evaluations in the Science of the Artificial - Reconsidering the Build-Evaluate Pattern in Design Science Research , 2012, DESRIST.

[36]  Stefan Feuerriegel,et al.  Design Science in Practice: Designing an Electricity Demand Response System , 2013, DESRIST.