Software-Embedded Evaluation Support in Design Science Research

Even though the practice of integrating evaluative features into software has long been applied in commercially available software, it is still underrepresented in the Information Systems (IS) community. This paper presents a framework for embedded evaluation support. We are aware of the challenges of evaluation of socio-technical systems and take this issue into consideration in our framework. Our framework is the result of conceptualizations drawing from the evaluation discourse discussion on the topics of artifact evaluation in DSR. We also demonstrate our ideas through two examples of embedded evaluation support mechanisms designed and used in a DSR project in the Swedish healthcare sector.

[1]  Claire Le Goues,et al.  Current challenges in automatic software repair , 2013, Software Quality Journal.

[2]  Fifteen Challenges in Establishing a Multidisciplinary Research Program on eHealth Research in a University Setting: A Case Study , 2017, Journal of medical Internet research.

[3]  Albrecht Schmidt,et al.  Knowing the User's Every Move – User Activity Tracking for Website Usability Evaluation and Implicit Interaction , 2006 .

[4]  Jon Whittle,et al.  Prototyping 'clasp': implications for designing digital technology for and with adults with autism , 2014, Conference on Designing Interactive Systems.

[5]  Juhani Iivari,et al.  Distinguishing and contrasting two strategies for design science research , 2015, Eur. J. Inf. Syst..

[6]  L. von Essen,et al.  The Origin and Impact of Ideals in eHealth Research: Experiences From the U-CARE Research Environment , 2014, JMIR research protocols.

[7]  H. T. Nagle,et al.  Design for testability and built-in self test: a review , 1989 .

[8]  Raimund Ubar,et al.  A hybrid BIST architecture and its optimization for SoC testing , 2002, Proceedings International Symposium on Quality Electronic Design.

[9]  Sandeep Purao,et al.  Action Design Research , 2011, MIS Q..

[10]  D. Stufflebeam.  The CIPP Model for Evaluation , 2000 .

[11]  Jonas Sjöström,et al.  In-Place Translation in Information Systems Development , 2012 .

[12]  Jan Pries-Heje,et al.  FEDS: a Framework for Evaluation in Design Science Research , 2016, Eur. J. Inf. Syst..

[13]  Veda C. Storey,et al.  Genres of Inquiry in Design-Science Research: Justification and Evaluation of Knowledge Production , 2015, MIS Q..

[14]  Angelos D. Keromytis,et al.  ASSURE: automatic software self-healing using rescue points , 2009, ASPLOS.

[15]  Rudy Hirschheim,et al.  Analysing information systems evaluation: another look at an old problem , 1998 .

[16]  Pär J. Ågerfalk,et al.  Sowing the seeds of self: a socio-pragmatic penetration of the web artefact , 2007, ICPW '07.

[17]  Pär J. Ågerfalk,et al.  Respondent Behavior Logging: An Opportunity for Online Survey Design , 2013, DESRIST.

[18]  Alan R. Hevner,et al.  Scrutinizing Privacy and Accountability in Online Psychosocial Care , 2017, IT Professional.

[19]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[20]  Claire Le Goues,et al.  GenProg: A Generic Method for Automatic Software Repair , 2012, IEEE Transactions on Software Engineering.

[21]  Vishwani D. Agrawal,et al.  A Tutorial on Built-in Self-Test. I. Principles , 1993, IEEE Des. Test Comput..

[22]  Jonas Sjöström,et al.  Supporting the therapist in Online Therapy , 2012, ECIS.

[23]  Vishwani D. Agrawal,et al.  A Tutorial on Built-In Self-Test, Part 2: Applications , 1993, IEEE Des. Test Comput..

[24]  Saskia M Kelders,et al.  Understanding the Usage of Content in a Mental Health Intervention for Depression: An Analysis of Log Data , 2014, Journal of medical Internet research.

[25]  Jan Pries-Heje,et al.  A Comprehensive Framework for Evaluation in Design Science Research 1 , 2022 .

[26]  Michael Sherwood-Smith,et al.  Maximise information systems value by continuous participative evaluation , 1999 .

[27]  Jonas Sjöström,et al.  Design Principles for Research Data Export: Lessons Learned in e-Health Design Research , 2013, DESRIST.

[28]  Göran Goldkuhl,et al.  Strategies for Information Systems Evaluation : Six Generic Types , 2003 .

[29]  Sandeep Purao,et al.  The Sciences of Design: Observations on an Emerging Field , 2008, Commun. Assoc. Inf. Syst..