The eleven years of European Conference on IT evaluation: Retrospectives and perspectives for possible future research

This paper provides an overview of the papers that have been presented at the European Conference on IT Evaluation during the past eleven years. It considers the main issues, and learning themes addressed in papers presented to these Conferences. The paper also reflects on the possible future direction, which this research may take and three major research themes are suggested. Some 356 papers have been presented at ECITE. Over the eleven year period it is clear that the level of understanding as reflected in the papers has significantly increased. Themes, which were particularly well addressed, include IT and IS value, the multidisciplinary nature of evaluation, the importance of stakeholder analysis, organisational learning and life cycle management. Three issues are identified as particularly important for further research. These are, the theoretical underpinning of IT evaluation, improving the data sets for research and establishing a more common core of concepts.

[1]  David Snowden,et al.  The First Age: Information for Decision Support 1995: the Transition to the Second Age Complex Acts of Knowing: Paradox and Descriptive Self-awareness , 2022 .

[2]  Graeme G. Shanks,et al.  Successfully completing case study research: combining rigour, relevance and pragmatism , 1998, Inf. Syst. J..

[3]  Jan Lambrecht Andresen,et al.  Evaluation of IT in the Danish Construction Industry , 1999 .

[4]  P. Drucker What business can learn from nonprofits. , 1989, Harvard business review.

[5]  Michael D. Myers,et al.  A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems , 1999, MIS Q..

[6]  James C. Wetherbe,et al.  Key Issues in Information Systems Management: 1994-95 SIM Delphi Results , 1996, MIS Q..

[7]  Steve Smithson,et al.  Information systems evaluation as an organizational institution – experience from a case study , 2003, Inf. Syst. J..

[8]  Pär J. Ågerfalk,et al.  Setting the Scene for Actability Evaluation : Understanding Information Systems in Context , 2002 .

[9]  Elske Ammenwerth,et al.  Evaluation of user acceptance of data management systems in hospitals - feasibility and usability - , 2002 .

[10]  Daniel P. Forbes Measuring the Unmeasurable: Empirical Studies of Nonprofit Organization Effectiveness from 1977 to 1997 , 1998 .

[11]  John W. Meyer,et al.  Institutionalized Organizations: Formal Structure as Myth and Ceremony , 1977, American Journal of Sociology.

[12]  R. Kaplan Strategic Performance Measurement and Management in Nonprofit Organizations , 2001 .

[13]  David Sinclair,et al.  Performance measurement as an obstacle to TQM , 1995 .

[14]  Richard L. Daft,et al.  Message Equivocality, Media Selection, and Manager Performance: Implications for Information Systems , 1987, MIS Q..

[15]  Kerstin Grundén An Evaluation Model for CSCW Systems , 2002 .

[16]  E. Erickson Increasing Leadership Effectiveness , 1978 .

[17]  Rosı́o Alvarez The Myth of Integration: A Case Study of an ERP Implementation , 2002 .

[18]  Hanne Nørreklit The Balance on the Balanced Scorecard: A Critical Analysis of Some of Its Assumptions , 2000 .

[19]  M. Buck-Lew,et al.  To outsource or not , 1992 .

[20]  Geoff Walsham,et al.  Interpreting Information Systems in Organizations , 1993 .

[21]  Lars Bo Henriksen,et al.  In Search of a Critical Theory of ICT Evaluation: From Heidegger to Habermas , 2001 .

[22]  D. Scott Brandt Evaluating information on the Internet , 1996 .

[23]  A. Atkinson,et al.  A Stakeholder Approach to Strategic Performance Measurement , 1997 .

[24]  Izak Benbasat,et al.  The Case Research Strategy in Studies of Information Systems , 1987, MIS Q..

[25]  M. Treacy,et al.  Utilization as a dependent variable in MIS research , 2015, DATB.

[26]  Paul Lillrank,et al.  Catching intangible IT benefits , 2001 .

[27]  María de Lourdes,et al.  Revistas científicas electrónicas , 1998 .

[28]  Steven Jones,et al.  Understanding IS evaluation as a complex social process: a case study of a UK local authority , 2001, Eur. J. Inf. Syst..

[29]  Zahir Irani,et al.  Information systems evaluation: past, present and future , 2001, Eur. J. Inf. Syst..

[30]  P.M.G. Van Veen-Dirks,et al.  Control of the information systems function: the role of cost allocation , 1997 .

[31]  Andrew Basden,et al.  How Habermas’ Action Types Can Influence KBS Design , 2003 .

[32]  Alastair G. Smith,et al.  Testing the Surf: Criteria for Evaluating Internet Information Resources , 1997 .

[33]  Hannu Salmela,et al.  Evaluation of information systems in health care : a framework and its application , 2000 .

[34]  James D. McKeen,et al.  An integrative research approach to assess the business value of information technology , 1998 .

[35]  Andrew Turk A contingency approach to designing usability evaluation procedures for WWW sites , 2000 .

[36]  Antonius A.M. Spil The evaluation of strategic information systems planning; From a quality undergrowth to a semiotic clearing , 1995 .

[37]  Nigel Magson ‘Information Technology for the Not‐For‐Profit Sector’. by Ian Harris and Michael Mainelli. ICSA Publishing, London; June 2001; ISBN 186072303; 224 pp , 2002 .

[38]  Leslie P. Willcocks,et al.  Measuring organizational IS effectiveness: an overview and update of senior management perspectives , 2002, DATB.

[39]  Pam J. Mayhew,et al.  Consensus Building in Formative and Participative IS Evaluation Approaches , 2001 .

[40]  Leslie P. Willcocks,et al.  Evaluating Information Technology investments: research findings and reappraisal , 1992, Inf. Syst. J..

[41]  Alison Cooke Authoritative Guide to Evaluating Information on the Internet. Neal-Schuman NetGuide Series. , 1999 .

[42]  Kuldeep Kumar,et al.  Post implementation evaluation of computer-based information systems: current practices , 1990, Commun. ACM.

[43]  P. Strassmann Information Payoff: The Transformation of Work in the Electronic Age , 1985 .

[44]  Kenneth E. Murphy,et al.  Intangible benefits valuation in ERP projects , 2002, Inf. Syst. J..

[45]  Goel Kahen,et al.  A Quantitative Model for Technological Risk Assessment in the Process of Information Technology Transfer , 2007 .

[46]  Alberto Pedroso García,et al.  SciELO: una metodología para la publicación electrónica , 2001 .

[47]  M. Lynne Markus,et al.  Power, politics, and MIS implementation , 1987, CACM.

[48]  Daniel Fitton,et al.  Smart mobs and technology probes: evaluating texting at work , 2004 .

[49]  R. Dorf,et al.  The Balanced Scorecard: Translating Strategy Into Action , 1997, Proceedings of the IEEE.

[50]  Terry Ann Rohe How Does Electronic Publishing Affect the Scholarly Communication Process , 1998 .

[51]  Paul Beynon-Davies,et al.  Evaluating the Experience of E-Business amongst SMEs , 2002 .

[52]  James Testa La base de datos del ISI y su proceso de selección de revistas , 2001 .

[53]  Margarita Lugo Hubp,et al.  Evaluación de recursos digitales: análisis de algunos parámetros , 2004 .

[54]  D. Sandy Staples,et al.  Dimensions of Information Systems Success , 1999, Commun. Assoc. Inf. Syst..

[55]  Enrique Dans,et al.  IT Responsiveness in Small and Medium Enterprises: It Pays to Be on Top of IT , 2001 .

[56]  D. Robey,et al.  Software stories: three cultural perspectives on the organizational practices of software development , 1999 .

[57]  Geoff Walsham,et al.  Information systems strategy and implementation: a case study of a building society , 1994, TOIS.

[58]  Egon Berghout,et al.  The four P's revisited : Appraising and managing the infrastructure impact of IT-investments , 1997 .

[59]  David Targett,et al.  A taxonomy of information systems applications: the benefits' evaluation ladder , 1995 .

[60]  Donald A. Schön,et al.  Theory in Practice: Increasing Professional Effectiveness , 1974 .

[61]  Gerhard Speckbacher,et al.  The economics of performance management in nonprofit organizations , 2003 .

[62]  Maria D. Molina,et al.  IT INVESTMENT IN SMALL AND MEDIUM ENTERPRISES : PARADOXICALLY PRODUCTIVE ? , 2001 .

[63]  Richard T. Watson,et al.  Global comparisons of key issues in IS management: extending key issues selection procedure and survey approach , 2000, Proceedings of the 33rd Annual Hawaii International Conference on System Sciences.

[64]  A. Gogh The impact of information technology. , 1997, Caring : National Association for Home Care magazine.

[65]  J. Habermas Theory of Communicative Action , 1981 .

[66]  Michael Sherwood-Smith,et al.  Achieving Maximum Value from Information Systems: A Process Approach , 1997 .

[67]  Clive Savory Co-Operative Inquiry as a basis for Evaluation of Knowledge Management Tools , 2001 .

[68]  M Looijen,et al.  Simulation as evaluation tool in IT service management , 1999 .

[69]  V. J. Symons,et al.  A review of information systems evaluation: content, context and process , 1991 .

[70]  Lisa Abrams,et al.  Using Mentoring and Storytelling to Transfer Knowledge in the Workplace , 2001, J. Manag. Inf. Syst..

[71]  Dirk Deschoolmeester,et al.  Evaluation of ERP Investments in the Belgian Assembly Industry , 2000 .

[72]  Susanne Bødker Scenarios in user-centred designsetting the stage for reflection and action , 2000, Interact. Comput..

[73]  Hasse Clausen Designing Computer Systems from a Human Perspective: The Use of Narratives , 1994, Scand. J. Inf. Syst..

[74]  Christina Soh,et al.  How IT Creates Business Value: A Process Theory Synthesis , 1995, ICIS.

[75]  Geoff Walsham,et al.  The Emergence of Interpretivism in IS Research , 1995, Inf. Syst. Res..

[76]  Egon Berghout,et al.  Cost management of IT beyond cost of ownership models: a state of the art overview of the Dutch financial services industry , 2001 .

[77]  Rosas Gutiérrez,et al.  La publicación electrónica: ¿Un paradigma de organización documental digital? , 1998 .

[78]  M Looijen,et al.  Managing the life cycle of Information and Communication Technology investments for added value. , 1997 .

[79]  John J. McCall,et al.  Information Economics , 2022, Management.

[80]  M. Porter Competitive Advantage: Creating and Sustaining Superior Performance , 1985 .

[81]  Sue Llewellyn,et al.  Narratives in accounting and management research , 1999 .

[82]  K. Sweeney The information paradox. , 1998, Occasional paper.

[83]  Annie I. Antón,et al.  A Representational Framework for Scenarios of System Use , 1998, Requirements Engineering.

[84]  Matthias Jarke,et al.  Scenario Management: An Interdisciplinary Approach , 1998, Requirements Engineering.

[85]  Egon Berghout,et al.  Problems and Issues Relating to IT Investment Evaluation: Experiences from Scottish and Benelux Studies , 2000 .

[86]  Hajer Kefi IS/IT Evaluation: a process-oriented and context-based approach , 2002 .

[87]  Steve Smithson,et al.  Information systems evaluation in practice: a case study of organizational change , 2000, J. Inf. Technol..

[88]  Jaime Ríos Ortega,et al.  Normalización de revistas científicas mexicanas:campo de investigación y aportación , 2000 .

[89]  Angèle L. M. Cavaye,et al.  Case study research: a multi‐faceted research approach for IS , 1996, Inf. Syst. J..

[90]  L. Willcocks,et al.  To outsource IT or not?: recent research on economics and evaluation practice , 1996 .

[91]  K. Weick Drop Your Tools: An Allegory for Organizational Studies. , 1996 .

[92]  Neil F. Doherty,et al.  The stakeholder dimension in information systems evaluation , 2002, J. Inf. Technol..

[93]  Michael Sherwood-Smith,et al.  Maximise information systems value by continuous participative evaluation , 1999 .

[94]  Karsten Ulrik Niss Model for Investment and Evaluation of medical Information Systems (MIEMIS) , 1999 .

[95]  Ian Ward The Myth of Integration , 1996 .