An interpretative evaluation of a healthcare intranet

The evaluation of information systems and health information systems in particular, is recognised as a complex and challenging activity. There is no agreement on a 'best way' to evaluate, on what and how to evaluate, who to involve and within what paradigm to proceed. This paper argues that evaluation is a situated activity that requires a more interpretative approach, broadly based on hermeneutic principles. The paper reviews a number of guidelines and frameworks for evaluation presented in the medical and information systems literature, focusing on two related approaches: the context, process and content framework, and structure, process and outcome framework. A synthesis of these were used in the case study described - the evaluation of an intranet pilot deployment within primary care. The literature review and research experience (including the case study) form the basis for a set of general recommendations presented in the conclusion section.

[1]  P. Taylor,et al.  A survey of research in telemedicine. 1: Telemedicine systems , 1998, Journal of telemedicine and telecare.

[2]  Blake Ives,et al.  The measurement of user information satisfaction , 1983, CACM.

[3]  Chris Sauer,et al.  Research Paper: Information Technology in Complex Health Services: Organizational Impediments to Successful Technology Transfer and Diffusion , 1997, J. Am. Medical Informatics Assoc..

[4]  D. Plamping,et al.  The new NHS. , 1991, BMJ.

[5]  James W. Fairfield-Sonn,et al.  The Awakening Giant: Continuity and Change in ICI. , 1987 .

[6]  Dan Remenyi,et al.  A user-satisfaction approach to IS effectiveness measurement , 1991, J. Inf. Technol..

[7]  B. Kaplan.  Development and acceptance of medical information systems: an historical overview. , 1988, Journal of health and human resources administration.

[8]  Robert M. O’Keefe,et al.  The evaluation of decision-aiding systems: Guidelines and methods , 1989, Inf. Manag..

[9]  Ananth Srinivasan,et al.  Alternative Measures of Systems Effectiveness: Associations and Implications , 1985, MIS Q..

[10]  Frank F. Land,et al.  Evaluation in a Socio-technical Context , 2000, Organizational and Social Perspectives on IT.

[11]  P. Taylor,et al.  A survey of research in telemedicine. 2: Telemedicine services , 1998, Journal of telemedicine and telecare.

[12]  Tony Cornford,et al.  Experience with a structure, process and outcome framework for evaluating an information system , 1994 .

[13]  Tom Renkema.  Boekbespreking Farbey, B.; Land, F.; Targett, D.: how to assess your IT investment, a study of methods and practice (Butterworth Heinemann, Oxford, 1993) , 1994 .

[14]  Tony Cornford,et al.  Evaluation and telehealth - an interpretative study , 2001, Proceedings of the 34th Annual Hawaii International Conference on System Sciences.

[15]  Kieran Mathieson.  Reducing bias in user's evaluations of information systems , 1993, Inf. Manag..

[16]  S. Olsson,et al.  Developments and directions of technology assessment in telemedicine , 1995, Proceedings of the Fourth International Conference on Image Management and Communication (IMAC 95).

[17]  Marilynne Hebert,et al.  Telehealth Success: Evaluation Framework Development , 2001, MedInfo.

[18]  Ann Bowling,et al.  Research Methods in Health: Investigating Health and Health Services , 1997 .

[19]  Jo Lenaghan.  Rethinking IT and Health , 1998 .

[20]  Chrisanthi Avgerou,et al.  Evaluating information systems by consultation and negotiation , 1995 .

[21]  V. J. Symons,et al.  A review of information systems evaluation: content, context and process , 1991 .

[22]  D Lobley,et al.  The economics of telemedicine , 1997, Journal of telemedicine and telecare.

[23]  R. Sheaff,et al.  Managing health service information systems : an introduction , 1995 .

[24]  Frank F. Land,et al.  The moving staircase - Problems of appraisal and evaluation in a turbulent environment , 1999, Inf. Technol. People.

[25]  Paul Taylor,et al.  Evaluating telemedicine systems and services , 2005, Journal of telemedicine and telecare.

[26]  Kuldeep Kumar,et al.  Post implementation evaluation of computer-based information systems: current practices , 1990, Commun. ACM.

[27]  Steve New,et al.  Information Systems Management: Opportunities and Risks , 1991 .

[28]  Rachel Jenkins.  Health of the nation. , 1969, British medical journal.

[29]  R. Hanka,et al.  Evaluating information technology in health care: barriers and challenges , 1998, BMJ.

[30]  J. Wyatt,et al.  Telemedicine in the NHS for the millennium and beyond. , 1998, Postgraduate medical journal.

[31]  Ephraim R. McLean,et al.  Information Systems Success: The Quest for the Dependent Variable , 1992, Inf. Syst. Res..

[32]  Steve Smithson,et al.  Information systems evaluation: myth and reality , 1987 .

[33]  Lucas D. Introna.  Management, information and power: A narrative of the involved manager , 1997 .

[34]  Philip Powell,et al.  Information Technology Evaluation: Is It Different? , 1992 .