Methods for evaluating hospital information systems: a literature review

Purpose – It is widely accepted that the use of information and communication technology (ICT) in the healthcare sector offers great potential for improving the quality of services provided, the efficiency and effectiveness of personnel, and also reducing organizational expenses. This paper seeks to examine various hospital information system (HIS) evaluation methods.Design/methodology/approach – In this paper a comprehensive search of the literature concerning the evaluation of complex health information systems is conducted and used to generate a synthesis of the literature around evaluation efforts in this field. Three approaches for evaluating hospital information systems are presented – user satisfaction, usage, and economic evaluation.Findings – The main results are that during the past decade, computers and information systems, as well as their resultant products, have pervaded hospitals worldwide. Unfortunately, methodologies to measure the various impacts of these systems have not evolved at the ...

[1]  D. Rigby.  Integrated continence services. , 2001, Nursing Standard.

[2]  James G. Anderson,et al.  Clearing the way for physicians' use of clinical information systems , 1997, CACM.

[3]  J. E. Bailey,et al.  Development of an Instrument for the Management of Computer User Attitudes in Hospitals , 1990, Methods of Information in Medicine.

[4]  J Dudeck,et al.  Evaluation of clinical information systems. What can be evaluated and what cannot? , 2001, Journal of evaluation in clinical practice.

[5]  M. Fieschi,et al.  Introduction to Clinical Informatics , 1996, Health Informatics Series.

[6]  Claude Sicotte,et al.  Assessment of a computerized medical record system: disclosing scripts of use , 1999 .

[7]  Ephraim R. McLean,et al.  Information Systems Success: The Quest for the Dependent Variable , 1992, Inf. Syst. Res..

[8]  Anthony R. Hendrickson,et al.  Using Davis's Perceived Usefulness and Ease-of-use Instruments for Decision Making: A Confirmatory and Multigroup Invariance Analysis , 1998 .

[9]  M. Treacy,et al.  Utilization as a dependent variable in MIS research , 2015, DATB.

[10]  Rahat H. Bokhari,et al.  The relationship between system usage and user satisfaction: a meta-analysis , 2005, J. Enterp. Inf. Manag..

[11]  N. Melone.  A theoretical assessment of the user-satisfaction construct in information systems research , 1990 .

[12]  Daniel Robey,et al.  User Attitudes and Management Information System Use , 1979 .

[13]  R L Arenson,et al.  Comprehensive analysis of a Radiology Operations Management computer system. , 1979, Radiology.

[14]  M. Drummond,et al.  Health Care Technology: Effectiveness, Efficiency and Public Policy@@@Methods for the Economic Evaluation of Health Care Programmes , 1988 .

[15]  Nick Freemantle,et al.  Computer support for determining drug dose: systematic review and meta-analysis , 1999, BMJ.

[16]  Eric W.T. Ngai,et al.  A critical review of end-user information system satisfaction research and a new research framework , 2002 .

[17]  Ruth C. King,et al.  An application of the cusp catastrophe model to user information satisfaction , 1998, Inf. Manag..

[18]  Ephraim R. McLean,et al.  The DeLone and McLean Model of Information Systems Success: A Ten-Year Update , 2003, J. Manag. Inf. Syst..

[19]  Jonathan M. Teich,et al.  Research Paper: Implementation of Physician Order Entry: User Satisfaction and Self-Reported usage Patterns , 1996, J. Am. Medical Informatics Assoc..

[20]  Peter B. Seddon.  A Respecification and Extension of the DeLone and McLean Model of IS Success , 1997, Inf. Syst. Res..

[21]  Timo Saarinen,et al.  An expanded instrument for evaluating information system success , 1996, Inf. Manag..

[22]  Shailendra C. Palvia,et al.  A socio-technical framework for quality assessment of computer information systems , 2001, Ind. Manag. Data Syst..

[23]  C G Ward,et al.  Clinical evaluation of a computer-based patient monitoring and data management system. , 1991, Heart & lung : the journal of critical care.

[24]  Roger McHaney,et al.  A validation of the end-user computing satisfaction instrument in Taiwan , 2002, Inf. Manag..

[25]  B Kaplan,et al.  Toward an Evaluation of an Integrated Clinical Imaging System: Identifying Clinical Benefits , 1996, Methods of Information in Medicine.

[26]  M. Gold.  Cost-effectiveness in health and medicine , 2016 .

[27]  R. Haux,et al.  A Randomized Evaluation of a Computer-Based Nursing Documentation System , 2001, Methods of Information in Medicine.

[28]  A Hasman,et al.  User satisfaction of general practitioners with HIOS+, a medical decision support system. , 1995, Computer methods and programs in biomedicine.

[29]  Detmar W. Straub,et al.  Measuring System Usage: Implications for IS Theory Testing , 1995 .

[30]  J Allan,et al.  Patient-centered documentation: an effective and efficient use of clinical information systems. , 2000, The Journal of nursing administration.

[31]  ICT penetration in public Greek hospitals. , 2002, Studies in health technology and informatics.

[32]  C. Ohmann,et al.  A systematic approach to the assessment of user satisfaction with health care systems: constructs, models and instruments. , 1997, Studies in health technology and informatics.

[33]  Arthur G. Money,et al.  The Quantification of Decision Support Benefits Within the Context of Value Analysis , 1988, MIS Q..

[34]  David W. Bates,et al.  Reducing the frequency of errors in medicine using information technology. , 2001, Journal of the American Medical Informatics Association : JAMIA.

[35]  Lucas Hc.  The evolution of an information system: from key-man to every person. , 1978 .

[36]  William J. Doll,et al.  The Measurement of End-User Computing Satisfaction , 1988, MIS Q..

[37]  J. March,et al.  A Behavioral Theory of the Firm , 1964 .

[38]  J Dudeck,et al.  Stepwise Evaluation of Information Systems in an University Hospital , 1999, Methods of Information in Medicine.

[39]  Sammy W. Pearson,et al.  Development of a Tool for Measuring and Analyzing Computer User Satisfaction , 1983 .

[40]  J. Miller.  Information systems effectiveness: the fit between business needs and system capabilities , 1989, ICIS '89.

[41]  Kenneth E. Murphy,et al.  Intangible benefits valuation in ERP projects , 2002, Inf. Syst. J..

[42]  Hugh J. Watson,et al.  Management's Role in the Approval and Administration of Decision Support Systems , 1983, MIS Q..

[43]  Amy W. Gatian.  Is user satisfaction a valid measure of system effectiveness? , 1994, Inf. Manag..

[44]  Ritu Agarwal,et al.  The Role of Innovation Characteristics and Perceived Voluntariness in the Acceptance of Information Technologies , 1997 .

[45]  Margaret Tan,et al.  The consequences of information technology acceptance on subsequent individual performance , 1997, Inf. Manag..

[46]  Wilhelm Hasselbring,et al.  Strategic information management plans: the basis for systematic information management in hospitals , 2001, Int. J. Medical Informatics.

[47]  Claude Sicotte,et al.  The Computer-Based Patient Record Challenges Towards Timeless and Spaceless Medical Practice , 1998, Journal of Medical Systems.

[48]  A Hasman,et al.  Medical informatics as a discipline at the beginning of the 21st century. , 2002, Methods of information in medicine.

[49]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[50]  A K Szczepura,et al.  Information for decision makers at hospital laboratory level: an example of a graphical method of representing costs and effects for a replacement automated technology in a haematology laboratory. , 1988, Social science & medicine.

[51]  Jinjoo Lee,et al.  Utilization and user satisfaction in end-user computing: a task contingent model , 1998 .

[52]  A R Bakker,et al.  Evaluation of automated information systems in health care: an approach to classifying evaluative studies. , 1995, Computer methods and programs in biomedicine.

[53]  E. Johnsen.  Richard M. Cyert & James G. March, A Behavioral Theory of The Firm, Prentice-Hall, Inc., Englewood Cliffs, New Jersey, 1963, 332 s. , 1964 .

[54]  Michael J. Ginzberg,et al.  Early Diagnosis of MIS Implementation Failure: Promising Results and Unanswered Questions , 1981 .

[55]  Yolande E. Chan.  IT Value: The Great Divide Between Qualitative and Quantitative and Individual and Organizational Measures , 2000, J. Manag. Inf. Syst..

[56]  Alexander M. Maish.  A User's Behavior Toward His MIS , 1979, MIS Q..

[57]  D. Sindhusake,et al.  A comparison of nursing activities associated with manual and automated documentation in an Australian intensive care unit. , 1997, Computers in nursing.

[58]  Jenny Owen,et al.  Job design within a human centred (system) design framework , 1992 .

[59]  Blake Ives,et al.  User Involvement and MIS Success: A Review of Research , 1984 .

[60]  Andrew Grant,et al.  The TEAM methodology for the evaluation of information systems in biomedicine , 2002, Comput. Biol. Medicine.

[61]  Reza Khorramshahgol,et al.  Hospital information systems quality: a customer satisfaction assessment tool , 1999, Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences. 1999. HICSS-32. Abstracts and CD-ROM of Full Papers.

[62]  Bernadette Szajna,et al.  Empirical evaluation of the revised technology acceptance model , 1996 .

[63]  Jeremy C. Wyatt,et al.  When and how to evaluate health information systems? , 2003, Int. J. Medical Informatics.

[64]  Albert H. Segars,et al.  Information systems effectiveness: The construct space and patters of application , 1996, Inf. Manag..

[65]  Dale Goodhue,et al.  Task-Technology Fit and Individual Performance , 1995, MIS Q..

[66]  Pirkko Nykänen,et al.  Decision support systems from a health informatics perspective , 2000 .

[67]  Dennis F. Galletta,et al.  Some Cautions on the Measurement of User Information Satisfaction , 1989 .

[68]  Bonnie Kaplan,et al.  Review: Addressing Organizational Issues into the Evaluation of Medical Systems , 1997, J. Am. Medical Informatics Assoc..

[69]  G E Pugh,et al.  Computerized Databases for Emergency Care: What Impact on Patient Care? , 1994, Methods of Information in Medicine.

[70]  Michael Weiner,et al.  Research Paper: Contrasting Views of Physicians and Nurses about an Inpatient Computer-based Provider Order-entry System , 1999, J. Am. Medical Informatics Assoc..

[71]  Edward J. Garrity,et al.  Information systems success measurement , 1998 .

[72]  P. Metnitz,et al.  Three Years' Experience with a Patient Data Management System at a Neonatal Intensive Care Unit , 1998, Journal of Clinical Monitoring and Computing.

[73]  D. Spiegelhalter,et al.  Field trials of medical decision-aids: potential problems and solutions. , 1991, Proceedings. Symposium on Computer Applications in Medical Care.

[74]  Gerald L. Matlin,et al.  SIM Competition Paper: What Is the Value of Investment in Information Systems? , 1979, MIS Q..

[75]  Mo Adam Mahmood,et al.  Variables affecting information technology end-user satisfaction: a meta-analysis of the empirical literature , 2000, Int. J. Hum. Comput. Stud..

[76]  Reed M. Gardner,et al.  Large-Scale Clinical Systems. Impact of Large-Scale Systems on Clinical Practice: Assessing the Effectiveness of a Computerized Pharmacy System , 1990 .

[77]  H L Chin,et al.  Evaluating a comprehensive outpatient clinical information system: a case study and model for system evaluation. , 1995, Proceedings. Symposium on Computer Applications in Medical Care.

[78]  Kerstin Grundén.  An Evaluation Model for CSCW Systems , 2002 .

[79]  George Ditsa,et al.  Our Mousetrap's Fine: So Why Aren't People Beating A Path To Our Door? , 1997 .

[80]  Viswanath Venkatesh,et al.  User Acceptance Enablers in Individual Decision Making About Technology: Toward an Integrated Model , 2002, Decis. Sci..

[81]  Reed M. Gardner,et al.  Research Paper: Evaluation of User Acceptance of a Clinical Expert system , 1994, J. Am. Medical Informatics Assoc..

[82]  Charles P. Friedman,et al.  Evaluation Methods in Medical Informatics , 1997, Computers and Medicine.

[83]  Blake Ives,et al.  The measurement of user information satisfaction , 1983, CACM.

[84]  Anne Moen,et al.  Introducing a Module for Laboratory Test Order Entry and Reporting of Results at a Hospital Ward: An Evaluation Study Using a Multi-Method Approach , 1997, Journal of Medical Systems.

[85]  Arie Hasman,et al.  Review Paper: Determinants of Success of Inpatient Clinical Information Systems: A Literature Review , 2003, J. Am. Medical Informatics Assoc..

[86]  Elske Ammenwerth,et al.  Evaluation of health information systems - problems and challenges , 2003, Int. J. Medical Informatics.