Task-oriented evaluation of electronic medical records systems: development and validation of a questionnaire for physicians

BackgroundEvaluation is a challenging but necessary part of the development cycle of clinical information systems like the electronic medical records (EMR) system. It is believed that such evaluations should include multiple perspectives, be comparative and employ both qualitative and quantitative methods. Self-administered questionnaires are frequently used as a quantitative evaluation method in medical informatics, but very few validated questionnaires address clinical use of EMR systems.MethodsWe have developed a task-oriented questionnaire for evaluating EMR systems from the clinician's perspective. The key feature of the questionnaire is a list of 24 general clinical tasks. It is applicable to physicians of most specialties and covers essential parts of their information-oriented work. The task list appears in two separate sections, about EMR use and task performance using the EMR, respectively. By combining these sections, the evaluator may estimate the potential impact of the EMR system on health care delivery. The results may also be compared across time, site or vendor. This paper describes the development, performance and validation of the questionnaire. Its performance is shown in two demonstration studies (n = 219 and 80). Its content is validated in an interview study (n = 10), and its reliability is investigated in a test-retest study (n = 37) and a scaling study (n = 31).ResultsIn the interviews, the physicians found the general clinical tasks in the questionnaire relevant and comprehensible. The tasks were interpreted concordant to their definitions. However, the physicians found questions about tasks not explicitly or only partially supported by the EMR systems difficult to answer. The two demonstration studies provided unambiguous results and low percentages of missing responses. In addition, criterion validity was demonstrated for a majority of task-oriented questions. Their test-retest reliability was generally high, and the non-standard scale was found symmetric and ordinal.ConclusionThis questionnaire is relevant for clinical work and EMR systems, provides reliable and interpretable results, and may be used as part of any evaluation effort involving the clinician's perspective of an EMR system.

[1]  S. Abrahamson,et al.  A national study of medical and surgical specialties. I. Background purpose, and methodology. , 1978, JAMA.

[2]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[3]  Jonathan M. Teich,et al.  Research Paper: Implementation of Physician Order Entry: User Satisfaction and Self-Reported usage Patterns , 1996, J. Am. Medical Informatics Assoc..

[4]  William J. Doll,et al.  A Confirmatory Factor Analysis of the End-User Computing Satisfaction Instrument , 1994, MIS Q..

[5]  Ronald E. Rice,et al.  Social worlds, individual differences, and implementation: Predicting attitudes toward a medical information system , 1991, Inf. Manag..

[6]  Barry Kirwan,et al.  A Guide To Task Analysis: The Task Analysis Working Group , 1992 .

[7]  Mathews Ka,et al.  Evaluation of clinical information systems. , 1993 .

[8]  Charles P. Friedman,et al.  Evaluation Methods in Medical Informatics , 1997, Computers and Medicine.

[9]  Paul N. Gorman,et al.  Information needs of physicians , 1995 .

[10]  A. Shepherd,et al.  Guide to Task Analysis , 2003 .

[11]  James D. Wright,et al.  Handbook of Survey Research. , 1985 .

[12]  William J. Doll,et al.  The measurement of end-user computing satisfaction: theoretical and methodological issues , 1991 .

[13]  Robert P. Bostrom,et al.  Some new factors influencing user information satisfaction: implications for systems professionals , 1986, SIGCPR '86.

[14]  R. Hanka,et al.  Evaluating information technology in health care: barriers and challenges , 1998, BMJ.

[15]  P. V. Marsden,et al.  Handbook of Survey Research , 1985 .

[16]  P Wirth,et al.  Comparability of Two Methods of Time and Motion Study Used in a Clinical Setting: Work Sampling and Continuous Observation , 1977, Medical care.

[17]  V L Patel,et al.  Cognitive computer-based video analysis: its application in assessing the usability of medical systems. , 1995, Medinfo. MEDINFO.

[18]  F. Kerlinger,et al.  Foundations of behavioral research : educational and psychological inquiry , 1966 .

[19]  H. Lærum,et al.  Doctors' use of electronic medical records systems in hospitals: cross sectional survey , 2001, BMJ : British Medical Journal.

[20]  T. Massaro.  Introducing Physician Order Entry at a Major Academic Medical Center: I. Impact on Organizational Culture and Behavior , 1993, Academic medicine : journal of the Association of American Medical Colleges.

[21]  Peter H. Rossi,et al.  QUANTITATIVE STUDIES IN SOCIAL RELATIONS , 1976 .

[22]  E. M. S. J. Van Gennip,et al.  Assessment and evaluation of information technogies in medicine , 1995 .

[23]  James G. Anderson,et al.  Use and Impact of Computers in Clinical Medicine , 2011, Computers and Medicine.

[24]  William J. Doll,et al.  The Measurement of End-User Computing Satisfaction , 1988, MIS Q..

[25]  D. K. Donker.  Assessment and evaluation of information technologies in medicine , 1996 .

[26]  Julie M. Fiskio,et al.  Evaluating physician satisfaction regarding user interactions with an electronic medical record system , 1999, AMIA.

[27]  Charles P. Friedman,et al.  Development and initial validation of an instrument to measure physicians' use of, knowledge about, and attitudes toward computers. , 1998, Journal of the American Medical Informatics Association : JAMIA.

[28]  S Abrahamson,et al.  A national study of medical and surgical specialties. II. Description of the survey instrument. , 1978, JAMA.

[29]  E. Nelson,et al.  A Study of the Validity of the Task Inventory Method of Job Analysis , 1975, Medical care.

[30]  Arild Faxvaag,et al.  Research Paper: Effects of Scanning and Eliminating Paper-based Medical Records on Hospital Physicians' Clinical Work Practice , 2003, J. Am. Medical Informatics Assoc..

[31]  I Jacoby,et al.  Task Analysis in National Health Service Corps Field Stations: A Methodological Evaluation , 1975, Medical care.

[32]  J Dudeck,et al.  Evaluation of clinical information systems. What can be evaluated and what cannot? , 2001, Journal of evaluation in clinical practice.

[33]  J. Ware,et al.  Testing the equivalence of translations of widely used response choice labels: results from the IQOLA Project. International Quality of Life Assessment. , 1998, Journal of clinical epidemiology.