Can evaluation studies benefit from triangulation? A case study

BACKGROUND Information and communication technologies (ICTs) are increasingly being used in health care. Rigorous evaluations of ICT applications during both introduction and routine use are of great importance for decision makers and users. Within evaluation research, two main (and often rather distinct) traditions can be found: the objectivistic and the subjectivistic tradition. METHODS The theory of triangulation deals with the integration of methods and approaches as to conduct better evaluation studies. In evaluation research, triangulation in general means the multiple employment of various sources of data, observers, methods, and/or theories in investigations of the same phenomenon. We applied triangulation aspects in the analysis of the effects of a computer-based nursing documentation system. RESULTS We discuss, based on this case study, what benefits can be obtained from applying triangulation in an evaluation study. We show how both the validation of results and the completeness of results can be supported by triangulation. DISCUSSION The decision whether triangulation may be useful for a given research question, and how it may be correctly applied, requires-like other evaluation methods-intensive training and methodological experience. Medical informatics evaluation research may profit from this well-established theory.

[1]  G. Burr,et al.  Contextualizing critical care family needs through triangulation: an Australian study. , 1998, Intensive & critical care nursing.

[2]  Dale Goodhue,et al.  Understanding user evaluations of information systems , 1995 .

[3]  Rosaline S. Barbour,et al.  The Case for Combining Qualitative and Quantitative Approaches in Health Services Research , 1999, Journal of health services research & policy.

[4]  N. Denzin The research act: A theoretical introduction to sociological methods , 1977 .

[5]  Elske Ammenwerth,et al.  Nursing process documentation systems in clinical routine - prerequisites and experiences , 2001, Int. J. Medical Informatics.

[6]  C. Kendall,et al.  Social network influences on reproductive health behaviors in urban northern Thailand. , 1999, Social science & medicine.

[7]  Charles P. Friedman,et al.  Evaluation Methods in Medical Informatics , 1997, Computers and Medicine.

[8]  Michael Rigby,et al.  Evaluation: 16 Powerful Reasons Why Not to Do It - And 6 Over-Riding Imperatives , 2001, MedInfo.

[9]  J. Marc Overhage,et al.  A plea for controlled trials in medical informatics , 1994, J. Am. Medical Informatics Assoc..

[10]  R. Haux,et al.  A Randomized Evaluation of a Computer-Based Nursing Documentation System , 2001, Methods of Information in Medicine.

[11]  E. Mitchell,et al.  Multiple triangulation: a methodology for nursing science. , 1986, ANS. Advances in nursing science.

[12]  D. Thompson,et al.  Nurses' attitudes towards the nursing process. , 1983, Journal of advanced nursing.

[13]  H A Heathfield,et al.  Current evaluations of information technology in health care are often inadequate , 1996, BMJ.

[14]  Bonnie Kaplan,et al.  Combining Qualitative and Quantitative Methods in Information Systems Research: A Case Study , 1988, MIS Q..

[15]  Fred D. Davis User Acceptance of Information Technology: System Characteristics, User Perceptions and Behavioral Impacts , 1993, Int. J. Man Mach. Stud..

[16]  Jochen R. Moehr Evaluation: salvation or nemesis of medical informatics? , 2002, Comput. Biol. Medicine.

[17]  Jos Aarts,et al.  Using a descriptive model of change when implementing large scale clinical information systems to identify priorities for further research , 1999, Int. J. Medical Informatics.

[18]  Gary S. Nickell,et al.  The computer attitude scale , 1986 .

[19]  Jytte Brender,et al.  Trends in assessment of IT-based solutions in healthcare and recommendations for the future , 1998, Int. J. Medical Informatics.

[20]  K. Knafl,et al.  Triangulation in Qualitative Research: Issues of Conceptual Clarity and Purpose , 1991 .

[21]  C. Lowry,et al.  Nurses' attitudes toward computerised care plans in intensive care. Part 2. , 1994, Intensive & critical care nursing.

[22]  J. Bortz,et al.  Forschungsmethoden und Evaluation für Human- und Sozialwissenschaftler , 2006 .

[23]  Vimla L. Patel,et al.  The patient clinical information system (PatCIS): technical solutions for and experience with giving patients access to their electronic medical records , 2002, Int. J. Medical Informatics.

[24]  J Allan,et al.  Patient-centered documentation: an effective and efficient use of clinical information systems. , 2000, The Journal of nursing administration.

[25]  R. Hanka,et al.  Evaluating information technology in health care: barriers and challenges , 1998, BMJ.

[26]  Bonnie Kaplan,et al.  Review: Addressing Organizational Issues into the Evaluation of Medical Systems , 1997, J. Am. Medical Informatics Assoc..

[27]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[28]  D. Brammen,et al.  An Anesthesia Information Management System (AIMS) as a tool for controlling resource management of operating rooms. , 2002, Methods of information in medicine.

[29]  Bonnie Kaplan,et al.  Evaluating informatics applications - some alternative approaches: theory, social interactionism, and call for methodological pluralism , 2001, Int. J. Medical Informatics.

[30]  K. Lewin Frontiers in Group Dynamics , 1947 .

[31]  Beveridge Rn,et al.  Documenting and communicating patient care: Are nursing care plans redundant? , 2000 .

[32]  Marc Berg,et al.  The contextual nature of medical information , 1999, Int. J. Medical Informatics.

[33]  K Sharp,et al.  A critical appraisal of the role of triangulation in nursing research. , 1998, International journal of nursing studies.

[34]  Elske Ammenwerth,et al.  Research Paper: Factors Affecting and Affected by User Acceptance of Computer-based Nursing Documentation: Results of a Two-year Study , 2003, J. Am. Medical Informatics Assoc..

[35]  O Barnett,et al.  Evaluation of a problem-based access knowledge system using triangulation methods. , 1995, Medinfo. MEDINFO.

[36]  B G Buchanan,et al.  Broadening our approach to evaluating medical information systems. , 1991, Proceedings. Symposium on Computer Applications in Medical Care.

[37]  R. T. Riley,et al.  Organizational Aspects of Health Informatics , 1994, Computers in Health Care.

[38]  Ellen Sullivan Mitcbell Multiple triangulation: a methodology for nursing science , 1986 .

[39]  T H Payne,et al.  Computer order entry system decreased use of sliding scale insulin regimens. , 2002, Methods of information in medicine.

[40]  N. Denzin,et al.  The Research Act , 1978 .

[41]  B Herbig,et al.  The care information system PIK--the fit between user and system. , 2002, Methods of information in medicine.

[42]  C. Begley,et al.  Using triangulation in nursing research. , 1996, Journal of advanced nursing.

[43]  Suzanne Bakken,et al.  Impact of a physician's order entry (POE) system on physicians' ordering patterns and patient length of stay , 2002, Int. J. Medical Informatics.

[44]  S. Cowman,et al.  Triangulation: a means of reconciliation in nursing research. , 1993, Journal of advanced nursing.

[45]  G. Hartrick,et al.  Health-promoting nursing practice: the demise of the nursing process? , 1996, Journal of advanced nursing.