Validation of a Method to Assess the Clinical Impact of Electronic Knowledge Resources

We previously developed a method to assess the clinical impact of electronic knowledge resources. This method has potential to advance e-health research by serving as a new outcome measure in the domain of clinical decision support or computer mediated communication. In this article, we outline how such a method can be validated. The validation includes a longitudinal field research study using a mixed methods approach. Doctors will use our method to complete a computerized impact assessment questionnaire for items of information they retrieve. This will reveal cognitive impacts and application of information in practice. Subsequently, doctors will be interviewed on their most recent information searches using log-stimulated recall. Archives and observations complement interviews. These analyses serve to validate the method for assessing the clinical impact of electronic knowledge resources.

[1]  Amit X. Garg,et al.  Application of Information Technology: McMaster PLUS: A Cluster Randomized Clinical Trial of an Intervention to Accelerate Clinical Use of Evidence-based Information from Digital Libraries , 2006, J. Am. Medical Informatics Assoc..

[2]  Donald O. Case,et al.  Looking for Information: A Survey of Research on Information Seeking, Needs and Behavior , 2012 .

[3]  Bengt Muthén,et al.  Multilevel Factor Analysis of Class and Student Achievement Components , 1991 .

[4]  P. Tugwell,et al.  Electronic access to health information and/or knowledge by health professionals to improve practice and patient care , 2004 .

[5]  B. Muthén,et al.  Multilevel Covariance Structure Analysis , 1994 .

[6]  William Snyder,et al.  Cultivating Communities of Practice: A Guide to Managing Knowledge , 2002 .

[7]  Roland M. Grad,et al.  Cognitive Impact Assessment of Electronic Knowledge Resources: A Mixed Methods Evaluation Study of a Handheld Prototype , 2006, AMIA.

[8]  A. Macaulay,et al.  Moving the Frontiers Forward: Incorporating Community-Based Participatory Research Into Practice-Based Research Networks , 2006, The Annals of Family Medicine.

[9]  Jochen R. Moehr,et al.  Terminological Problems in Information Retrieval , 2003, Journal of Medical Systems.

[10]  Pranab Kumar Sen,et al.  Within‐cluster resampling , 2001 .

[11]  S. Woolf.  The need for perspective in evidence-based medicine. , 1999, JAMA.

[12]  Shelley Martin,et al.  More than half of MDs under age 35 now using PDAs. , 2003, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[13]  C. Argyris,et al.  Argyris, Chris, Robert Putnam, and Diane Maclain Smith, Action Science: Concepts, Methods, and Skills for Research and Intervention . San Franciso: Jossey-Bass, 1985. , 1985 .

[14]  G. Leung,et al.  Randomised controlled trial of clinical decision support tools to improve learning of evidence based medicine in medical students , 2003, BMJ : British Medical Journal.

[15]  J. Swan,et al.  The Construction of `Communities of Practice' in the Management of Innovation , 2002 .

[16]  E. Wenger.  Communities of Practice: Learning, Meaning, and Identity , 1998 .

[17]  J. McNally.  Self-Directed Learning, A Guide for Learners and Teachers , 1976 .

[18]  W Hersh,et al.  "A world of knowledge at your fingertips": the promise, reality, and future directions of on-line information retrieval. , 1999, Academic medicine : journal of the Association of American Medical Colleges.

[19]  Roland M. Grad,et al.  A New Impact Assessment Method to Evaluate Knowledge Resources , 2005, AMIA.

[20]  R. Haynes,et al.  Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. , 1998, JAMA.

[21]  Paul B. Kantor,et al.  Studying the Value of Library and Information Services. Part I: Establishing a Theoretical Framework. , 1997 .

[22]  Farah Magrabi,et al.  Clinicians' Assessments of the Usefulness of Online Evidence to Answer Clinical Questions , 2004, MedInfo.

[23]  Keith H. Nuechterlein,et al.  An Illustration of Multilevel Factor Analysis , 2005, Journal of personality assessment.

[24]  J. Parboosingh,et al.  Physician communities of practice: where learning and practice are inseparable. , 2002, The Journal of continuing education in the health professions.

[25]  H. Bradbury,et al.  Handbook of action research , 2006 .

[26]  S. Shiffman,et al.  Ecological Momentary Assessment (Ema) in Behavioral Medicine , 1994 .

[27]  John Seely Brown,et al.  Book Reviews : The Social Life of Information By John Seely Brown and Paul Duguid. Boston: Harvard Business School Press, 2000. 320 pages , 2000 .

[28]  N. Roberts,et al.  Value-added processes in information systems , 1986 .

[29]  Farah Magrabi,et al.  General practitioners' use of online evidence during consultations , 2005, Int. J. Medical Informatics.

[30]  Jerome A Osheroff,et al.  Research Paper: Answering Physicians' Clinical Questions: Obstacles and Potential Solutions , 2005, J. Am. Medical Informatics Assoc..

[31]  T. Greenhalgh,et al.  Promoting the skills of knowledge translation in an online master of science course in primary health care , 2006, The Journal of continuing education in the health professions.

[32]  Allen F. Shaughnessy,et al.  Papers of two former Medical Library Association presidents cataloged in Philadelphia. , 2002 .

[33]  Peter Reason,et al.  CO-OPERATIVE INQUIRY, PARTICIPATORY ACTION RESEARCH & ACTION INQUIRY: THREE APPROACHES TO PARTICIPATIVE INQUIRY , 1994 .

[34]  K. El Emam,et al.  Who’s Using PDAs? Estimates of PDA Use by Health Care Providers: A Systematic Review of Surveys , 2006, Journal of medical Internet research.

[35]  Etienne Wenger,et al.  Communities of Practice: Learning, Meaning, and Identity , 1998 .

[36]  R. Haynes,et al.  Effects of Computer-Based Clinical Decision Support Systems on Physician Performance and Patient Outcomes , 1998 .

[37]  Johanna I. Westbrook,et al.  Do online information retrieval systems help experienced clinicians answer clinical questions? , 2005, Journal of the American Medical Informatics Association : JAMIA.

[38]  Farah Magrabi,et al.  What factors are associated with the integration of evidence retrieval technology into routine general practice settings? , 2007, Int. J. Medical Informatics.

[39]  P. Pluye,et al.  How information retrieval technology may impact on physician practice: an organizational case study in family medicine. , 2004, Journal of evaluation in clinical practice.

[40]  Joan C. Bartlett,et al.  Seven reasons why health professionals search clinical information-retrieval technology (CIRT): toward an organizational model. , 2007, Journal of evaluation in clinical practice.

[41]  T. D. Wilson,et al.  Models in information behaviour research , 1999, J. Documentation.

[42]  A. McAlearney,et al.  Doctors' experience with handheld computers in clinical practice: qualitative study , 2004, BMJ : British Medical Journal.

[43]  Jacques Lesourne,et al.  Le nouveau management : la decision par les ordinateurs , 1980 .

[44]  N. Denzin,et al.  Handbook of Qualitative Research , 1994 .

[45]  Paul B. Kantor,et al.  Studying the Value of Library and Information Services. Part I. Establishing a Theoretical Framework , 1997, J. Am. Soc. Inf. Sci..

[46]  Nancy L. Wilczynski,et al.  The medical literature as a resource for health care practice , 1995 .

[47]  H B Slotnick,et al.  How doctors learn: the role of clinical problems across the medical school‐to‐practice continuum , 1996, Academic medicine : journal of the Association of American Medical Colleges.

[48]  L. Green,et al.  Validating an instrument for selecting interventions to change physician practice patterns: a Michigan Consortium for Family Practice Research study. , 2002, The Journal of family practice.

[49]  Paul Duguid,et al.  The social life of information - Table of contents , 2000 .

[50]  Johanna I Westbrook,et al.  The Impact of an Online Evidence System on Confidence in Decision Making in a Controlled Setting , 2005, Medical decision making : an international journal of the Society for Medical Decision Making.

[51]  Paul N. Gorman,et al.  Information needs of physicians , 1995 .

[52]  David Slawson,et al.  Update on InfoRetriever software. , 2002, Journal of the Medical Library Association : JMLA.

[53]  D. Christakis,et al.  Pediatricians' use of and attitudes about personal digital assistants. , 2004, Pediatrics.

[54]  L. Gruppen,et al.  A Controlled Comparison Study of the Efficacy of Training Medical Students in Evidence-Based Medicine Literature Searching Skills , 2005, Academic medicine : journal of the Association of American Medical Colleges.

[55]  Robyn Tamblyn,et al.  Assessing the impact of clinical information-retrieval technology in a family practice residency. , 2005, Journal of evaluation in clinical practice.

[56]  Pierre Pluye,et al.  Effect of a PDA-assisted evidence-based medicine course on knowledge of common clinical problems. , 2005, Family medicine.

[57]  H. Mcdonald,et al.  Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. , 2005, JAMA.

[58]  Roland M. Grad,et al.  Impact of clinical information-retrieval technology on physicians: A literature review of quantitative, qualitative and mixed methods studies , 2005, Int. J. Medical Informatics.

[59]  E. Wenger,et al.  Cultivating Communities of Practice: A Guide to Managing Knowledge , 2002 .

[60]  J. Wyatt,et al.  Basic concepts in medical informatics , 2002, Journal of epidemiology and community health.

[61]  Robert S. Taylor,et al.  Value-Added Processes in Information Systems , 1987 .

[62]  David W. Bates,et al.  Research Paper: Clinician Use of a Palmtop Drug Reference Guide , 2002, J. Am. Medical Informatics Assoc..

[63]  Glenn Robert,et al.  Knowledge management and communities of practice in the private sector: lessons for modernizing the National Health Service in England and Wales , 2002 .

[64]  Sharon M. Gray.  Looking for Information: A Survey of Research on Information Seeking, Needs, and Behavior. , 2003 .

[65]  D. Schoen.  Educating the reflective practitioner , 1987 .

[66]  William R. Hersh,et al.  Information Retrieval: A Health and Biomedical Perspective , 2002 .