The feasibility of a multi-format Web-based assessment of physicians' communication skills.

OBJECTIVE Little is known about the best approaches and format for measuring physicians' communication skills in an online environment. This study examines the reliability and validity of scores from two Web-based communication skill assessment formats. METHODS We created two online communication skill assessment formats: (a) MCQ (multiple-choice questions) consisting of video-based multiple-choice questions; (b) multi-format including video-based multiple-choice questions with rationales, Likert-type scales, and free text responses of what physicians would say to a patient. We randomized 100 general internists to each test format. Peer and patient ratings collected via the American Board of Internal Medicine (ABIM) served as validity sources. RESULTS Seventy-seven internists completed the tests (MCQ: 38; multi-format: 39). The adjusted reliability was 0.74 for both formats. Excellent communicators, as based on their peer and patient ratings, performed slightly better on both tests than adequate communicators, though this difference was not statistically significant. Physicians in both groups rated test format innovative (4.2 out of 5.0). CONCLUSION The acceptable reliability and participants' overall positive experiences point to the value of ongoing research into rigorous Web-based communication skills assessment. PRACTICE IMPLICATIONS With efficient and reliable scoring, the Web offers an important way to measure and potentially enhance physicians' communication skills.

[1]  Jonathan Sherbino,et al.  The role of assessment in competency-based medical education , 2010, Medical teacher.

[2]  G. Makoul,et al.  Essential Elements of Communication in Medical Encounters: The Kalamazoo Consensus Statement , 2001, Academic medicine : journal of the Association of American Medical Colleges.

[3]  K. Mazor,et al.  The Video-Based Test of Communication Skills: Description, Development, and Preliminary Findings , 2007, Teaching and learning in medicine.

[4]  J. D. Donnison-Speijer,et al.  Assessment of medical communication skills by computer: assessment method and student experiences , 2004, Medical education.

[5]  R. Reznick,et al.  How surgeons disclose medical errors to patients: a study using standardized patients. , 2005, Surgery.

[6]  Comparing narrative and multiple‐choice formats in online communication skill assessment , 2009, Medical education.

[7]  C. Fertleman,et al.  Video improved role play for teaching communication skills , 2005, Medical education.

[8]  Jocelyn Lockyer,et al.  Assessment of a matched‐pair instrument to examine doctor−patient communication skills in practising doctors , 2007, Medical education.

[9]  Cees P. M. van der Vleuten,et al.  Different written assessment methods: what can be said about their strengths and weaknesses? , 2004 .

[10]  R. Frankel,et al.  Assessing Competence in Communication and Interpersonal Skills: The Kalamazoo II Report , 2004, Academic medicine : journal of the Association of American Medical Colleges.

[11]  P Wallace,et al.  Following the threads of an innovation: the history of standardized patients in medical education. , 1997, Caduceus.

[12]  Robert A. Virzi,et al.  Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough? , 1992 .

[13]  David B. Swanson,et al.  Assessment of clinical skills with standardized patients: State of the art , 1990 .

[14]  G. Humphris,et al.  Assessing the development of communication skills in undergraduate medical students , 2001, Medical education.

[15]  Sara Kim,et al.  A Web-Based Team-Oriented Medical Error Communication Assessment Tool: Development, Preliminary Reliability, Validity, and User Ratings , 2011, Teaching and learning in medicine.

[16]  J. Bensing,et al.  The effectiveness of a computer‐assisted instruction programme on communication skills of medical specialists in oncology , 2002, Medical education.

[17]  R. Lipner,et al.  The Value of Patient and Peer Ratings in Recertification , 2002, Academic medicine : journal of the Association of American Medical Colleges.

[18]  Jakob Nielsen,et al.  Estimating the number of subjects needed for a thinking aloud test , 1994, Int. J. Hum. Comput. Stud..

[19]  Debra L Roter,et al.  Use of an innovative video feedback technique to enhance communication skills training , 2004, Medical education.

[20]  Delia S Anderson,et al.  Communication assessment using the common ground instrument: psychometric properties. , 2004, Family medicine.

[21]  A. Dietrich,et al.  Recognizing and managing depression in primary care: a standardized patient study. , 1999, The Journal of family practice.

[22]  G. Norman,et al.  Conceptual and methodological issues in studies comparing assessment formats , 1996 .

[23]  William C. Ward,et al.  A Comparison of Free-Response and Multiple-Choice Forms of Verbal Aptitude Tests , 1981 .

[24]  Bill Curtis,et al.  Applying Discount Usability Engineering , 1995, IEEE Softw..