Simulations in evaluation of training: a medical example using standardised patients

Abstract In the evaluation of the effectiveness of continuing medical education, observation of a change in physicians' practice offers sound evidence of learning. Such measurements are technically challenging and carry ethical issues of patient confidentiality and vulnerability. This study aimed to address these issues by evaluating the impact of a training program on the performance of family physicians in a simulated consultation with adolescents trained as standardised patients (SPs). One hundred and eight physicians were randomised into intervention or control group, and were tested pre-training, 6-months and 12-months post-training. Physicians rated self-perceived competency, SPs rated rapport and satisfaction, and confidentiality discussions and independent faculty observers rated clinical competency. All measures detected significant impact of the training on physicians' performance except for the SP's subjective rating on rapport and satisfaction. Results indicate the method is feasible and sensitive to changes in performance. Further research is needed to clarify questions raised about the SPs' subjective ratings.

[1]  G. Norman,et al.  Reliability and learning from the objective structured clinical examination , 1990, Medical education.

[2]  Evaluation of the effectiveness of an educational intervention for general practitioners in adolescent health care: randomised controlled trial. , 2000, BMJ.

[3]  H. Barrows,et al.  Use of Standardized Patients in Clinical Assessments: Recent Developments and Measurement Findings , 1994 .

[4]  F. Degruy,et al.  Stability of standardized patients' performance in a study of clinical decision making. , 1995, Family medicine.

[5]  Continuing medical education: Part 1. Introduction to the series , 1988, The Medical journal of Australia.

[6]  L. Sanci,et al.  Adolescent health care: perspectives of Victorian general practitioners , 1995, The Medical journal of Australia.

[7]  G. Norman,et al.  Pitfalls in the pursuit of objectivity: issues of reliability , 1991, Medical education.

[8]  J. Colliver,et al.  Effect of Repeated Simulations by Standardized Patients on Intercase Reliability. , 1991 .

[9]  David B. Swanson,et al.  Assessment of clinical skills with standardized patients: State of the art , 1990 .

[10]  F Sturmans,et al.  Assessment of the performance of general practitioners by the use of standardized (simulated) patients. , 1991, The British journal of general practice : the journal of the Royal College of General Practitioners.

[11]  J. Ward.  Continuing medical education. Part 4. Evaluation of continuing medical education. , 1988, The Medical journal of Australia.

[12]  Valerie J. Caracelli,et al.  Advances in mixed-method evaluation : the challenges and benefits of integrating diverse paradigms , 1997 .

[13]  P Tugwell,et al.  A comparison of resident performance on real and simulated patients. , 1982, Journal of medical education.

[14]  H. Barrows,et al.  Three studies of the effect of multiple standardized patients on intercase reliability of five standardized‐patient examinations , 1990 .

[15]  J. Colliver,et al.  Assessing clinical performance with standardized patients. , 1997, JAMA.

[16]  O'Hagan Jj,et al.  The use of simulated patients in the assessment of actual clinical performance in general practice. , 1986 .

[17]  G. Norman,et al.  Objective measurement of clinical performance , 1985, Medical education.

[18]  Geoff N Masters,et al.  Competency based assessment in the professions , 1990 .

[19]  H. Barrows.  An overview of the uses of standardized patients for teaching and evaluating clinical skills. AAMC , 1993, Academic medicine : journal of the Association of American Medical Colleges.

[20]  R. Pill,et al.  Potential of using simulated patients to study the performance of general practitioners. , 1993, The British journal of general practice : the journal of the Royal College of General Practitioners.

[21]  Michael Eraut,et al.  Concepts of competence , 1998 .

[22]  J. Colliver.  Validation of standardized-patient assessment: a meaning for clinical competence. , 1995, Academic medicine : journal of the Association of American Medical Colleges.

[23]  H. Barrows,et al.  Direct, standardized assessment of clinical competence , 1987, Medical education.

[24]  R. A. Melker,et al.  Simulated patients in assessing consultation skills of trainees in general practice vocational training: a validity study , 1994, Medical education.

[25]  A D Oxman,et al.  Changing physician performance. A systematic review of the effect of continuing medical education strategies. , 1995, JAMA.

[26]  L. Sanci,et al.  Barriers to effective primary health care for adolescents , 1996, The Medical journal of Australia.

[27]  Andrew Gonczi,et al.  Establishing competency-based standards in the professions , 1990 .

[28]  H. Barrows,et al.  Standardized (simulated) patients' accuracy in recording clinical performance check‐list items , 1992, Medical education.

[29]  C. Woodward,et al.  Measuring physicians' performances by using simulated patients. , 1985, Journal of medical education.

[30]  D. Goldberg,et al.  Management of somatic presentations of psychiatric illness in general medical settings: evaluation of a new training course for general practitioners , 1992, Medical education.

[31]  E. Petrusa.  The effect of number of cases on performance on a standardized multiple-station clinical examination. , 1987, Journal of medical education.

[32]  T. Usherwood,et al.  Evaluation of a training package in the assessment and management of depression in primary care , 1998, Medical education.

[33]  Nicholas M. Sanders.  Evaluation of training by trainers , 1989 .

[34]  F Sturmans,et al.  Does competence of general practitioners predict their performance? Comparison between examination setting and actual practice. , 1991, BMJ.

[35]  Robert O. Brinkerhoff,et al.  Evaluating Training Programs in Business and Industry. , 1989 .

[36]  M. Nelson.  The Physician as Learner: Linking Research to Practice , 1995 .

[37]  Robyn Tamblyn,et al.  Sources of unreliability and bias in standardized‐patient rating , 1991 .

[38]  F. Sturmans,et al.  A method for introducing standardized (simulated) patients into general practice consultations. , 1991, The British journal of general practice : the journal of the Royal College of General Practitioners.

[39]  R. Tamblyn,et al.  Factors associated with the accuracy of standardized patient presentation , 1990, Academic medicine : journal of the Association of American Medical Colleges.

[40]  J. Rethans,et al.  Simulated patients in general practice: a different look at the consultation. , 1987, British medical journal.

[41]  D. Steward,et al.  An assessment of the consistency and accuracy of standardized patients' simulations. , 1987, Journal of medical education.

[42]  P J Sanazaro,et al.  Determining Physicians' Performance , 1983, Evaluation & the health professions.

[43]  Geoffrey R. Norman,et al.  Performance-Based Assessment: Lessons From the Health Professions , 1995 .