Development of workplace‐based assessments of non‐technical skills in anaesthesia *

Non‐technical skills are recognised as crucial to good anaesthetic practice. We designed and evaluated a specialty‐specific tool to assess non‐technical aspects of trainee performance in theatre, based on a system previously found reliable in a recruitment setting. We compared inter‐rater agreement (multir‐ater kappa) for live assessments in theatre with that in a selection centre and a video‐based rater training exercise. Twenty‐seven trainees participated in the first in‐theatre assessment round and 40 in the second. Round‐ 1 scores had poor inter‐rater agreement (mean kappa = 0.20) and low reliability (generalisability coefficient G = 0.50). A subsequent assessor training exercise showed good inter‐rater agreement, (mean kappa = 0.79) but did not improve performance of the assessment tool when used in round 2 (mean kappa = 0.14, G = 0.42). Inter‐rater agreement in two selection centres (mean kappa = 0.61 and 0.69) exceeded that found in theatre. Assessment tools that perform reliably in controlled settings may not do so in the workplace.

[1]  R. Kearney.  Defining professionalism in anaesthesiology , 2005, Medical education.

[2]  R. Glavin.  Excellence in anesthesiology: the role of nontechnical skills. , 2009, Anesthesiology.

[3]  T. Gale,et al.  Predictive validity of a selection centre testing non-technical skills for recruitment to training in anaesthesia. , 2010, British journal of anaesthesia.

[4]  J C Nelson,et al.  Statistical description of interrater variability in ordinal ratings , 2000, Statistical methods in medical research.

[5]  G. Hocking,et al.  Anaesthesia non-technical skills: Can anaesthetists be trained to reliably use this behavioural marker system in 1 day? , 2010, British journal of anaesthesia.

[6]  J. D. Greaves,et al.  Defining excellence in anaesthesia: the role of personal qualities and practice environment. , 2011, British journal of anaesthesia.

[7]  W. Grove.  Statistical Methods for Rates and Proportions, 2nd ed , 1981 .

[8]  Eric S. Holmboe,et al.  Faculty and the Observation of Trainees’ Clinical Skills: Problems and Opportunities , 2004, Academic medicine : journal of the Association of American Medical Colleges.

[9]  J. Norcini,et al.  Peer assessment of competence , 2003, Medical education.

[10]  Marjan J. B. Govaerts,et al.  Broadening Perspectives on Clinical Performance Assessment: Rethinking the Nature of In-training Assessment , 2007, Advances in health sciences education : theory and practice.

[11]  R. Flin,et al.  The role of non-technical skills in anaesthesia: a review of current literature. , 2002, British journal of anaesthesia.

[12]  P. Farrant,et al.  Assessing the assessments: U.K. dermatology trainees’ views of the workplace assessment tools , 2009, The British journal of dermatology.

[13]  J. D. Greaves,et al.  Watching anaesthetists work: using the professional judgement of consultants to assess the developing clinical competence of trainees. , 2000, British journal of anaesthesia.

[14]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[15]  A. Merry,et al.  Investigation of trainee and specialist reactions to the mini-Clinical Evaluation Exercise in anaesthesia: implications for implementation. , 2009, British journal of anaesthesia.

[16]  J. D. Greaves,et al.  Beyond competence: defining and promoting excellence in anaesthesia , 2010, Anaesthesia.

[17]  B. Everitt,et al.  Statistical methods for rates and proportions , 1973 .

[18]  P. Shrout.  Measurement reliability and agreement in psychiatry , 1998, Statistical methods in medical research.

[19]  E. Ferguson,et al.  Using job analysis to identify core and specific competencies: implications for selection and recruitment , 2008, Medical education.

[20]  Julian Archer,et al.  mini-PAT (Peer Assessment Tool): A Valid Component of a National Assessment Programme in the UK? , 2008, Advances in health sciences education : theory and practice.

[21]  A. Merry,et al.  Mini-clinical evaluation exercise in anaesthesia training. , 2009, British journal of anaesthesia.

[22]  Steven M Downing,et al.  Reliability: on the reproducibility of assessment data , 2004, Medical education.

[23]  J. Norcini.  Current perspectives in assessment: the assessment of performance at work , 2005, Medical education.

[24]  D. Murray,et al.  Simulation-based Assessment in Anesthesiology: Requirements for Practical Implementation , 2010, Anesthesiology.

[25]  Educational tools in the assessment of trainees in anaesthesia , 2011 .

[26]  Nikolaus Gravenstein,et al.  Role of simulators, educational programs, and nontechnical skills in anesthesia resident selection, education, and competency assessment. , 2008, Journal of critical care.

[27]  The struggling resident: avoiding Pygmalion and learned helplessness by using nontechnical performance assessment systems. , 2010, Anesthesiology.

[28]  K. Eva,et al.  The reliability of workplace-based assessment in postgraduate medical education and training: a national evaluation in general practice in the United Kingdom , 2009, Advances in health sciences education : theory and practice.