Board 350 - Research Abstract Gathering Validity Evidence for a New Assessment Tool in a Simulated Environment (Submission #1197)

Introduction/Background Nurse Administered Propofol Sedation (NAPS) for gastroenterological endoscopic procedures has proven to be equally safe or safer than conventional sedation with benzodiazepines/opiates.1 But because of the small therapeutic interval of propofol, the lack of an antagonist and the potentially life threatening side-effects, the administration requires skill and is debated.2–3 A gold standard of skills required for NAPS has been proposed but not established,4 and there are no evidence-based NAPS training requirements. Due to the potentially hazardous nature of NAPS and uncertainty about the necessary level of competency, an assessment tool is needed to objectively judge the adequacy of training and for future certification. The aim of this study was to develop an assessment tool for measuring competency in propofol sedation and to explore the reliability and validity of the tool. Methods Twenty one key items were identified through interviews of six physicians and supported by a review of the literature. The assessment tool (NAPSAT) was revised through consensus discussion meetings in a group of topic experts in a Delphi-like fashion. The group consisted of three physician experts on endoscopy, anaesthesia and educational measures respectively. Consensus on relevance was achieved on 17 items. A 5-point Likert-like rating scale with anchors attached to the one, three and five point was used for 15 items (maximum score=75 points) and a pass/fail checklist for two items. Validity evidence was gathered in a simulation setting. Six experienced nurses and six novice nurses participated in a 2.5-day course on sedation including simulation. The full-scale scenarios were conducted in an operation theatre simulation with a Laerdal SimMan™ Hi-fidelity manikin and software, arranged with an Olympus colonoscope™ and a colonoscopy phantom. The performance of each nurse participant was video-recorded in two scenarios for later assessment using the Laerdal Debrief Viewer™, which gave a total of 24 video recordings. Three content expert raters reviewed and rated the performances. Results Seventy two NAPSAT assessment forms were included. Inter-rater reliabilty (three raters), Cronbach´s &agr; = 0.54, generalizability coefficient = 0.68. The experienced nurses had a higher total score than the novices, 46.5 vs. 61.4, p<0.001. The mean score was significantly higher for the experienced nurses in 14 of the 15 global rating scale items. The provided pass/borderline/fail assessment showed significant difference, p = <0.001, the novices being more likely to fail and the experienced more likely to pass. Conclusion With this study, we tried to encapture the set of skills required to perform a high-quality sedation safely in a simulation. Assessing sedation skills in a simulated environment is feasible but requires meticulous preparation and has both advantages and disadvantages. Simulation provided for a standardized simulation matrix with the scenario and the course of events being relatively homogenous but clinical tells, such as level of consciousness, colour and movements were difficult if not impossible to mimic.4 Hence, some of the skills obtained with clinical experience were lost. The expectation of an adverse event made prediction easier for the novices. Video rating allowed for several raters to rate without being present and with the possibility to review parts over again. But video rating did not provide the same onsite overview as live rating, and required expert knowledge of the procedure and the rating matrix. Furthermore, rater instruction, camera angles of the theatre and sound quality provided obstacles during rating. Overall, NAPSAT showed fair inter-rater reliability and good construct validity. This makes NAPSAT fit for formative assessment and proficiency feedback but high stakes, summative assessment in the clinic should be conducted with care. Gathering validity evidence in a simulated environment was possible, although issues with generalizability should be considered carefully. References 1. Rex DK, Deenadayalu VP, Eid E, Imperiale TF, Walker JA, Sandhu K, Clarke AC, Hillman LC, Horiuchi A, Cohen LB, Heuss LT, Peter S, Beglinger C, Sinnott JA, Welton T, Rofail M, Subei I, Sleven R, Jordan P, Goff J, Gerstenberger PD, Munnings H, Tagle M, Sipe BW, Wehrmann T, Di Palma JA, Occhipinti KE, Barbi E, Riphaus A, Amann ST, Tohda G, McClellan T, Thueson C, Morse J, Meah N.. Endoscopist-directed administration of propofol: a worldwide safety experience. Gastroenterology 2009;137:1229–37. 2. American Society of Anesthesiologists Task Force on Sedation and Analgesia by Non-Anesthesiologists. Practice guidelines for sedation and analgesia by non-anesthesiologists. Anesthesiology 2002;96:1004–17. 3. Perel A. Non-anaesthesiologists should not be allowed to administer propofol for procedural sedation: a Consensus Statement of 21 European National Societies of Anaesthesia. Eur J Anaesthesiol 2011;28:580–4. 4.Dumonceau JM, Riphaus A, Aparicio JR, Beilenhoff U, Knape JT, Ortmann M, Paspatis G, Ponsioen CY, Racz I, Schreiber F, Vilmann P, Wehrmann T, Wientjes C, Walder B; NAAP Task Force Members. European Society of Gastrointestinal Endoscopy, European Society of Gastroenterology and Endoscopy Nurses and Associates, and the European Society of Anaesthesiology Guideline: Non-anaesthesiologist administration of propofol for GI endoscopy. Eur J Anaesthesiol. 2010;27(12):1016–30. 5. Schulz CM, Schneider E, Fritz L, Vockeroth J, Hapfelmeier A, Brandt T, Kochs EF, Schneider G. Visual attention of anaesthetists during simulated critical incidents. Br J Anaesth 2011;106:807–13. Disclosures None.