The reliability of multiple objective measures of surgery and the role of human performance.

BACKGROUND There is a need for reliable and valid objective methods of technical skills in surgery. Six-bench surgical top stations have been combined to assess basic surgical trainees (BSTs) objectively. The current study examines its reliability and validity across repeat sittings. METHODS Eleven surgical trainees (6 senior BSTs and 5 higher surgical trainees [HSTs]) undertook 5 sittings of the 6-station assessment designed to be completed within 90 minutes. The 6 stations consisted of knot tying, suturing, closure of enterotomy, excision of sebaceous cyst, laparoscopic task, and instrument examination. Methods of analysis employed were motion analysis, observation with criteria, and inbuilt simulation metrics. RESULTS On analysis 3 knot tying and suturing stations exhibited significant differences in either time or movement; any difference was over by the second run. The intertest reliabilities were .66, .74, .55, .51, and .65 for the 5 runs. The intratest reliability across repeated sittings varied from .56 to .96. The inter-rater reliability for video assessment varied from .77 to .94. CONCLUSION The assessment is reliable and valid across repeated sittings. Its use in assessment of basic technical skills needs to be encouraged.

[1]  A Darzi,et al.  The challenge of objective assessment of surgical skill. , 2001, American journal of surgery.

[2]  C Sutton,et al.  Learning rate for laparoscopic surgical skills on MIST VR, a virtual reality simulator: quality of human-computer interface. , 1999, Annals of the Royal College of Surgeons of England.

[3]  C. Boreham,et al.  Seasonal variation of selected performance parameters in épée fencers. , 1993, British journal of sports medicine.

[4]  M. Donnelly,et al.  Faculty sensitivity in detecting medical students' clinical competence , 1995 .

[5]  P J Baldwin,et al.  Validity of surgical simulation for the assessment of operative skill , 2001, The British journal of surgery.

[6]  E M Hennig,et al.  The influence of stretching and warm-up exercises on Achilles tendon reflex activity. , 1995, Journal of sports sciences.

[7]  R. Reznick,et al.  Teaching and testing technical skills. , 1993, American journal of surgery.

[8]  M. Pietroni.  The assessment of competence in surgical trainees. , 1993, Annals of the Royal College of Surgeons of England.

[9]  A. Cuschieri,et al.  Reliability of the Dundee Endoscopic Psychomotor Tester (DEPT) for dominant hand performance , 2001, Surgical Endoscopy.

[10]  A Middlebrook,et al.  MIST VR. A laparoscopic surgery procedures trainer and evaluator. , 1997, Studies in health technology and informatics.

[11]  Vtvek Datta,et al.  Motion Analysis in the Assessment of Surgical Skill , 2001 .

[12]  A. Darzi,et al.  Effect of sleep deprivation on surgeons' dexterity on laparoscopy simulator , 1998, The Lancet.

[13]  T. Crofts,et al.  Surgical training: an objective assessment of recent changes for a single health board , 1997, BMJ.

[14]  Y. Koutedakis.  Seasonal Variation in Fitness Parameters in Competitive Athletes , 1995, Sports medicine.

[15]  Ara W. Darzi,et al.  Technical performance: Relation between surgical dexterity and technical knowledge , 2004, World Journal of Surgery.

[16]  C Bulstrode,et al.  Education for educating surgeons , 1996, BMJ.

[17]  J. Kopta,et al.  An approach to the evaluation of operative skills. , 1971, Surgery.

[18]  A. Darzi,et al.  The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. , 2001, Journal of the American College of Surgeons.

[19]  Fiona Smith,et al.  Making a difference taking action. , 2001, Nursing management.

[20]  R. Reznick,et al.  Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE‐format examination , 1998, Academic medicine : journal of the Association of American Medical Colleges.

[21]  T. Grantcharov,et al.  Laparoscopic performance after one night on call in a surgical department: prospective study , 2001, BMJ : British Medical Journal.

[22]  E. Moore,et al.  Does the in-service training examination correlate with clinical performance in surgery? , 1985, Current surgery.

[23]  A. Darzi,et al.  Measurement of Surgical Dexterity Using Motion Analysis of Simple Bench Tasks , 2003, World Journal of Surgery.

[24]  E C DeLand,et al.  Clinical performance versus in-training examinations as measures of surgical competence. , 1980, Surgery.

[25]  R. Reznick,et al.  Testing technical skill via an innovative "bench station" examination. , 1997, American journal of surgery.

[26]  A. Darzi,et al.  The PreOp flexible sigmoidoscopy trainer , 2002, Surgical Endoscopy And Other Interventional Techniques.

[27]  R. Mace,et al.  Relationship between Emotional State and Performance during International Field Hockey Matches , 2000, Perceptual and motor skills.

[28]  R. Newton,et al.  Senior house officers in medicine: postal survey of training and work experience , 1997, BMJ.

[29]  A. Darzi,et al.  Assessing operative skill , 1999, BMJ.

[30]  A. Darzi,et al.  Multiple Objective Measures of Skill (MOMS): A New Approach to the Assessment of Technical Ability in Surgical Trainees , 2003, Annals of surgery.

[31]  C. Stride,et al.  Psychological morbidity and job satisfaction in hospital consultants and junior house officers: multicentre, cross sectional survey , 1998, BMJ.

[32]  D. N. Hendricks,et al.  Effect of time of day on aerobic and anaerobic responses to high-intensity exercise. , 1992, Canadian journal of sport sciences = Journal canadien des sciences du sport.

[33]  Pollock Av.  How do we measure surgical competence , 1996 .