Comparison of proficiency in an anesthesiology course across distinct medical student cohorts: Psychometric approaches to test equating

Background: Examinations are necessary for assessment of student proficiency in medical education, but comparison of achievement across different cohorts in different tests is challenging. We applied psychometric test equating methods to compare student proficiency in two different examinations for a clinical anesthesiology course. Methods: Each examination contained 50 multiple choice items and nine common items were identified from the two examinations (administered in 2011 and 2012). The common item design was used for test equating. Two psychometric test‐equating approaches, chained linear equating and item response theory, were used to compare student proficiency in anesthesiology across distinct medical student cohorts. Raw scores from the 2012 test were linearly transformed to the 2011 scale using the chained method, and then Rasch analysis was applied to calibrate examinee ability and item difficulty in the two examinations on a common scale. Results: Both the linear equating method and Rasch analysis indicated that students in the 2011 examination performed better than those who took the 2012 examination (both p < 0.001). Rasch analysis revealed that the range of student ability was between −0.53 and 4.16, while the difficulty of all items ranged from −5.25 to 6.32. No significant difference in mean item difficulty was noted among the common items and other items in the two examinations. Conclusion: Although both the chained linear equating method and Rasch analysis can be readily applied to practical test‐equating issues in medical education, Rasch analysis exhibited more versatility in test parameter estimation and item bank development for clinical curriculums.