Multiple performance measures are needed to evaluate triage systems in the emergency department.

OBJECTIVES Emergency department triage systems can be considered prediction rules with an ordinal outcome, where different directions of misclassification have different clinical consequences. We evaluated strategies to compare the performance of triage systems and aimed to propose a set of performance measures that should be used in future studies. STUDY DESIGN AND SETTING We identified performance measures based on literature review and expert knowledge. Their properties are illustrated in a case study evaluating two triage modifications in a cohort of 14,485 pediatric emergency department visits. Strengths and weaknesses of the performance measures were systematically appraised. RESULTS Commonly reported performance measures are measures of statistical association (34/60 studies) and diagnostic accuracy (17/60 studies). The case study illustrates that none of the performance measures fulfills all criteria for triage evaluation. Decision curves are the performance measures with the most attractive features but require dichotomization. In addition, paired diagnostic accuracy measures can be recommended for dichotomized analysis, and the triage-weighted kappa and Nagelkerke's R2 for ordinal analyses. Other performance measures provide limited additional information. CONCLUSION When comparing modifications of triage systems, decision curves and diagnostic accuracy measures should be used in a dichotomized analysis, and the triage-weighted kappa and Nagelkerke's R2 in an ordinal approach.

[1]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[2]  Henriëtte A Moll,et al.  Challenges in the validation of triage systems at emergency departments. , 2010, Journal of clinical epidemiology.

[3]  Kevin Mackway-Jones,et al.  Emergency triage : Manchester Triage Group , 2013 .

[4]  S. Manzano,et al.  Validity of the Canadian Paediatric Triage and Acuity Scale in a tertiary care hospital. , 2009, CJEM.

[5]  Gerard FitzGerald,et al.  Emergency department triage revisited , 2009, Emergency Medicine Journal.

[6]  H. V. van Stel,et al.  Adjusting weighted kappa for severity of mistriage decreases reported reliability of emergency department triage systems: a comparative study. , 2009, Journal of clinical epidemiology.

[7]  D. Yealy,et al.  Are we asking the right triage questions? , 2013, Annals of emergency medicine.

[8]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[9]  Mark Steyvers,et al.  Choosing a Strictly Proper Scoring Rule , 2013, Decis. Anal..

[10]  Tadahiro Goto,et al.  Machine Learning–Based Prediction of Clinical Outcomes for Children During Emergency Department Triage , 2019, JAMA network open.

[11]  E. Steyerberg,et al.  Accuracy of Triage for Children With Chronic Illness and Infectious Symptoms , 2013, Pediatrics.

[12]  Lukas Faessler,et al.  Performance of the Manchester Triage System in Adult Medical Emergency Patients: A Prospective Cohort Study. , 2016, The Journal of emergency medicine.

[13]  I. V. D. Wulp,et al.  Adjusting weighted kappa for severity of mistriage decreases reported reliability of emergency department triage systems: a comparative study. , 2009 .

[14]  S. Mace,et al.  Emergency Department Overcrowding and Children , 2007, Pediatric emergency care.

[15]  David A Thompson,et al.  Emergency department triage: why we need a research agenda. , 2005, Annals of emergency medicine.

[16]  E. Steyerberg,et al.  The Manchester triage system: improvements for paediatric emergency care , 2012, Emergency Medicine Journal.

[17]  E W Steyerberg,et al.  Polytomous logistic regression analysis could be applied more often in diagnostic research. , 2008, Journal of clinical epidemiology.

[18]  D. Aronsky,et al.  Systematic review of emergency department crowding: causes, effects, and solutions. , 2008, Annals of emergency medicine.

[19]  E. Elkin,et al.  Decision Curve Analysis: A Novel Method for Evaluating Prediction Models , 2006, Medical decision making : an international journal of the Society for Medical Decision Making.

[20]  D. Eitel,et al.  The emergency severity index triage algorithm version 2 is reliable and valid. , 2003, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[21]  Yvonne Vergouwe,et al.  Discrimination ability of prediction models for ordinal outcomes: Relationships between existing measures and a new measure , 2012, Biometrical journal. Biometrische Zeitschrift.

[22]  N. Obuchowski,et al.  Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures , 2010, Epidemiology.

[23]  T. Strout,et al.  Evaluation of the Emergency Severity Index (version 3) triage algorithm in pediatric patients. , 2005, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[24]  André Peralta Santos,et al.  Manchester triage system version II and resource utilisation in the emergency department , 2013, Emergency Medicine Journal.

[25]  E. Steyerberg,et al.  Improving the Manchester Triage System for Pediatric Emergency Care: An International Multicenter Study , 2014, PloS one.

[26]  Manchester triage system in paediatric emergency care: prospective observational study , 2008, BMJ : British Medical Journal.