Reliability and validity of the Manchester Triage System in a general emergency department patient population in the Netherlands: results of a simulation study

Objective: To assess the reliability and validity of the Manchester Triage System (MTS) in a general emergency department patient population. Methods: A prospective evaluation study was conducted in two general hospitals in the Netherlands. Emergency department nurses from both hospitals triaged 50 patient vignettes into one of five triage categories in the MTS. Triage ratings were compared with the ratings of two Dutch MTS experts to measure inter-rater reliability. Nineteen days after triaging the patient vignettes, triage nurses were asked to rate the same vignettes again to measure test-retest reliability. Reliability in relation to the work experience of emergency department nurses was also studied. Validity was assessed by calculating percentages for overtriage, undertriage, sensitivity and specificity. Results: Inter-rater reliability was “substantial” (weighted kappa 0.62 (95% CI 0.60 to 0.65)) and test-retest reliability was high (intraclass correlation coefficient 0.75 (95% CI 0.72 to 0.77)). No significant association was found between the experience of emergency department nurses and the reliability score (kappa). Undertriage occurred more frequently than overtriage, especially in elderly patients (25.3% vs 7.6%). Sensitivity for urgent patients in the MTS was 53.2% and specificity was 95.1%. The patient vignettes representing children aged <16 years revealed a higher sensitivity (83.3%). Conclusions: Inter-rater reliability is “moderate” to “substantial” and test-retest reliability is high. The reliability of the MTS is not influenced by nurses’ work experience. Undertriage mainly occurs in the MTS categories orange and yellow. The MTS is more sensitive for children who need immediate or urgent care than for other patients in the emergency department.