Development and validation of a data dictionary for a feasibility analysis of emergency department key performance indicators

OBJECTIVES The primary study objective was to describe the development of a data dictionary for a feasibility analysis of 11 emergency department (ED) key performance indicators (KPIs). The secondary objective was to internally validate the data dictionary by measuring the inter-observer agreement between data abstractors at participating study sites. METHODS A list of data variables based on the minimum data set elements relevant to the KPIs was developed by a panel of emergency medicine (EM) specialists and from the EM literature. A summit involving the relevant stakeholders, including ED frontline staff, a health economist, an ED clinical data manager and a health care informatician, was convened. For the feasibility analysis project, each data abstractor was furnished with a copy of the data dictionary and attended a one-hour training session prior to commencing data abstraction. Data was independently abstracted for each KPI by two abstractors at each of 12 participating EDs. Inter-rater agreement between abstractors was calculated using Cohen's kappa and results were reported using the Landis and Koch criteria. RESULTS A data dictionary was developed by creating clear definitions and establishing abstraction instructions for each variable. A total of 43 data variables were included in the study data dictionary: 4 on patient demographics; 19 time variables; 5 outcome variables; 8 ED service and staffing units and 7 medical definitions. A clear definition and a set of data abstraction instructions including data sources were developed for each variable to aid data abstraction during the feasibility analysis. Overall 9,276 ED patient records were used for data abstraction to internally validate the data dictionary. The median Cohen kappa score ranged between 0.56 to 0.81. CONCLUSION There is a continued need to standardize definitions of KPIs for the purpose of comparing ED performance and for research purposes. This is a necessary first step in the implementation of valid and reliable ED performance measures. This study successfully developed an internally valid data dictionary that can be used for day-to-day ED operations and for research purposes.