Assessing inter-rater agreement of environmental audit data in a matched case-control study on bicycling injuries

Background Environmental audit tools must be reliable in order to accurately estimate the association between built environmental characteristics and bicycling injury risk. Objective To examine the inter-rater agreement of a built environment audit tool within a case-control study on the environmental determinants of bicycling injuries. Methods Auditor pairs visited locations where bicycling injuries occurred and independently recorded location characteristics using the Systematic Pedestrian and Cyclist Environmental Scan (SPACES). Two case groups were defined: (1) where a bicyclist was struck by a motor-vehicle (MV) and (2) where the bicyclist's injuries required hospitalisation. The two corresponding control groups were (1) where non-MV bicycle-related injuries occurred and (2) where minor bicycle-related injuries occurred. Inter-rater reliability of each item on the tool was assessed using observed agreement and κ with 95% CI. Results Ninety-seven locations were audited. Inter-observer agreement was generally high (≥95%); most items had a 1–2% difference in responses. Items with ≥5% differences between raters included path condition, slope and obstructions. For land use, path and roadway characteristics, κ ranged from 0.3 for presence of offices and cleanliness to 0.9 for schools and number of lanes; overall, 78% of items had at least substantial agreement (κ≥0.61). For bicyclists struck by a MV the proportion of items with substantial agreement was 60%, compared with 73% for non-MV related injuries. For hospitalisations and minor bicycle-related injuries, 76% of items had substantial agreement. Conclusions Agreement was substantial for most, but not all SPACES items. The SPACES provides reliable quantitative descriptions of built environmental characteristics at bicycling injury locations.