Are audits wasting resources by measuring the wrong things? A survey of methods used to select audit review criteria

Objectives: This study measured the extent to which a systematic approach was used to select criteria for audit, and identified problems in using such an approach with potential solutions. Design: A questionnaire survey using the Audit Criteria Questionnaire (ACQ), created, piloted, and validated for the purpose. Possible ACQ scores ranged from 0 to 1, indicating how systematically the criteria had been selected and how usable they were. Setting: A stratified random sample of 10 audit leads in each of 83 randomly selected NHS trusts and all practices in each of 11 randomly selected primary care audit group areas in England and Wales. Participants: Audit leads of ongoing audits in each organisation in which a first data collection had started less than 12 months earlier and a second data collection was not completed. Main outcome measures: ACQ scores, problems identified in the audit criteria selection process, and solutions found. Results: The mean ACQ score from all 83 NHS trusts and the 11 primary care audit groups was 0.52 (range 0.0–0.98). There was no difference between mean ACQ scores for criteria used in audits on clinical (0.51) and non-clinical (0.52) topics. The mean ACQ scores from nationally organised audits (0.59, n=33) was higher than for regional (0.51, n=21), local (0.53, n=77), or individual organisation (0.52, n=335) audits. The mean ACQ score for published audit protocols (0.56) was higher than for locally developed audits (0.49). There was no difference in ACQ scores for audits reported by general practices (0.49, n=83) or NHS trusts (0.53, n=383). Problems in criteria selection included difficulties in coordination of staff to undertake the task, lack of evidence, poor access to literature, poor access to high quality data, lack of time, and lack of motivation. Potential solutions include investment in training, protected time, improved access to literature, support staff and availability of published protocols. Conclusions: Methods of selecting review criteria were often less systematic than is desirable. Published usable audit protocols providing evidence based review criteria with information on their provenance enable appropriate review criteria to be selected, so that changes in practice based on these criteria lead to real improvement in quality rather than merely change. The availability and use of high quality audit protocols would be a valuable contribution to the evolution of clinical governance. The ACQ should be developed into a tool to help in selecting appropriate criteria to increase the effectiveness of audit.