Hitting and missing targets by ambulance services for emergency calls: effects of different systems of performance measurement within the UK

Following devolution, differences developed between UK countries in systems of measuring performance against a common target that ambulance services ought to respond to 75% of calls for what may be immediately life threatening emergencies (category A calls) within 8 minutes. Only in England was this target integral to a ranking system of ‘star rating’, which inflicted reputational damage on services that failed to hit targets, and only in England has this target been met. In other countries, the target has been missed by such large margins that services would have been publicly reported as failing, if they had been covered by the English system of star ratings. The paper argues that this case-study adds to evidence from comparisons of different systems of hospital performance measurement that, to have an effect, these systems need to be designed to inflict reputational damage on those that have performed poorly; and it explores implications of this hypothesis. The paper also asks questions about the adequacy of systems of performance measurement of ambulance services in UK countries.

[1]  John N. Friedman,et al.  Performance Improvement and Performance Dysfunction: An Empirical Examination of Impacts of the Emergency Room Wait-Time Target in the English National Health Service , 2007 .

[2]  K. Abbasi “Butchers and gropers” , 1998, BMJ.

[3]  B. Hutter Risk regulation and health care , 2008 .

[4]  I. Snelling,et al.  Do star ratings really reflect hospital performance? , 2003, Journal of health organization and management.

[5]  M. Chassin,et al.  Achieving and sustaining improved quality: lessons from New York State and cardiac surgery. , 2002, Health affairs.

[6]  Peter C. Smith,et al.  Performance Measurement in Health Care: History, Challenges and Prospects , 2005 .

[7]  G. Bevan Changing paradigms of governance and regulation of quality of healthcare in England , 2008 .

[8]  Charles Goodhart,et al.  Monetary Theory and Practice , 1984 .

[9]  L. Iezzoni 100 Apples Divided by 15 Red Herrings: A Cautionary Tale from the Mid-19th Century on Comparing Hospital Mortality Rates , 1996, Annals of Internal Medicine.

[10]  J. Pell,et al.  Effect of reducing ambulance response times on deaths from out of hospital cardiac arrest: cohort study , 2001, BMJ : British Medical Journal.

[11]  E. Codman Uniformity in Hospital Morbidity Reports , 1917 .

[12]  J. Grand Motivation, Agency, and Public Policy: Of Knights and Knaves, Pawns and Queens , 2003 .

[13]  David J. Spiegelhalter,et al.  Surgical audit: statistical lessons from Nightingale and Codman , 1999 .

[14]  Gwyn Bevan,et al.  Setting Targets for Health Care Performance: Lessons from a Case Study of the English NHS , 2006, National Institute Economic Review.

[15]  Great Britain. Foreign Office.,et al.  Department of Health and Social Security (Social Security) , 1971 .

[16]  P Slovic,et al.  Making health care quality reports easier to use. , 2001, The Joint Commission journal on quality improvement.

[17]  C. Hood,et al.  Public Service Management by Numbers: Why Does it Vary? Where Has it Come From? What Are the Gaps and the Puzzles? , 2007 .

[18]  R. Lilford,et al.  Cross sectional study of performance indicators for English Primary Care Trusts: testing construct validity and identifying explanatory variables , 2006, BMC Health Services Research.

[19]  Jack V Tu,et al.  Measuring and reducing waiting times: a cross-national comparison of strategies. , 2007, Health affairs.

[20]  W. Diebold,et al.  Soviet industry from Stalin to Gorbachev , 1988 .

[21]  M FISHBEIN,et al.  Health and social security. , 1948, Journal of the American Medical Association.

[22]  N. Wintfeld,et al.  Report cards on cardiac surgeons. Assessing New York State's approach. , 1995, The New England journal of medicine.

[23]  Peter H. Millard,et al.  Balancing the NHS balanced scorecard! , 2008, Eur. J. Oper. Res..

[24]  C. Hood,et al.  Gaming in Targetworld: The Targets Approach to Managing British Public Services , 2006 .

[25]  Gwyn Bevan,et al.  Targets, inspections, and transparency , 2004, BMJ : British Medical Journal.

[26]  A. Donabedian Evaluating the quality of medical care. 1966. , 1966, The Milbank quarterly.

[27]  N. Mays,et al.  Effect of diverging policy across the NHS , 2005, BMJ : British Medical Journal.

[28]  Martin Marshall,et al.  Cultures for Performance in Health Care , 2004 .

[29]  R. Horton Why is Ian Kennedy's Healthcare Commission damaging NHS care? , 2004, The Lancet.

[30]  C. Hood,et al.  What's measured is what matters: targets and gaming in the English public health care system , 2006 .

[31]  Paul G Shekelle,et al.  Public reporting on quality in the United States and the United Kingdom. , 2003, Health affairs.

[32]  James E. Radcliffe,et al.  Performance Measurement and the English Ambulance Service , 2007 .

[33]  L. Carroll,et al.  Alice's Adventures in Wonderland: Princeton University Press , 2015 .

[34]  Anne West,et al.  Publishing School Examination Results in England: Incentives and consequences , 2000 .

[35]  J. Hibbard,et al.  It Isn't Just about Choice: The Potential of a Public Performance Report to Affect the Public Image of Hospitals , 2005, Medical care research and review : MCRR.

[36]  J. Hibbard,et al.  Condition-Specific Performance Information: Assessing Salience, Comprehension, and Approaches for Communicating Quality , 1996, Health care financing review.

[37]  Peter C Smith,et al.  Performance management in British health care: will it deliver? , 2002, Health affairs.

[38]  J. Hibbard,et al.  Hospital performance reports: impact on quality, market share, and reputation. , 2005, Health affairs.

[39]  Russell Mannion,et al.  Performance measurement and improvement in health care. , 2002, Applied health economics and health policy.

[40]  A. Street,et al.  Do targets matter? A comparison of English and Welsh National Health priorities. , 2007, Health economics.

[41]  Rowena Jacobs,et al.  How Do Performance Indicators Add Up? An Examination of Composite Indicators in Public Services , 2007 .

[42]  Burkhard Schafer,et al.  All changed, changed utterly? , 2011, Datenschutz und Datensicherheit - DuD.

[43]  The Performance Assessment Framework: experiences and perceptions of NHS Scotland , 2004 .

[44]  Judith H Hibbard,et al.  Consumer Competencies and the Use of Comparative Quality Information , 2007, Medical care research and review : MCRR.

[45]  David J Spiegelhalter,et al.  Funnel plots for comparing institutional performance , 2005, Statistics in medicine.

[46]  Donald M. Berwick,et al.  Connections Between Quality Measurement and Improvement , 2003, Medical care.

[47]  R. Mannion,et al.  Decentralisation strategies and provider incentives in healthcare , 2005 .

[48]  A. Nove The Problem of "Success Indicators" in Soviet Industry , 1958 .

[49]  Matt Sutton,et al.  Did 'Targets and Terror' Reduce Waiting Times in England for Hospital Care? , 2008 .

[50]  Michael Barber,et al.  Instruction to deliver : Tony Blair, public services and the challenge of achieving targets , 2007 .

[51]  R. Mannion,et al.  Impact of published clinical outcomes data: case study in NHS hospital trusts , 2001, BMJ : British Medical Journal.

[52]  C. K. Mertz,et al.  Less Is More in Presenting Quality Information to Consumers , 2007, Medical care research and review : MCRR.

[53]  Peter C Smith,et al.  On the unintended consequences of publishing performance data in the public sector , 1995 .

[54]  J. Hibbard What Can We Say about the Impact of Public Reporting? Inconsistent Execution Yields Variable Results , 2008, Annals of Internal Medicine.

[55]  M. O’Mahony,et al.  Metrics, Targets and Performance , 2006, National Institute Economic Review.

[56]  R. Mannion,et al.  Public disclosure of comparative clinical performance data: lessons from the Scottish experience. , 2003, Journal of evaluation in clinical practice.

[57]  Gwyn Bevan Setting Targets for Health Care Performance: Lessons from a Case Study of the English NHS , 2006, National Institute Economic Review.

[58]  G. Bevan,et al.  Structure and logic of regulation and governance of quality of health care: was OFSTED a model for the Commission for Health Improvement? , 2006, Health Economics, Policy and Law.

[59]  Charles Goodhart,et al.  Monetary Theory and Practice: The UK Experience , 1983 .

[60]  Why not give consumers a framework for understanding quality? , 2004, Joint Commission journal on quality and safety.

[61]  S. Kelman,et al.  Effort as Investment: Analyzing the Response to Incentives , 2007 .

[62]  P Slovic,et al.  Informing consumer decisions in health care: implications from decision-making research. , 1997, The Milbank quarterly.

[63]  Neil Carter,et al.  How Organisations Measure Success: The Use of Performance Indicators in Government , 1992 .

[64]  E. Ferlie,et al.  The Oxford Handbook of Public Management , 2007 .

[65]  J. Dixon,et al.  The NHS plan , 2000, BMJ : British Medical Journal.

[66]  R Smith,et al.  All changed, changed utterly , 1998, BMJ.

[67]  Thomas A. Louis,et al.  League tables and their limitations: Statistical issues in comparisons of institutional performance - Discussion , 1996 .

[68]  D. Spiegelhalter,et al.  Reliability of league tables of in vitro fertilisation clinics: retrospective analysis of live birth rates. , 1998, BMJ.

[69]  David Spiegelhalter The mystery of the lost star: A statistical detective story , 2005 .

[70]  Hans de Bruijn,et al.  Managing Performance in the Public Sector , 2002 .

[71]  Paul R. Milgrom,et al.  Multitask Principal–Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design , 1991 .

[72]  P. Shekelle,et al.  Systematic Review: The Evidence That Publishing Patient Care Performance Data Improves Quality of Care , 2008, Annals of Internal Medicine.

[73]  Martin Marshall,et al.  Impact of star performance ratings in English acute hospital trusts , 2005, Journal of health services research & policy.

[74]  G. Bevan,et al.  The interplay between economic and political logics: path dependency in health care in England. , 2005, Journal of health politics, policy and law.

[75]  K. Rowan,et al.  Hospitals' star ratings and clinical outcomes: ecological study , 2004, BMJ : British Medical Journal.

[76]  José Maria Paganni,et al.  Evaluating the quality of medical care. , 1966, The Milbank Memorial Fund quarterly.

[77]  J. L. Grand,et al.  Motivation, Agency, and Public Policy , 2003 .

[78]  S. Allender,et al.  Coronary heart disease statistics. , 2008 .

[79]  C. Propper,et al.  Explaining Differences in Hospital Performance: Does the Answer Lie in the Labour Market? , 2003 .

[80]  Rowena Jacobs,et al.  Exploring the determinants of NHS performance ratings: lessons for performance assessment systems , 2006, Journal of health services research & policy.

[81]  S. Leatherman,et al.  Regulation and quality improvement A review of the evidence , 2006 .

[82]  P. Riddell The Unfulfilled Prime Minister: Tony Blair's Quest for a Legacy , 2005 .

[83]  Oliver James,et al.  The UK Core Executive's Use of Public Service Agreements as a Tool of Governance , 2004 .

[84]  S. Bird,et al.  Performance indicators: good, bad, and ugly , 2004 .

[85]  S. Greer Four Way Bet: How devolution has led to four different models for the NHS , 2004 .

[86]  C. Schultze,et al.  The public use of the private interest , 1977 .

[87]  Harvey Goldstein,et al.  League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance , 1996 .

[88]  A M Epstein,et al.  Influence of cardiac-surgery performance reports on referral practices and access to care. A survey of cardiovascular specialists. , 1996, The New England journal of medicine.

[89]  H. Goossens,et al.  Antibiotics for respiratory tract infections in primary care , 2007, BMJ : British Medical Journal.

[90]  Simon Stevens Reform strategies for the English NHS. , 2004, Health affairs.

[91]  Judith H Hibbard,et al.  Does publicizing hospital performance stimulate quality improvement efforts? , 2003, Health affairs.

[92]  P. Shekelle,et al.  Dying to Know , 2000 .

[93]  A. Enthoven,et al.  In pursuit of an improving National Health Service. , 2000, Health affairs.

[94]  Paul Slovic,et al.  Strategies for reporting health plan performance information to consumers: evidence from controlled studies. , 2002, Health services research.

[95]  J. Hibbard,et al.  Will quality report cards help consumers? , 1997, Health affairs.

[96]  Gwyn Bevan,et al.  Have targets improved performance in the English NHS? , 2006, BMJ : British Medical Journal.

[97]  Reporting health care performance: learning from the past, prospects for the future. , 2002, Journal of evaluation in clinical practice.

[98]  R H Brook,et al.  The public release of performance data: what do we expect to gain? A review of the evidence. , 2000, JAMA.

[99]  Bill Shaw,et al.  THE PUBLIC USE OF PRIVATE INTEREST , 1980 .