Improving the Reliability of Physician “Report Cards”

Background:Performance measures are widely used to profile primary care physicians (PCPs) but their reliability is often limited by small sample sizes. We evaluated the reliability of individual PCP profiles and whether they can be improved by combining measures into composites or by profiling practice groups. Methods:We performed a cross-sectional analysis of electronic health record data for patients with diabetes (DM), congestive heart failure (CHF), ischemic vascular disease (IVD), or eligible for preventive care services seen by a PCP within a large, integrated health care system between April 2009 and May 2010. We evaluated performance on 14 measures of DM care, 9 of CHF, 7 of IVD, and 4 of preventive care. Results:There were 51,771 patients observed by 163 physicians in 17 clinics. Few PCPs (0%–60%) could be profiled with 80% reliability using single process or intermediate-outcome measures. Combining measures into single-disease composites improved reliability for DM and preventive care with 74.5% and 76.7% of PCPs having sufficient panel sizes, but composites remained unreliable for CHF and IVD. A total of 85.3% of PCPs could be reliably profiled using a single overall composite. Aggregating PCPs into practice groups (3 to 21 PCPs per group) did not improve reliability in most cases because of little between-group practice variation. Conclusions:Single measures rarely differentiate between individual PCPs or groups of PCPs reliably. Combining measures into single-disease or multidisease composites can improve reliability for some common conditions, but not all. Assessing PCP practice groups within a single health care system, rather than individual PCPs, did not substantially improve reliability.

[1]  Diane P. Martin,et al.  The Reliability of Medical Group Performance Measurement in a Single Insurer’s Pay for Performance Program , 2012, Medical care.

[2]  Stephen D. Persell,et al.  ACCF/AHA/AMA-PCPI 2011 performance measures for adults with coronary artery disease and hypertension: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Performance Measures and the American Medical Association-Physician Consortium for Performance Impr , 2011, Circulation.

[3]  Kaveh G Shojania,et al.  Tell me about the context, and more , 2011, Quality and Safety in Health Care.

[4]  Peter J Pronovost,et al.  A framework for classifying patient safety practices: results from an expert consensus process , 2011, Quality and Safety in Health Care.

[5]  D. Safran,et al.  Reliability of Medical Group and Physician Performance Measurement in the Primary Care Setting , 2011, Medical care.

[6]  E. McGlynn,et al.  Cost profiles: should the focus be on individual physicians or physician groups? , 2010, Health affairs.

[7]  J. Alexander,et al.  A Report Card on Provider Report Cards: Current Status of the Health Care Transparency Movement , 2010, Journal of General Internal Medicine.

[8]  John Hsu,et al.  Meaningful Variation in Performance: A Systematic Literature Review , 2010, Medical care.

[9]  William B Weeks,et al.  Relationship of primary care physicians' patient caseload with measurement of quality and cost performance. , 2009, JAMA.

[10]  S. Kaplan,et al.  Improving the Reliability of Physician Performance Assessment: Identifying the “Physician Effect” on Quality and Creating Composite Measures , 2009, Medical care.

[11]  John L Adams,et al.  Benchmarking physician performance: reliability of individual and composite measures. , 2008, The American journal of managed care.

[12]  R. Hayward.  Access to clinically-detailed patient information: a fundamental element for improving the efficiency and quality of healthcare. , 2008, Medical care.

[13]  J. Wheeler,et al.  Using Knowledge of Multiple Levels of Variation in Care to Target Performance Incentives to Providers , 2008, Medical care.

[14]  Paul E. Johnson,et al.  Variation in Quality of Diabetes Care at the Levels of Patient, Physician, and Clinic , 2007, Preventing chronic disease.

[15]  R. Hayward,et al.  Overestimating outcome rates: statistical estimation when reliability is suboptimal. , 2007, Health services research.

[16]  J. Selby,et al.  Predicted Quality-Adjusted Life Years as a Composite Measure of the Clinical Value of Diabetes Risk Factor Control , 2007, Medical care.

[17]  L. Sandy,et al.  Pay for performance in commercial HMOs. , 2007, The New England journal of medicine.

[18]  T. Mills,et al.  Measuring Health: A Guide to Rating Scales and Questionnaires , 2006 .

[19]  Robert Sherrick,et al.  Who is at greatest risk for receiving poor-quality health care? , 2006, The New England journal of medicine.

[20]  Harlan M Krumholz,et al.  ACC/AHA Clinical Performance Measures for Adults with Chronic Heart Failure: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures (Writing Committee to Develop Heart Failure Clinical Performance Measures): endorsed by the Heart Failure Society , 2005, Circulation.

[21]  A. Wu,et al.  Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance. , 2005, JAMA.

[22]  F. Dominici,et al.  Variations of physician group profiling indicators for asthma care. , 2005, The American journal of managed care.

[23]  P. Shekelle,et al.  Comparison of Quality of Care for Patients in the Veterans Health Administration and Patients in a National Sample , 2004, Annals of Internal Medicine.

[24]  E. Guadagnoli,et al.  The Influence of Physicians’ Practice Management Strategies and Financial Arrangements on Quality of Care Among Patients With Diabetes , 2004, Medical care.

[25]  N. Schenker,et al.  Overlapping confidence intervals or standard error intervals: What do they mean in terms of statistical significance? , 2003, Journal of insect science.

[26]  Rodney A Hayward,et al.  Building a Better Quality Measure: Are Some Patients With ‘Poor Quality’ Actually Getting Good Care? , 2003, Medical care.

[27]  Rodney A Hayward,et al.  Comparing clinical automated, medical record, and hybrid data sources for diabetes quality measures. , 2002, The Joint Commission journal on quality improvement.

[28]  Rodney A Hayward,et al.  Whom should we profile? Examining diabetes care practice variation among primary care providers, provider groups, and health care facilities. , 2002, Health services research.

[29]  J J Mohr,et al.  Improving safety on the front lines: the role of clinical microsystems , 2002, Quality & safety in health care.

[30]  W. Katon,et al.  Are there detectable differences in quality of care or outcome of depression across primary care providers? , 2000, Medical care.

[31]  W. Manning,et al.  The unreliability of individual physician "report cards" for assessing the costs and quality of care of a chronic disease. , 1999, JAMA.

[32]  T. Louis,et al.  Consistency in performance among primary care practitioners. , 1996, Medical care.

[33]  E. Orav,et al.  Issues of variability and bias affecting multisite measurement of quality of care. , 1996, Medical care.

[34]  Hong Chang,et al.  Measuring patients’ experiences with individual primary care physicians , 2007, Journal of General Internal Medicine.

[35]  Maureen Bisognano,et al.  THE COMMONWEALTH FUND Commission on a High Performance Health System , 2005 .

[36]  A. Zaslavsky,et al.  Variation in Patient-Reported Quality Among Health Care Organizations , 2002, Health care financing review.

[37]  J J Mohr,et al.  Building a Quality Future , 1998, Frontiers of health services management.

[38]  H. Goldstein,et al.  The Graphical Presentation of a Collection of Means , 1995 .