Which data source in clinical performance assessment? A pilot study comparing self-recording with patient records and observation.

OBJECTIVE A pilot study aimed to determine the extent to which each of three data sources could provide complete and reliable data for valid assessment of clinical performance. DESIGN Clinical decisions taken in 168 consultations by seven family physicians were reviewed against guidelines for 15 clinical conditions. In total, 206 criteria were reviewed using three sources: medical records, observation in surgery, and structured self-recording by the physicians. SETTING AND PARTICIPANTS Seven family practices in the Netherlands. MAIN MEASURE Scores (%) of data recorded/total were obtained for each method. Kappa scores for the agreement between the three data sources were also obtained. RESULTS Medical record examination provided 40%, observation 72%, and physician self-recording 95% of the data required for the review against guidelines. Nine per cent of the clinical decisions could be reviewed when using medical records, 46% when using observation data, and 69% when using data from prospective self-recording. In particular, decisions in the area of patient education and diagnostic examinations could not be reviewed validly using medical records only. Kappa agreements between the data available from the three sources as well as between the review results appeared to be 0.79. CONCLUSIONS Medical records alone only supply sufficient information for the review of a very limited set of clinical decisions. Physician self-recording has significantly more potential for valid review of a broad range of clinical decisions. Furthermore, self-recording seems a reliable data collection method that deserves further research.

[1]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[2]  C J McDonald.  Quality measures and electronic medical systems. , 1999, JAMA.

[3]  H. Davies,et al.  Trust in performance indicators? , 1998, Quality in health care : QHC.

[4]  S. Blumenfeld,et al.  Comparative validity of three methods for assessment of the quality of primary health care. , 1999, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[5]  M. Field,et al.  Guidelines for Clinical Practice: From Development to Use , 1992 .

[6]  M Pringle,et al.  Assessment of the completeness and accuracy of computer medical records in four practices committed to recording data on computer. , 1995, The British journal of general practice : the journal of the Royal College of General Practitioners.

[7]  S. Lydeard,et al.  The questionnaire as a research tool. , 1991, Family practice.

[8]  S Westin,et al.  Methods for quality assessment in general practice. , 1996, Family practice.

[9]  J B Lowe,et al.  Improving general practitioner clinical records with a quality assurance minimal intervention. , 1998, The British journal of general practice : the journal of the Royal College of General Practitioners.

[10]  R. Bobrow.  The unintended consequences of measuring quality on the quality of medical care. , 2000, The New England journal of medicine.

[11]  J Lomas,et al.  Evidence of self-report bias in assessing adherence to guidelines. , 1999, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[12]  R. Wachter,et al.  The unintended consequences of measuring quality on the quality of medical care. , 2000, The New England journal of medicine.

[13]  L. Iezzoni.  Assessing Quality Using Administrative Data , 1997, Annals of Internal Medicine.

[14]  T. Sheldon,et al.  Measuring performance in the NHS , 1998, BMJ.

[15]  G Mosser,et al.  The three faces of performance measurement: improvement, accountability, and research. , 1997, The Joint Commission journal on quality improvement.

[16]  W. Manning,et al.  The unreliability of individual physician "report cards" for assessing the costs and quality of care of a chronic disease. , 1999, JAMA.

[17]  D M Eddy,et al.  Performance measurement: problems and solutions. , 1998, Health affairs.

[18]  B. Gerbert,et al.  Measuring Physician Behavior , 1986, Medical care.

[19]  R. Grol,et al.  Development and implementation of guidelines for family practice: lessons from The Netherlands. , 1995, The Journal of family practice.

[20]  R H Brook,et al.  Quality of health care. Part 2: measuring quality of care. , 1997, The New England journal of medicine.

[21]  M. Pringle,et al.  Do general practice computer systems assist in medical audit? , 1994, Family practice.

[22]  Developing and evaluating performance measures for ambulatory care quality: a preliminary report of the DEMPAQ project. , 1993, The Joint Commission journal on quality improvement.

[23]  A. Donabedian.  The methods and findings of quality assessment and monitoring : an illustrated analysis , 1985 .

[24]  E. McGlynn,et al.  Part 2: Measuring Quality of Care , 1996 .

[25]  E. McGlynn,et al.  How good is the quality of health care in the United States? , 1998, The Milbank quarterly.

[26]  A. Wilson,et al.  Comparison of patient questionnaire, medical record, and audio tape in assessment of health promotion in general practice consultations , 1994, BMJ.

[27]  J. G. Adair,et al.  The Hawthorne effect: A reconsideration of the methodological artifact. , 1984 .

[28]  Martin Roland,et al.  Measuring quality of care with routine data: avoiding confusion between performance indicators and health outcomes , 1999, BMJ.

[29]  M. Chassin,et al.  The urgent need to improve health care quality. Institute of Medicine National Roundtable on Health Care Quality. , 1998, JAMA.

[30]  Peter Orton,et al.  The New NHS: Modern and Dependable? , 1998 .

[31]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[32]  M. Field,et al.  Guidelines for Clinical Practice: From Development to Use , 1992 .

[33]  John Gabbay,et al.  Reactions to the use of evidence-based performance indicators in primary care: a qualitative study , 2000, Quality in health care : QHC.

[34]  P L Heywood,et al.  Real world data--retrieval and validation of consultation data from four general practices. , 1996, Family practice.

[35]  M Pringle,et al.  Does awareness of being video recorded affect doctors' consultation behaviour? , 1990, The British journal of general practice : the journal of the Royal College of General Practitioners.

[36]  S. Blumenfeld,et al.  Methodology matters. Comparative validity of three methods for assessment of the quality of primary health care , 1999 .

[37]  John Gabbay,et al.  Clinical governance in primary care groups: the feasibility of deriving evidence-based performance indicators , 2000, Quality in health care : QHC.

[38]  E Martin,et al.  To what extent do clinical notes by general practitioners reflect actual medical performance? A study using simulated patients. , 1994, The British journal of general practice : the journal of the Royal College of General Practitioners.

[39]  R. Palmer,et al.  Physician attitudes, self-estimated performance and actual compliance with locally peer-defined quality evaluation criteria. , 1999, International journal for quality in health care : journal of the International Society for Quality in Health Care.