How valid are medical records and patient questionnaires for physician profiling and health services research? A comparison with direct observation of patients visits.

OBJECTIVES This study was designed to determine the optimal nonobservational method of measuring the delivery of outpatient medical services. METHODS As part of a multimethod study of the content of primary care practice, research nurses directly observed consecutive patient visits to 138 practicing family physicians. Data on services delivered were collected using a direct observation checklist, medical record review, and patient exit questionnaires. For each medical service, the sensitivity, specificity, and Kappa statistic were calculated for medical record review and patient exit questionnaires compared with direct observation. Interrater reliability among eight research nurses was calculated using the Kappa statistic for a separate sample of videotaped visits and medical records. RESULTS Visits by 4,454 patients were observed. Exit questionnaires were returned by 74% of patients. Research nurse interrater reliabilities were generally high. The specificity of both the medical record and the patient exit questionnaire was high for most services. The sensitivity of the medical record was low for measuring health habit counseling and moderate for physical examination, laboratory testing, and immunization. The patient exit questionnaire showed moderate to high sensitivity for health habit counseling and immunization and variable sensitivity for physical examination and laboratory services. CONCLUSIONS The validity of the medical record and patient questionnaire for measuring delivery of different health services varied with the service. This report can be used to choose the optimal nonobservational method of measuring the delivery of specific ambulatory medical services for research and physician profiling and to interpret existing health services research studies using these common measures.

[1]  Dauphinee Wd.  Assessing clinical performance. Where do we stand and what might we expect , 1995 .

[2]  A. J. Conger.  Integration and generalization of kappas for multiple raters. , 1980 .

[3]  K C Stange,et al.  One size doesn't fit all. Multimethod research yields new insights into interventions to increase prevention in family practice. , 1996, The Journal of family practice.

[4]  A. Lawthers,et al.  Focus on quality: Profiling physicians' practice patterns , 1994, The Journal of ambulatory care management.

[5]  A. Feinstein,et al.  High agreement but low kappa: I. The problems of two paradoxes. , 1990, Journal of clinical epidemiology.

[6]  W. McAuliffe,et al.  Studies of Process–Outcome Correlations in Medical Care Evaluations: A Critique , 1978, Medical care.

[7]  K. Stange.  Practice-based research networks. Their current level of validity, generalizability, and potential for wider application. , 1993, Archives of family medicine.

[8]  Schappert Sm.  National Ambulatory Medical Care Survey: 1992 summary. , 1994, Advance data.

[9]  D. A. Goldman,et al.  Validity of Pap smear and mammogram self-reports in a low-income Hispanic population. , 1995, American journal of preventive medicine.

[10]  S. Leatherman,et al.  Medical Practice Profiling: Concepts and Caveats , 1995, Medical care research and review : MCRR.

[11]  M. Adams,et al.  Patients as Reliable Reporters of Medical Care Process: Recall of Ambulatory Encounter Events , 1992, Medical care.

[12]  T. M. Wynn,et al.  Accuracy of women's self-report of their last Pap smear. , 1989, American journal of public health.

[13]  S. Fagerhaugh,et al.  Participant Observation , 1979 .

[14]  A. Lawthers,et al.  Developing a Quality Improvement Database Using Health Insurance Data: A Guided Tour with Application to Medicare's National Claims History File , 1995, American journal of medical quality : the official journal of the American College of Medical Quality.

[15]  W. Dauphinee.  Assessing clinical performance. Where do we stand and what might we expect? , 1995, JAMA.

[16]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[17]  D. Lane,et al.  The use of mammography vans by low-income women: the accuracy of self-reports. , 1994, American journal of public health.

[18]  A. Lawthers,et al.  Variation in office-based quality. A claims-based profile of care provided to Medicare patients with diabetes. , 1995, JAMA.

[19]  Taking Health Status Into Account When Setting Capitation Rates: A Comparison of Risk-Adjustment Methods , 1996 .

[20]  K. Bertakis,et al.  Development and validation of the Davis Observation Code. , 1991, Family medicine.

[21]  Developing and evaluating performance measures for ambulatory care quality: a preliminary report of the DEMPAQ project. , 1993, The Joint Commission journal on quality improvement.

[22]  J. Iglehart,et al.  The National Committee for Quality Assurance. , 1996, The New England journal of medicine.

[23]  K. Stange.  Primary care research: barriers and opportunities. , 1996, The Journal of family practice.

[24]  A. Coulter,et al.  Lifestyle advice in general practice: rates recalled by patients. , 1992, BMJ.

[25]  Michael J. Strube.  A general program for the calculation of the kappa coefficient , 1989 .

[26]  S. Harlow,et al.  Agreement between questionnaire data and medical records. The evidence for accuracy of recall. , 1989, American journal of epidemiology.

[27]  S. Greenfield,et al.  Agreement Among Physician Assessment Methods Searching for the Truth Among Fallible Methods , 1988, Medical care.

[28]  C. S. Johnson,et al.  Accuracy of Pap smear and mammogram self-reports in a southwestern Native American tribe. , 1995, American journal of preventive medicine.

[29]  John S. Uebersax,et al.  A Generalized Kappa Coefficient , 1982 .

[30]  K C Stange,et al.  Multimethod research: approaches for integrating qualitative and quantitative methods. , 1994, Journal of general internal medicine.

[31]  D. Mark,et al.  Bias in the coding of hospital discharge data and its implications for quality assessment. , 1994, Medical care.

[32]  N. Powe,et al.  Applying insurance claims data to assess quality of care: a compilation of potential indicators. , 1990, QRB. Quality review bulletin.

[33]  K C Stange,et al.  Illuminating the 'black box'. A description of 4454 patient visits to 138 family physicians. , 1998, The Journal of family practice.

[34]  B Starfield,et al.  Concordance Between Medical Records and Observations Regarding Information on Coordination of Care , 1979, Medical care.

[35]  E. Fisher,et al.  The accuracy of Medicare's hospital claims data: progress has been made, but problems remain. , 1992, American journal of public health.

[36]  W. Willett,et al.  Misinterpretation and misuse of the kappa statistic. , 1987, American journal of epidemiology.

[37]  B. Starfield,et al.  Primary care as part of US health services reform. , 1993, JAMA.

[38]  E F Cook,et al.  Screening procedures in the asymptomatic adult. Comparison of physicians' recommendations, patients' desires, published guidelines, and actual practice. , 1985, JAMA.

[39]  A. E. Zuckerman,et al.  MISCELLANEOUS: Validating the content of pediatric outpatient medical records by means of tape‐recording doctor‐patient encounters , 1976, Pediatrics.

[40]  W. Phillips,et al.  Cancer screening by primary care physicians: a comparison of rates obtained from physician self-report, patient survey, and chart audit. , 1995, American journal of public health.

[41]  D. Quade,et al.  Measuring the use of mammography: two methods compared. , 1992, American journal of public health.

[42]  L Appleton,et al.  A reason for visit classification for ambulatory care. , 1979, Vital and health statistics. Series 2, Data evaluation and methods research.

[43]  A. Feinstein,et al.  High agreement but low kappa: II. Resolving the paradoxes. , 1990, Journal of clinical epidemiology.

[44]  Medical competence and performance assessment. A new era. , 1991, JAMA.

[45]  J. Jollis,et al.  A comparison of administrative versus clinical data: coronary artery bypass surgery as an example. Ischemic Heart Disease Patient Outcomes Research Team. , 1994, Journal of clinical epidemiology.

[46]  S. Schappert.  National Ambulatory Medical Care Survey: 1994 summary. , 1996, Advance data.

[47]  M. Speers,et al.  Agreement between patient self-reports and medical records for Pap smear histories. , 1992, American journal of preventive medicine.

[48]  J P Kassirer,et al.  The use and abuse of practice profiles. , 1994, The New England journal of medicine.

[49]  C. Sherbourne,et al.  Preliminary Tests of a 6-Item General Health Survey , 1992 .

[50]  J. Zapka,et al.  Mammography use among sociodemographically diverse women: the accuracy of self-report. , 1996, American journal of public health.

[51]  M. Rohrbaugh,et al.  What did the doctor do? When physicians and patients disagree. , 1994, Archives of family medicine.

[52]  J. Kralewski,et al.  Constructing episodes of care from encounter and claims data: some methodological issues. , 1995, Inquiry : a journal of medical care organization, provision and financing.

[53]  K. N. Williams,et al.  Quality assurance today and tomorrow: forecast for the future. , 1976, Annals of internal medicine.

[54]  The Ambulatory Care Medical Audit Demonstration Project: Research Design , 1996 .

[55]  K C Stange,et al.  Integrating qualitative and quantitative research methods. , 1989, Family medicine.

[56]  S. Schappert.  National Ambulatory Medical Care Survey: 1991 summary. , 1993, Advance data.