Identifying Adverse Events Caused by Medical Care: Degree of Physician Agreement in a Retrospective Chart Review

Retrospective case review has long been a main-stay of peer review. It supports scientific studies and medical audits as well as assessments of the appropriateness, effectiveness, and quality of health care provided by physicians, hospitals, or regions. As part of quality assurance, hospitals and clinics regularly use formal and informal case review. Insurers and managed care organizations rely on case review when making decisions about coverage. All forms of case review depend heavily on expert opinion. Case review also underlies current and proposed systems of compensating patients for injuries caused by medical care. Under the current litigation system, the patient must prove, with the support of expert medical opinion, that medical care contributed to the injury (causation) and fell below the standards of practice in the community (negligence). Under proposed no-fault alternatives to litigation, entitlement to compensation and liability for payment might also depend on an expert's opinion as to whether the patient's outcome was caused by medical care rather than by a preexisting disease or condition [1-4]. Critics have identified several problems with case review. First, experts cannot form a consensus about which outcomes are adverse. Second, medical technology changes rapidly and creates uncertainty about the appropriateness and effectiveness of practices. Third, administrative or transaction costs in making individualized determinations of causation might be high [5-8]. The American College of Physicians [9] and others [10] have called for further demonstration projects. Building on previous research on the reliability of clinical judgments, we used a large sample of physician reviews of medical records to estimate the degree of agreement on the cause of adverse patient outcomes. We also discuss the implications of the results for quality assurance, performance assessment, and proposals for no-fault patient compensation. Methods Cases were obtained from the Medical Practice Study, a project designed to estimate the rate of adverse events occurring among inpatients in a random sample of 31 429 medical records from 51 health care facilities in New York State. We defined an adverse event as an injury that 1) was caused at least in part by medical management and 2) required or prolonged hospitalization or led to disability after discharge. The injury could result from a provider's action or inaction in either inpatient or outpatient settings or from a drug or medical device. The medical management did not have to be substandard or inappropriate; the injury could follow an unexpected complication. Adverse outcomes caused solely by underlying disease or by the intended consequences of treatment were not considered to be adverse events. For example, an injury to the recurrent laryngeal nerve during partial thyroidectomy (an unplanned and unintended but recognized complication) would be considered an adverse event, but the intentional destruction of the same nerve in a radical thyroid resection for cancer would not. A broken experimental balloon that led to an embolus and stroke during cardiac catheterization would, as a complication of treatment, also be an adverse event, especially if the patient's risk was unknown. This result would apply even in a study approved by a Human Subjects Committee. Other aspects of the Medical Practice Study and the general methods have been widely reported [11-15]. The following methods are relevant to our report. Record Review Records were reviewed in two stages. In stage 1 (which is not the subject of this report), nurses and medical records administrators used a single review per case to screen the entire sample of records for the presence of 1 or more of 18 explicit criteria (Figure 2). These criteria were based primarily on previous research [16] and were revised by the physician investigators of the Medical Practice Study. Although explicit, the criteria were broad and open to interpretation. The nurses and records administrators received an extensive manual, which contained detailed examples of the criteria, and 2 hours of focused classroom training from team leaders chosen for this project. To increase the efficiency and accuracy of screening, the nurses and records administrators used preprinted forms generated by the project management team. Nurses were instructed to refer any questionable cases for stage 2 review. Questions of a more general nature were referred to supervisors and then to the project office for consistent responses. The estimated negative predictive value of the screening was 99.5% [17]. Figure 2. Screening Criteria Implemented at Stage 1 Review by Nurses and Medical Records Administrators. Judgments on adverse events by pairs of physician-reviewers and rate of agreement on occurrence of adverse events compared with extreme disagreement. If a = cases of extreme disagreement (one reviewer scored the outcome as 0 [no possible adverse event] and the other scored the case as 4, 5, or 6) and b = cases for which both reviewers found adverse events (both scored the case as 4, 5, or 6), then the reported rate of agreement = a/(a + b). Bars represent exact binomial 95% CIs. Numbers in parentheses are the population-weighted estimates of the number of cases in New York State in 1984 that are represented by the sampled cases reported in this figure. In stage 2, each record that had or may have had at least one criterion present was further analyzed by two physicians who worked independently. Physicians were recruited primarily from New York State through a network of personal contacts of the study investigators. The physicians could not review records at the hospitals in which they practiced. Most were board certified in surgery (23%) or internal medicine (68%); the remaining were certified in obstetrics and gynecology, family practice, pediatrics, urology, or emergency medicine. Eighty-five percent were male. Most physicians were in the early stage of their careers: Fifty-five percent had received board certification within the 10 years before the study began. All physicians had telephone access to a panel of experts. A separate manual and a structured abstraction form guided the stage 2 review. As described previously [17], both were revised repeatedly after extensive pilot testing. This 65-page manual included explicit instructions on several types of adverse events. According to the manual, for example, all surgical wound infections were almost invariably adverse events, as were all falls and all drug reactions that prolonged hospitalization or caused disability. A 14-page abstraction form first asked the physician reviewer to assess whether an adverse event might have occurred. If the physician found no possible adverse event, the review was stopped and the case received a score of 0. If an adverse event might have occurred, the reviewer considered a list of factors on the cause of the injury and rated his or her confidence about the occurrence of an adverse event on an interval scale of 1 to 6 (Figure 1). For a confidence score of 2 (slight to modest evidence of an adverse event) or greater, the reviewer indicated the type of event (fall, drug reaction, wound infection, error of omission, or failure to diagnose), the number of additional days of hospitalization (if applicable), and the degree of disability over and above the underlying disease. Finally, the reviewers considered whether the error amounted to negligence. Within this structure, however, the physician could be discreet in judging the cause of the injury, hospitalization, or disability (a structured implicit review). All physician reviewers identified themselves by number, with the understanding that their confidential opinions would not be used for quality assurance, peer review, or litigation. Copies of the abstraction booklet are available from the authors. Figure 1. Our report focuses on the two independent expert opinions obtained during stage 2 review as to whether an adverse outcome identified during stage 1 had been caused at least in part by medical management. Results of each assessment of causation were linked to the patient's computerized discharge data summary to identify the patient's age, diagnosis, and discharge status. Statistical Analysis Agreement between Reviewers We calculated a rate of agreement between the two physician reviewers in each pair on adverse events using a statistic described by Grant [18] for assessing agreement on abnormal tracings from electronic fetal monitoring. In our application, the numerator of this statistic was the number of cases in which both reviewers assessed their confidence in an adverse event as more likely than not or greater. This assessment corresponded to a score of 4, 5, or 6. The denominator was the sum of the numerator and the number of cases of extreme disagreement, for which one reviewer scored the case as 4, 5, or 6 and the other physician found no possible adverse event (a score of 0). The statistic therefore compared the number of cases with agreed-upon adverse events with the number of clear disagreements. This statistic does not include cases for which both physicians agreed that no adverse event had occurred. It recognizes that agreement about whether a patient's condition is normal (no adverse event) is usually greater than agreement about whether a patient has disease or an abnormal condition [19-22]. The statistic is also not affected by the number of clearly normal cases in the samples of cases for review. In our study, the number of cases clearly without adverse events at stage 2 was influenced by the coarseness of the previous screening process. The stage 1 reviewers were cautioned to avoid false-negative determinations if they were in doubt, so that adverse events would not be overlooked. This statistic also facilitated comparisons of rates of agreement across subsets of such adverse events as drug reactions, which are