Evidence of Misclassification of Drug–Event Associations Classified as Gold Standard ‘Negative Controls’ by the Observational Medical Outcomes Partnership (OMOP)

AbstractIntroductionPharmacovigilance includes analysis of large databases of information on drugs and events using algorithms that detect disproportional frequencies of associations. In order to test such algorithms, attempts have been made to provide canonical reference lists of so-called ‘positive controls’ and ‘negative controls’. Reference sets with even modest levels of misclassification may result in under- or overstatement of the performance of algorithms.AimWe sought to determine the extent to which ‘negative control’ drug–event pairs in the Observational Medical Outcomes Partnership (OMOP) database are misclassifiedMethodsWe searched the medical literature for evidence of associations between drugs and events listed by OMOP as negative controls.ResultsThe criteria used in OMOP to classify positive and negative controls are asymmetric; drug–event associations published only as case series or case reports are classified as positive controls if they are cited in Drug-Induced Diseases by Tisdale and Miller, but as negative controls if case series or case reports exist but are not cited in Tisdale and Miller. Of 233 drug–event pairs classified in the 2013 version of OMOP as negative controls, 21 failed to meet pre-specified OMOP adjudication criteria; in another 19 cases we found case reports, case series, or observational evidence that the drug and event are associated. Overall, OMOP misclassified, or may have misclassified, 40 (17 %) of all ‘negative controls.’ConclusionsResults from studies of the performance of signal-detection algorithms based on the OMOP gold standard should be viewed with circumspection, because imperfect gold standards may lead to under/overstatement of absolute and relative signal detection algorithm performance. Improvements to OMOP would include omitting misclassified drug–event pairs, assigning more specific event labels, and using more extensive sources of information.

[1]  M. A. Luongo,et al.  The liver in ferrous sulfate poisoning; a report of three fatal cases in children and an experimental study. , 1954, The New England journal of medicine.

[2]  R. J. Brown,et al.  The mechanism of acute ferrous sulphate poisoning. , 1955, Canadian Medical Association journal.

[3]  Y. Kapanci,et al.  Lupoid hepatitis following administration of penicillin. Case report and immunological studies. , 1967, Helvetica medica acta.

[4]  C. Kunin.  Nephrotoxicity of antibiotics. , 1967, JAMA.

[5]  M. S. Fedotin,et al.  Ergotamine poisoning producing renal arterial spasm. , 1970, The New England journal of medicine.

[6]  K. Ishak,et al.  Hepatic injury associated with penicillin therapy. , 1974, Archives of pathology.

[7]  Ishak Kg,et al.  Hepatic injury associated with penicillin therapy. , 1974 .

[8]  G. Edwards.  THE ALCOHOLIC DOCTOR A Case of Neglect , 1975, The Lancet.

[9]  JAUNDICE AFTER ORAL PENICILLIN , 1976, The Lancet.

[10]  [Ergotamine abuse and retroperitoneal fibrosis (author's transl)]. , 1976, Zentralblatt fur allgemeine Pathologie u. pathologische Anatomie.

[11]  C. Pusey,et al.  St Anthony's fire and pseudochronic renal failure. , 1977, British medical journal.

[12]  R. Charles,et al.  Myocardial infarction and marijuana. , 1979, Clinical toxicology.

[13]  Ergotamine as a possible cause of retroperitoneal fibrosis. , 1982, Clinical pharmacy.

[14]  D. Blank,et al.  Acute renal failure and seizures associated with chlorambucil overdose. , 1983, Journal of toxicology. Clinical toxicology.

[15]  D. MacInnes,et al.  Fatal coronary artery thrombosis associated with cannabis smoking. , 1984, The Journal of the Royal College of General Practitioners.

[16]  D. Silk,et al.  Ergotamine abuse and extra-hepatic portal hypertension. , 1985, Postgraduate medical journal.

[17]  S. Webb,et al.  Myocardial infarction during marijuana smoking in a young female. , 1985, European heart journal.

[18]  [Renal insufficiency and ergotism]. , 1985, Medicina clinica.

[19]  M. Girard.  Conclusiveness of rechallenge in the interpretation of adverse drug reactions. , 1987, British journal of clinical pharmacology.

[20]  J. Devière,et al.  Ischemic pancreatitis and hepatitis secondary to ergotamine poisoning. , 1987, Journal of clinical gastroenterology.

[21]  [Lithium clearance and essential arterial hypertension]. , 1987, Medicina clinica.

[22]  W. Pearl,et al.  Cardiovascular effects of adolescent drug abuse. , 1989, Journal of adolescent health care : official publication of the Society for Adolescent Medicine.

[23]  P N Valenstein,et al.  Evaluating diagnostic tests with imperfect standards. , 1990, American journal of clinical pathology.

[24]  J. Chevrolet,et al.  Very severe self-poisoning lithium carbonate intoxication causing a myocardial infarction. , 1991, Chest.

[25]  J. Lund.  Prolonged renal impairment after chronic ergotamine intoxication. , 1992, Nephrology, dialysis, transplantation : official publication of the European Dialysis and Transplant Association - European Renal Association.

[26]  G. Ruiz-Irastorza,et al.  Hepatotoxicity associated with penicillin V therapy. , 1995, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[27]  C E Phelps,et al.  Estimating Diagnostic Test Accuracy Using a "Fuzzy Gold Standard" , 1995, Medical decision making : an international journal of the Society for Medical Decision Making.

[28]  A. Cohen.  Gastrointestinal blood loss induced by bromfenac sodium, aspirin, and placebo. , 1995, Clinical therapeutics.

[29]  J. Ottervanger,et al.  PHARMACOEPIDEMIOLOGY AND PRESCRIPTION , 1997 .

[30]  M. Tenenbein.  Toxicokinetics and toxicodynamics of iron poisoning. , 1998, Toxicology letters.

[31]  L. Bricker,et al.  Myocardial Infarction in HIV-Infected Men Receiving Protease Inhibitors , 1999, Annals of Internal Medicine.

[32]  R. Faull,et al.  Clozapine-induced acute interstitial nephritis , 1999, The Lancet.

[33]  Doan Rj,et al.  Clozapine treatment and neuroleptic malignant syndrome. , 2000 .

[34]  M. Jibani,et al.  An unexpected and serious complication of treatment with the atypical antipsychotic drug clozapine. , 2000, Clinical nephrology.

[35]  Acute tubulo-interstitial nephritis and renal infarction secondary to ergotamine therapy. , 2000, Nephrology, dialysis, transplantation : official publication of the European Dialysis and Transplant Association - European Renal Association.

[36]  R. Doan,et al.  Clozapine treatment and neuroleptic malignant syndrome. , 2000, Canadian journal of psychiatry. Revue canadienne de psychiatrie.

[37]  J. Muller,et al.  Triggering Myocardial Infarction by Marijuana , 2001, Circulation.

[38]  D M Hawkins,et al.  Some issues in resolution of diagnostic tests using an imperfect gold standard , 2001, Statistics in medicine.

[39]  C. Coleman,et al.  Possible Gatifloxacin-Induced Fulminant Hepatic Failure , 2002, The Annals of pharmacotherapy.

[40]  Ahmet Bilici,et al.  Flutamide-induced acute renal failure in a patient with metastatic prostate cancer , 2002, Medical oncology.

[41]  S. Sidney.  Cardiovascular Consequences of Marijuana Use , 2002, Journal of clinical pharmacology.

[42]  M. Bergeron,et al.  Acute phosphate nephropathy and renal failure. , 2003, The New England journal of medicine.

[43]  J. Guerediaga,et al.  [Acute interstitial nephritis induced by loratadine]. , 2003, Nefrologia : publicacion oficial de la Sociedad Espanola Nefrologia.

[44]  [Cardiac rehabilitation in a HIV-patient treated with protease inhibitors]. , 2003, Monaldi archives for chest disease = Archivio Monaldi per le malattie del torace.

[45]  Acute renal failure and nephrotic syndrome associated with zafirlukast therapy. , 2003, Nephrology, dialysis, transplantation : official publication of the European Dialysis and Transplant Association - European Renal Association.

[46]  K. Clauson.  Drug-Induced Diseases: Prevention, Detection, and Management , 2005 .

[47]  M. Pirmohamed,et al.  The quality of information on monitoring for haematological adverse drug reactions. , 2005, British journal of clinical pharmacology.

[48]  B. Melegh,et al.  Drug-induced myocardial infarction in young patients: report of two cases. , 2005, International journal of cardiology.

[49]  Response to letter by Levine et al. , 2006 .

[50]  Manfred Hauben,et al.  Anecdotes that provide definitive evidence , 2006, BMJ : British Medical Journal.

[51]  Hulya Akgun,et al.  Renal failure and nephrocalcinosis associated with oral sodium phosphate bowel cleansing: clinical patterns and renal biopsy findings. , 2006, Archives of pathology & laboratory medicine.

[52]  C. Jackson,et al.  Evidence-based review of the black-box warning for droperidol. , 2007, American journal of health-system pharmacy : AJHP : official journal of the American Society of Health-System Pharmacists.

[53]  M. Perazella,et al.  Acute oxalate nephropathy associated with orlistat, a gastrointestinal lipase inhibitor. , 2007, American journal of kidney diseases : the official journal of the National Kidney Foundation.

[54]  J. Aronson,et al.  Gold standards in pharmacovigilance: the use of definitive anecdotal reports of adverse drug reactions as pure gold and high-grade ore. , 2007, Drug safety.

[55]  H. Rennke,et al.  Adverse renal and metabolic effects associated with oral sodium phosphate bowel preparation. , 2008, Clinical journal of the American Society of Nephrology : CJASN.

[56]  F. Mach,et al.  Cannabinoid receptors in acute and chronic complications of atherosclerosis , 2008, British journal of pharmacology.

[57]  Stuart A. Jones,et al.  Pharmaceutical salts: a formulation trick or a clinical conundrum? , 2009 .

[58]  F. de Vries,et al.  The pattern of risk of myocardial infarction in patients taking asthma medication: a study with the General Practice Research Database , 2009, Journal of hypertension.

[59]  G. Bella,et al.  Letter to the Editor Acute myocardial infarction and Kounis syndrome , 2009 .

[60]  O. Nielsen,et al.  The safety of osmotically acting cathartics in colonic cleansing , 2010, Nature Reviews Gastroenterology &Hepatology.

[61]  G. Niklas Norén,et al.  A Decade of Data Mining and Still Counting , 2010, Drug safety.

[62]  O. Kirk,et al.  Risk of myocardial infarction in patients with HIV infection exposed to specific individual antiretroviral drugs from the 3 major drug classes: the data collection on adverse events of anti-HIV drugs (D:A:D) study. , 2010, The Journal of infectious diseases.

[63]  J. Goudevenos,et al.  Kounis syndrome: a new twist on an old disease. , 2011, Future cardiology.

[64]  [Acute phosphate nephropathy as a complication to bowel cleansing with oral sodium phosphate]. , 2011, Ugeskrift for laeger.

[65]  R. Dissmann,et al.  [Cannabinoid hyperemesis syndrome inducing acute prerenal failure and electrolyte disturbance]. , 2011, Deutsche medizinische Wochenschrift.

[66]  Kounis Syndrome together with Myocardial Bridging Leading to Acute Myocardial Infarction at Young Age , 2011, Case reports in medicine.

[67]  J. Cornily,et al.  [Cannabis and myocardial infarction without angiographic stenosis in young patient: guilty or not guilty? A case report]. , 2011, Annales de cardiologie et d'angeiologie.

[68]  H. Conjeevaram,et al.  Clinical and histopathologic features of fluoroquinolone-induced liver injury. , 2011, Clinical gastroenterology and hepatology : the official clinical practice journal of the American Gastroenterological Association.

[69]  P. Castellant,et al.  [Cannabis and myocardial infarction without angiographic stenosis in young patient: guilty or not guilty? A case report]. , 2011, Annales de cardiologie et d'angeiologie.

[70]  Amit X Garg,et al.  Orlistat and acute kidney injury: an analysis of 953 patients. , 2011, Archives of internal medicine.

[71]  C. Tremblay,et al.  Association Between HIV Infection, Antiretroviral Therapy, and Risk of Acute Myocardial Infarction: A Cohort and Nested Case–Control Study Using Québec's Public Health Insurance Database , 2011, Journal of acquired immune deficiency syndromes.

[72]  W. Tormey.  Cannabis misinterpretation and misadventure in a coroner's court , 2012, Medicine, science, and the law.

[73]  W. Tormey.  Cannabis, possible cardiac deaths and the coroner in Ireland , 2012, Irish Journal of Medical Science.

[74]  D. Madigan,et al.  Empirical assessment of methods for risk identification in healthcare data: results from the experiments of the Observational Medical Outcomes Partnership , 2012, Statistics in medicine.

[75]  N. Shah,et al.  Performance of Pharmacovigilance Signal‐Detection Algorithms for the FDA Adverse Event Reporting System , 2013, Clinical pharmacology and therapeutics.

[76]  M. Schuemie,et al.  Defining a Reference Set to Support Methodological Research in Drug Safety , 2013, Drug Safety.

[77]  William DuMouchel,et al.  Empirical bayes model to combine signals of adverse drug reactions , 2013, KDD.

[78]  Charles E. Leonard,et al.  Comment on: “Desideratum for Evidence-Based Epidemiology” , 2014, Drug Safety.

[79]  N. Shah,et al.  Comment on: “Zoo or Savannah? Choice of Training Ground for Evidence-Based Pharmacovigilance” , 2014, Drug Safety.

[80]  Authors’ Reply to Hennessy and Leonard’s Comment on “Desideratum for Evidence-Based Epidemiology” , 2014, Drug Safety.

[82]  Ryen W. White,et al.  Toward Enhanced Pharmacovigilance Using Patient-Generated Data on the Internet , 2014, Clinical pharmacology and therapeutics.

[83]  Olivier Bodenreider,et al.  A time-indexed reference standard of adverse drug reactions , 2014, Scientific Data.

[84]  Jefferson Rosa Cardoso,et al.  What is gold standard and what is ground truth? , 2014, Dental press journal of orthodontics.

[85]  T Bürkle,et al.  Inconsistencies and Misleading Information in Officially Approved Prescribing Information From Three Major Drug Markets , 2014, Clinical pharmacology and therapeutics.

[86]  C. Friedman,et al.  A Method to Combine Signals from Spontaneous Reporting Systems and Observational Healthcare Data to Detect Adverse Drug Reactions , 2015, Drug Safety.

[87]  Takaya Saito,et al.  The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets , 2015, PloS one.

[88]  Rapidly progressive renal failure associated with successful pharmacotherapy for obesity , .