Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project

Introduction and ObjectiveSocial media has been proposed as a possibly useful data source for pharmacovigilance signal detection. This study primarily aimed to evaluate the performance of established statistical signal detection algorithms in Twitter/Facebook for a broad range of drugs and adverse events.MethodsPerformance was assessed using a reference set by Harpaz et al., consisting of 62 US Food and Drug Administration labelling changes, and an internal WEB-RADR reference set consisting of 200 validated safety signals. In total, 75 drugs were studied. Twitter/Facebook posts were retrieved for the period March 2012 to March 2015, and drugs/events were extracted from the posts. We retrieved 4.3 million and 2.0 million posts for the WEB-RADR and Harpaz drugs, respectively. Individual case reports were extracted from VigiBase for the same period. Disproportionality algorithms based on the Information Component or the Proportional Reporting Ratio and crude post/report counting were applied in Twitter/Facebook and VigiBase. Receiver operating characteristic curves were generated, and the relative timing of alerting was analysed.ResultsAcross all algorithms, the area under the receiver operating characteristic curve for Twitter/Facebook varied between 0.47 and 0.53 for the WEB-RADR reference set and between 0.48 and 0.53 for the Harpaz reference set. For VigiBase, the ranges were 0.64–0.69 and 0.55–0.67, respectively. In Twitter/Facebook, at best, 31 (16%) and four (6%) positive controls were detected prior to their index dates in the WEB-RADR and Harpaz references, respectively. In VigiBase, the corresponding numbers were 66 (33%) and 17 (27%).ConclusionsOur results clearly suggest that broad-ranging statistical signal detection in Twitter and Facebook, using currently available methods for adverse event recognition, performs poorly and cannot be recommended at the expense of other pharmacovigilance activities.

[1]  Taha A. Kass-Hout,et al.  Digital Drug Safety Surveillance: Monitoring Pharmaceutical Products in Twitter , 2014, Drug Safety.

[2]  S. Evans,et al.  Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports , 2001, Pharmacoepidemiology and drug safety.

[3]  Hsinchun Chen,et al.  Social Media Analytics and Intelligence , 2010, IEEE Intell. Syst..

[4]  M. Lindquist,et al.  Zoo or Savannah? Choice of Training Ground for Evidence-Based Pharmacovigilance , 2014, Drug Safety.

[5]  A. Bate,et al.  Quantitative signal detection using spontaneous ADR reporting , 2009, Pharmacoepidemiology and drug safety.

[6]  Jeffery L. Painter,et al.  Social Media Listening for Routine Post-Marketing Safety Surveillance , 2016, Drug Safety.

[7]  Stephen Lin,et al.  Web-Based Signal Detection Using Medical Forums Data in France: Comparative Analysis , 2018, Journal of medical Internet research.

[8]  J. Wooten,et al.  Reporting adverse drug reactions. , 2009, Southern medical journal.

[9]  J. Brownstein,et al.  Evaluation of Facebook and Twitter Monitoring to Detect Safety Signals for Medical Products: An Analysis of Recent FDA Safety Alerts , 2017, Drug Safety.

[10]  Fan Yu,et al.  Towards large-scale twitter mining for drug-related adverse events , 2012, SHB '12.

[11]  Rachel E. Ginn,et al.  Social Media Mining for Toxicovigilance: Automatic Monitoring of Prescription Medication Abuse from Twitter , 2016, Drug Safety.

[12]  Johan Hopstadius,et al.  Shrinkage observed-to-expected ratios for robust and transparent large-scale pattern discovery , 2011, Statistical methods in medical research.

[13]  Melissa M. Truffa,et al.  Using Social Media Data in Routine Pharmacovigilance: A Pilot Study to Identify Safety Signals and Patient Perspectives , 2017, Pharmaceutical Medicine.

[14]  A. Bate,et al.  A Bayesian neural network method for adverse drug reaction signal generation , 1998, European Journal of Clinical Pharmacology.

[15]  Kristina Juhlin,et al.  Comparison of Statistical Signal Detection Methods Within and Across Spontaneous Reporting Databases , 2015, Drug Safety.

[16]  Marie Lindquist,et al.  Social Media and Networks in Pharmacovigilance , 2011, Drug safety.

[17]  Jian Yang,et al.  Towards Internet-Age Pharmacovigilance: Extracting Adverse Drug Reactions from User Posts in Health-Related Social Networks , 2010, BioNLP@ACL.

[18]  Jeffery L Painter,et al.  Using Social Listening Data to Monitor Misuse and Nonmedical Use of Bupropion: A Content Analysis , 2017, JMIR public health and surveillance.

[19]  Christopher C. Yang,et al.  Social media mining for drug safety signal detection , 2012, SHB '12.

[20]  Olivier Bodenreider,et al.  A time-indexed reference standard of adverse drug reactions , 2014, Scientific Data.

[21]  M. Lindquist.  VigiBase, the WHO Global ICSR Database System: Basic Facts , 2008 .

[22]  B. I. Evans.  XIII , 1902, Menschenwürde und Strafvollzug.

[23]  Anne Cocos,et al.  Deep learning for pharmacovigilance: recurrent neural network architectures for labeling adverse drug reactions in Twitter posts , 2017, J. Am. Medical Informatics Assoc..