Towards Real-Time Measurement of Public Epidemic Awareness : Monitoring Influenza Awareness through Twitter

This study analyzes temporal trends in Twitter data pertaining to both influenza awareness and influenza infection during the 2012–13 influenza season in the US. We make use of classifiers to distinguish tweets that express a personal infection (“sick with the flu”) versus a more general awareness (“worried about the flu”). While previous research has focused on estimating prevalence of influenza infection, little is known about trends in public awareness of the disease. Our analysis shows that infection and awareness have very different trends. In contrast to infection trends, awareness trends have little regional variation, and our experiments suggest that public awareness is primarily driven by news media. Introduction Influenza surveillance plays a critical role in the public health mission of agencies at the national and local level. However, infection is only part of the story. Several studies have pointed out that a population’s awareness of a disease, and their reaction to it, are major factors that can influence the spread of a disease (Funk et al. 2009; Jones and Salathe 2009; Granell, Gomez, and Arenas 2013). Public health organizations must often manage the perceptions of a population to effectively respond to an outbreak. However, there is no effective, efficient and up-to-date method for tracking population awareness of influenza in the way that there are clinical surveillance systems for influenza infection. Enter web and social media data. In the past few years, there have been numerous studies demonstrating the ability of these data sources to provide cheap, real-time data for influenza surveillance. Google Flu Trends (GFT) (Ginsberg et al. 2009), among other systems (Yuan et al. 2013; Santillana et al. 2014; Preis and Moat 2014), demonstrated the ability to track flu rates based on search queries. Work using Twitter extended this ability to social media (Culotta 2010; Aramaki, Maskawa, and Morita 2011; Lampos and Cristianini 2012). These data sources are now recognized as real-time enhancements to the existing traditional influenza surveillance infrastructure. However, during the 2012–13 flu season, which we examine in this study, GFT received widespread criticism Copyright c © 2015, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. for failing to accurately track the flu (Lazer et al. 2014; Santillana et al. 2014). A primary cause of the inaccuracies was the conflation of search behaviors looking for general information about the flu (awareness) versus those by searchers looking for treatment information (infection). By failing to recognize this distinction, GFT greatly overestimated the infection rate. In contrast, work on Twitter by Lamb, Paul, and Dredze (2013) and Broniatowski, Paul, and Dredze (2013) explicitly modeled this distinction between awareness and infection. This work demonstrated that statistical machine learning classifiers could effectively differentiate awareness from infection. This allowed for the isolation of influenza infection tweets for surveillance, improving over GFT (Paul, Dredze, and Broniatowski 2014). In this paper, we consider the tweets previously ignored during influenza surveillance: tweets that demonstrate awareness of flu. Twitter provides a solution to the awareness surveillance problem. While those that are concerned about flu in general may not go to their doctor (and are thus not captured by clinical surveillance systems), they are likely to share these concerns with others, including via social media. This paper presents steps toward understanding and building a real-time surveillance system for disease awareness during an epidemic, specifically that of influenza. The construction of this system allows us to not only provide public health officials with awareness trends, for which they often have no other data source, but also study what drives awareness of influenza in a population. Our analysis will validate our surveillance capabilities—when gold standard data is available—and characterize the surveillance signal to determine which factors most influence a population’s awareness. Data This study focuses on the 2012–13 flu season in the United States, from September 30, 2012 (week 40) to May 25, 2013 (week 21). This is a particularly relevant flu season for this study since it had a high peak rate of infection and received considerable media attention. We consider national data as well as regional data for the ten regions defined by the Department of Health and Human Services.1 1 http://cdc.gov/flu/weekly/regions2009-2010/hhssenusmap.htm Twitter We downloaded Twitter flu trends from HealthTweets.org (Dredze et al. 2014), which tracks weekly levels of influenza in the United States. The data are based on the state-of-theart influenza system of Lamb, Paul, and Dredze (2013). This system uses NLP to classify flu-related tweets into two categories: tweets that indicate a personal infection (“I’m sick with the flu”) and tweets that express an awareness of influenza (“worried about getting the flu”). Our study makes use of both types of tweets. To estimate the level of influenza in a given week, we count the number of tweets classified as infection-related and we normalize the count by the total number of publicly available tweets published that week. We similarly estimate the level of influenza awareness in a given week by using normalized counts of tweets classified as awareness-related. We restrict our experiments to tweets from the US, using the Carmen geolocation toolkit (Dredze et al. 2013). Full details of the computational pipeline can be found in Broniatowski, Paul, and Dredze (2013). Government Influenza Statistics The Centers for Disease Control and Prevention (CDC) maintains the US Outpatient Influenza-like Illness Surveillance Network (ILINet), a series of healthcare facilities throughout the United States that report weekly cases of influenza-like illness. The CDC releases weekly numbers from the network throughout flu season. The data are publicly available via the CDC’s flu dashboard2 for current and past flu seasons. For this study we obtained the latest available ILINet values for the 2012–13 flu season, using the normalized “weighted ILI” values that adjust for regional differences. News Media Volume We obtained weekly counts for the number of flu-related newspaper articles from NewsLibrary.com, a news archive indexing thousands of US publications. We queried for articles matching the keywords “influenza” or “flu” in the headline or body. We also obtained the US state of the publication associated with each article (if not national), resulting in weekly article counts for each location. Analysis We organize our analysis towards answering the question: what factors influence the public’s awareness of influenza? We address this question in multiple stages. Comparison to Gold Standard Data We first compare the Twitter data—both infection and awareness—to the CDC’s ILINet data, which is considered the gold standard for measuring prevalence of influenza-like illness in the US. The purpose is twofold. First, it evaluates the ability of the Twitter infection classifier to track influenza prevalence in the ten HHS regions, which are a level of geographic granularity at which the HealthTweets.org 2 http://gis.cdc.gov/grasp/fluview/fluportaldashboard.html Region Infection Awareness 1 .802 .588 2 .804 .620 3 .815 .575 4 .812 .489 5 .818 .547 6 .868 .633 7 .885 .626 8 .869 .667 9 .778 .548 10 .846 .658 National .827 .555 Table 1: Correlations between the Twitter infection and awareness data and the CDC’s ILINet influenza prevalence data. data have been validated. Second, and more central to this study, it illustrates how well influenza awareness aligns with influenza prevalence. Table 1 shows the Pearson correlation between the weekly ILINet data and the two types of Twitter data. At the national level, the awareness data have a significantly lower (p=.029) correlation than the infection data, which shows that the distinction between infection and awareness matters for influenza surveillance, and it also shows that awareness does not fully align with prevalence. Characteristics of Awareness We seek to understand the general characteristics of how the weekly awareness levels vary across the season, in contrast to infection levels, which have well-understood trends. Both the Twitter-derived awareness and infection values for all ten regions across the 2012–13 season can be seen in Figure 1(a). Looking at the figure, we make several observations. First, the regional awareness trends are more similar to each other than the regional infection trends. Awareness rises sharply in week 2 of 2013 in all ten regions, peaking in eight of the ten regions. The infection trends, in contrast, have more variation in their peaks. We note that the cities of Boston and New York respectively declared public health emergencies in response to the influenza epidemics on January 103 and 12,4 corresponding to week 2 of 2013, which is when most of the awareness trends peaked. News of these epidemics may have driven national awareness. We quantified the variance of the two trends across regions by computing the Pearson correlation between each region’s values and the values from all other regions (that is, the national value if the states from the current region are excluded). Averaged across the ten regions, the infection levels had a mean correlation of .984 (SD .011), while infection levels had mean .963 (SD .036). Thus, each region’s trend was generally very similar to all other regions for both awareness and infection, but this was particularly true for 3 http://www.theguardian.com/world/2013/jan/10/ flu-boston-massachusetts-health-emergency 4 http://www.reuters.com/article/2013/01/12/ us-usa-flu-idUSBRE9080WD20130112