Assessing the Accuracy of Automated Twitter Sentiment Coding

ABSTRACT Social media have provided consumers with numerous outlets for disseminating their brand-related comments. Given the impact of these comments on brand image and brand success, marketers have begun to rely on third-party companies to automatically track, collect and analyze these comments with regard to their content and sentiment. There is, however, an absence of controlled research which objectively and systematically assesses the accuracy of these companies' automated sentiment coding. This research evaluated the automated sentiment coding accuracy and misclassification errors of six leading third-party companies for a broad range of comment types and forms. Overall, automated sentiment coding appears to have limited reliability and appears to be accurately accomplished only for very simple statements in which a keyword is used to convey its typical meaning. Statements without keywords or statements in which keyword meaning is reversed through negation or context are accurately coded at very low levels. Neutral statements appear to be problematic for some, but not all, companies. Implications of the research for the use of automated sentiment analysis for brand decisionmaking are presented. INTRODUCTION Marketers' attempt to manage brand image in the pre-social media era was straightforward. Brand managers would decide what they wanted consumers to think about their brand and then, through advertising and other consumer-directed communications, would tightly control the messages to which consumers were exposed. Ideally, these communications would create or reinforce the desired brand image. Communications planning took the perspective of "we ?? tell you the things that we think are really important [and what you should think] about our brand and product" (Bostic, 2010) as consumers had little opportunity to disseminate their own perspective. Social media have fundamentally altered how brand image is created, maintained and changed. Today's successful marketers understand that social media have reduced their control and they realize that they are but one (albeit important) contributor to a dialogue about the brand. These marketers realize that their brand's image will not only be the result of what they say on their brand's own behalf, but additionally, what consumers say and how they interact with and respond to these consumers' comments (Davenport & Beck, 2002; Esch et al., 2006). Consumers are now in at least a shared leadership position with regard to brand image creation, in essence telling marketers "your brand is whatever [we] say it is" (Li & Bernoff, 2008). Li and Bernoff (2008) refer to the transfer of power from businesses to individuals as "the groundswell." Examples of the groundswell are the abundance of online brand and product reviews as well as online communities such as Get Satisfaction (http://www. getsatisfaction.com), which provides a forum for customers to raise questions or complain about a wide range of companies, and for the resulting discussions to be displayed for other consumers to search and view. Online product reviews and sites such as Get Satisfaction can have a profound effect on brand image and brand success given that the consumer decision-making process is greatly influenced by other consumers' comments (Goldberg et al., 2001; Kelsey Group, 2007; Jansen et al., 2009), especially at the time of product evaluation and purchase (Zabin and Jefferies, 2008). Clearly, it is important for marketers to understand what consumers are saying about their brands and products. Specifically, marketers need to be aware of what consumers are saying (i.e., the specific comments about the brand) and they need to understand what consumers are feeling (i.e., the sentiment associated with a brand comment, that is, whether a particular comment is positive, negative or neutral). Cai et al. (2010) provide a rationale for the specific focus on sentiment monitoring: "The voice of the web to gain consumer, brand and market insights can be truly differentiating and valuable to today's corporations [and] one important form of insights can be derived from sentiment analysis. …