SemEval-2013 Task 2: Sentiment Analysis in Twitter

In recent years, sentiment analysis in social media has attracted a lot of research interest and has been used for a number of applications. Unfortunately, research has been hindered by the lack of suitable datasets, complicating the comparison between approaches. To address this issue, we have proposed SemEval-2013 Task 2: Sentiment Analysis in Twitter, which included two subtasks: A, an expression-level subtask, and B, a messagelevel subtask. We used crowdsourcing on Amazon Mechanical Turk to label a large Twitter training dataset along with additional test sets of Twitter and SMS messages for both subtasks. All datasets used in the evaluation are released to the research community. The task attracted significant interest and a total of 149 submissions from 44 teams. The bestperforming team achieved an F1 of 88.9% and 69% for subtasks A and B, respectively.

[1]  Andrea Esuli,et al.  Optimizing Text Quantifiers for Multivariate Loss Functions , 2015, TKDD.

[2]  Ari Rappoport,et al.  Semi-Supervised Recognition of Sarcasm in Twitter and Amazon , 2010, CoNLL.

[3]  Fabrizio Sebastiani,et al.  An Axiomatically Derived Measure for the Evaluation of Classification Algorithms , 2015, ICTIR.

[4]  Tommaso Caselli,et al.  SemEval-2015 Task 9: CLIPEval Implicit Polarity of Events , 2015, *SEMEVAL.

[5]  Preslav Nakov,et al.  SemEval-2014 Task 9: Sentiment Analysis in Twitter , 2014, *SEMEVAL.

[6]  Carlo Strapparava,et al.  SemEval-2007 Task 14: Affective Text , 2007, Fourth International Workshop on Semantic Evaluations (SemEval-2007).

[7]  Verena Rieser,et al.  Benchmarking Machine Translated Sentiment Analysis for Arabic Tweets , 2015, HLT-NAACL.

[8]  Haris Papageorgiou,et al.  SemEval-2016 Task 5: Aspect Based Sentiment Analysis , 2016, *SEMEVAL.

[9]  Andrea Esuli,et al.  Sentiment Quantification , 2010, IEEE Intell. Syst..

[10]  Sune Lehmann,et al.  Understanding the Demographics of Twitter Users , 2011, ICWSM.

[11]  Walid Magdy,et al.  Content and Network Dynamics Behind Egyptian Political Polarization on Twitter , 2014, CSCW.

[12]  José Carlos González,et al.  TASS - Workshop on Sentiment Analysis at SEPLN , 2013, Proces. del Leng. Natural.

[13]  Bernard J. Jansen,et al.  Twitter power: Tweets as electronic word of mouth , 2009 .

[14]  Roberto Basili,et al.  A context-based model for Sentiment Analysis in Twitter , 2014, COLING.

[15]  Muhammad Abdul-Mageed,et al.  SAMAR: Subjectivity and sentiment analysis for Arabic social media , 2014, Comput. Speech Lang..

[16]  George Forman,et al.  Quantifying counts and costs via classification , 2008, Data Mining and Knowledge Discovery.

[17]  Geoff Holmes,et al.  Detecting Sentiment Change in Twitter Streaming Data , 2011, WAPA.

[18]  Ismail Hakki Toroslu,et al.  Transfer Learning Using Twitter Data for Improving Sentiment Classification of Turkish Political News , 2013, ISCIS.

[19]  Dong Nguyen,et al.  "How Old Do You Think I Am?" A Study of Language and Age in Twitter , 2013, ICWSM.

[20]  José Carlos González Cristóbal,et al.  TASS - Workshop on Sentiment Analysis at SEPLN , 2013 .

[21]  Suresh Manandhar,et al.  SemEval-2014 Task 4: Aspect Based Sentiment Analysis , 2014, *SEMEVAL.

[22]  Andrea Esuli,et al.  Evaluation Measures for Ordinal Regression , 2009, 2009 Ninth International Conference on Intelligent Systems Design and Applications.

[23]  Ari Rappoport,et al.  Semi-Supervised Recognition of Sarcasm in Twitter and Amazon , 2010, CoNLL.

[24]  Tao Chen,et al.  Creating a live, public short message service corpus: the NUS SMS corpus , 2011, Lang. Resour. Evaluation.

[25]  Isabell M. Welpe,et al.  Predicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment , 2010, ICWSM.

[26]  Antal van den Bosch,et al.  The perfect solution for detecting sarcasm in tweets #not , 2013, WASSA@NAACL-HLT.

[27]  Paolo Rosso,et al.  SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter , 2015, *SEMEVAL.

[28]  Oren Etzioni,et al.  Named Entity Recognition in Tweets: An Experimental Study , 2011, EMNLP.

[29]  Johanna D. Moore,et al.  Twitter Sentiment Analysis: The Good the Bad and the OMG! , 2011, ICWSM.

[30]  Bo Pang,et al.  Thumbs up? Sentiment Classification using Machine Learning Techniques , 2002, EMNLP.

[31]  David Yarowsky,et al.  Exploring Demographic Language Variations to Improve Multilingual Sentiment Analysis in Social Media , 2013, EMNLP.

[32]  George Forman,et al.  Counting Positives Accurately Despite Inaccurate Classification , 2005, ECML.

[33]  Dirk Hovy,et al.  Demographic Factors Improve Classification Performance , 2015, ACL.

[34]  Patrick Paroubek,et al.  Twitter Based System: Using Twitter for Disambiguating Sentiment Ambiguous Adjectives , 2010, *SEMEVAL.

[35]  Saif Mohammad,et al.  NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets , 2013, *SEMEVAL.

[36]  Muhammad Abdul-Mageed,et al.  Subjectivity and Sentiment Annotation of Modern Standard Arabic Newswire , 2011, Linguistic Annotation Workshop.

[37]  Andrea Esuli,et al.  SentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining , 2010, LREC.

[38]  Preslav Nakov,et al.  SemEval-2015 Task 10: Sentiment Analysis in Twitter , 2015, *SEMEVAL.

[39]  Oren Etzioni,et al.  Open domain event extraction from twitter , 2012, KDD.

[40]  Muhannad Quwaider,et al.  Human Annotated Arabic Dataset of Book Reviews for Aspect Based Sentiment Analysis , 2015, 2015 3rd International Conference on Future Internet of Things and Cloud.

[41]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[42]  Brendan T. O'Connor,et al.  Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments , 2010, ACL.

[43]  Yi Yang,et al.  Putting Things in Context: Community-specific Embedding Projections for Sentiment Analysis , 2015, ArXiv.

[44]  Bernard J. Jansen,et al.  Twitter power: Tweets as electronic word of mouth , 2009, J. Assoc. Inf. Sci. Technol..

[45]  Suzan Burton,et al.  Interactive or reactive? : marketing with Twitter , 2011 .

[46]  Christopher M. Danforth,et al.  Temporal Patterns of Happiness and Information in a Global Social Network: Hedonometrics and Twitter , 2011, PloS one.

[47]  Leonidas J. Guibas,et al.  The Earth Mover's Distance as a Metric for Image Retrieval , 2000, International Journal of Computer Vision.

[48]  Ingmar Weber,et al.  Twitter: A Digital Socioscope , 2015 .

[49]  Nina Wacholder,et al.  Identifying Sarcasm in Twitter: A Closer Look , 2011, ACL.

[50]  Nathanael Chambers,et al.  Learning for Microblogs with Distant Supervision: Political Forecasting with Twitter , 2012, EACL.

[51]  Saif Mohammad,et al.  Sentiment after Translation: A Case-Study on Arabic Social Media Posts , 2015, NAACL.

[52]  Johan Bollen,et al.  Twitter mood predicts the stock market , 2010, J. Comput. Sci..

[53]  Kareem Darwish,et al.  Subjectivity and Sentiment Analysis of Modern Standard Arabic and Arabic Microblogs , 2013, WASSA@NAACL-HLT.

[54]  Claire Cardie,et al.  Annotating Expressions of Opinions and Emotions in Language , 2005, Lang. Resour. Evaluation.

[55]  Gabriella Pasi,et al.  Clustering with Error-Estimation for Monitoring Reputation of Companies on Twitter , 2013, AIRS.

[56]  Junlan Feng,et al.  Robust Sentiment Detection on Twitter from Biased and Noisy Data , 2010, COLING.

[57]  Brendan T. O'Connor,et al.  From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series , 2010, ICWSM.

[58]  Geoffrey Zweig,et al.  Linguistic Regularities in Continuous Space Word Representations , 2013, NAACL.

[59]  Akshi Kumar,et al.  Sentiment Analysis on Twitter , 2012 .