An evaluation of machine translation for multilingual sentence-level sentiment analysis

Sentiment analysis has become a key tool for several social media applications, including analysis of user's opinions about products and services, support to politics during campaigns and even for market trending. There are multiple existing sentiment analysis methods that explore different techniques, usually relying on lexical resources or learning approaches. Despite the large interest on this theme and amount of research efforts in the field, almost all existing methods are designed to work with only English content. Most existing strategies in specific languages consist of adapting existing lexical resources, without presenting proper validations and basic baseline comparisons. In this paper, we take a different step into this field. We focus on evaluating existing efforts proposed to do language specific sentiment analysis. To do it, we evaluated twenty-one methods for sentence-level sentiment analysis proposed for English, comparing them with two language-specific methods. Based on nine language-specific datasets, we provide an extensive quantitative analysis of existing multi-language approaches. Our main result suggests that simply translating the input text on a specific language to English and then using one of the existing English methods can be better than the existing language specific efforts evaluated. We also rank those implementations comparing their prediction performance and identifying the methods that acquired the best results using machine translation across different languages. As a final contribution to the research community, we release our codes and datasets. We hope our effort can help sentiment analysis to become English independent.

[1]  Andrea Esuli,et al.  SENTIWORDNET: A Publicly Available Lexical Resource for Opinion Mining , 2006, LREC.

[2]  Clement Levallois,et al.  Umigon: Sentiment Analysis for Tweets Based on Lexicons and Heuristics , 2013 .

[3]  M. Thelwall Heart and Soul : Sentiment Strength Detection in the Social Web with , 2013 .

[4]  Christopher M. Danforth,et al.  Measuring the Happiness of Large-Scale Written Expression: Songs, Blogs, and Presidents , 2010, ArXiv.

[5]  Saif Mohammad,et al.  #Emotional Tweets , 2012, *SEMEVAL.

[6]  Saif Mohammad,et al.  CROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON , 2013, Comput. Intell..

[7]  Mike Thelwall,et al.  The Heart and Soul of the Web? Sentiment Strength Detection in the Social Web with SentiStrength , 2017 .

[8]  Rada Mihalcea,et al.  Multilingual Subjectivity Analysis Using Machine Translation , 2008, EMNLP.

[9]  Gerhard Heyer,et al.  SentiWS - A Publicly Available German-language Resource for Sentiment Analysis , 2010, LREC.

[10]  Nafissa Yussupova,et al.  Applying of Sentiment Analysis for Texts in Russian Based on Machine Learning Approach , 2012 .

[11]  Walter Daelemans,et al.  Pattern for Python , 2012, J. Mach. Learn. Res..

[12]  Finn Årup Nielsen,et al.  A New ANEW: Evaluation of a Word List for Sentiment Analysis in Microblogs , 2011, #MSM.

[13]  Sune Lehmann,et al.  Tweetin' in the Rain: Exploring Societal-Scale Effects of Weather on Mood , 2012, ICWSM.

[14]  Andrei Popescu-Belis,et al.  Sentiment analysis of user comments for one-class collaborative filtering over ted talks , 2013, SIGIR.

[15]  Renata Vieira,et al.  Sentiment Analysis on Twitter Data for Portuguese Language , 2012, PROPOR.

[16]  Mahmoud Al-Ayyoub,et al.  Arabic sentiment analysis: Lexicon-based and corpus-based , 2013, 2013 IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies (AEECT).

[17]  Christopher Potts,et al.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , 2013, EMNLP.

[18]  Riham Mansour,et al.  Cross-lingual Twitter Polarity Detection via Projection across Word-Aligned Corpora , 2014 .

[19]  Fabrício Benevenuto,et al.  Comparing and combining sentiment analysis methods , 2013, COSN '13.

[20]  Shrikanth S. Narayanan,et al.  A System for Real-time Twitter Sentiment Analysis of 2012 U.S. Presidential Election Cycle , 2012, ACL.

[21]  Júlio Cesar dos Reis,et al.  Breaking the News: First Impressions Matter on Online News , 2015, ICWSM.

[22]  Mykola Pechenizkiy,et al.  Cross-lingual polarity detection with machine translation , 2013, WISDOM '13.

[23]  Fabrício Benevenuto,et al.  iFeel: a system that compares and combines sentiment analysis methods , 2014, WWW.

[24]  S. Albayrak,et al.  Language-Independent Twitter Sentiment Analysis , 2012 .

[25]  Fabrício Benevenuto,et al.  A Benchmark Comparison of State-of-the-Practice Sentiment Analysis Methods , 2015, ArXiv.

[26]  Fabrício Benevenuto,et al.  PANAS-t: A Pychometric Scale for Measuring Sentiments on Twitter , 2013, ArXiv.

[27]  Milam Aiken,et al.  An Analysis of Google Translate Accuracy , 2012 .

[28]  Erik Cambria,et al.  SenticNet: A Publicly Available Semantic Resource for Opinion Mining , 2010, AAAI Fall Symposium: Commonsense Knowledge.

[29]  Zheng Lin,et al.  Language-independent sentiment classification using three common words , 2011, CIKM '11.

[30]  Eric Gilbert,et al.  VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text , 2014, ICWSM.

[31]  Maite Taboada,et al.  Lexicon-Based Methods for Sentiment Analysis , 2011, CL.

[32]  Bing Liu,et al.  Mining and summarizing customer reviews , 2004, KDD.

[33]  Claire Cardie,et al.  OpinionFinder: A System for Subjectivity Analysis , 2005, HLT.

[34]  Alexandra Balahur,et al.  Multilingual Sentiment Analysis using Machine Translation? , 2012, WASSA@ACL.

[35]  Fabrício Benevenuto,et al.  Exploiting New Sentiment-Based Meta-level Features for Effective Sentiment Analysis , 2016, WSDM.

[36]  Muhammad Abdul-Mageed,et al.  Subjectivity and Sentiment Analysis of Modern Standard Arabic , 2011, ACL.

[37]  Saif Mohammad,et al.  NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets , 2013, *SEMEVAL.

[38]  J. Pennebaker,et al.  The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods , 2010 .

[39]  Tao Qin,et al.  LETOR: A benchmark collection for research on learning to rank for information retrieval , 2010, Information Retrieval.