YASO: A New Benchmark for Targeted Sentiment Analysis

Sentiment analysis research has shifted over the years from the analysis of full documents or single sentences to a finer-level of detail – identifying the sentiment towards single words or phrases – with the task of Targeted Sentiment Analysis (TSA). While this problem is attracting a plethora of works focusing on algorithmic aspects, they are typically evaluated on a selection from a handful of datasets, and little effort, if any, is dedicated to the expansion of the available evaluation data. In this work, we present YASO – a new crowd-sourced TSA evaluation dataset, collected using a new annotation scheme for labeling targets and their sentiments. The dataset contains 2215 English sentences from movie, business and product reviews, and 7415 terms and their corresponding sentiments annotated within these sentences. Our analysis verifies the reliability of our annotations, and explores the characteristics of the collected data. Lastly, benchmark results using five contemporary TSA systems lay the foundation for future work, and show there is ample room for improvement on this challenging new dataset.

[1]  Luyao Huang,et al.  Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence , 2019, NAACL.

[2]  Christopher Potts,et al.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , 2013, EMNLP.

[3]  Xing Xie,et al.  Exploring Sequence-to-Sequence Learning in Aspect Term Extraction , 2019, ACL.

[4]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[5]  Ting Liu,et al.  Aspect Level Sentiment Classification with Deep Memory Network , 2016, EMNLP.

[6]  Erik Cambria,et al.  Targeted Aspect-Based Sentiment Analysis via Embedding Commonsense Knowledge into an Attentive LSTM , 2018, AAAI.

[7]  Philip Ogunbona,et al.  Modelling Context and Syntactical Features for Aspect-based Sentiment Analysis , 2020, ACL.

[8]  Hao Li,et al.  Learning Explicit and Implicit Structures for Targeted Sentiment Analysis , 2019, EMNLP/IJCNLP.

[9]  Philip S. Yu,et al.  BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis , 2019, NAACL.

[10]  Haris Papageorgiou,et al.  SemEval-2016 Task 5: Aspect Based Sentiment Analysis , 2016, *SEMEVAL.

[11]  Andreas van Cranenburgh,et al.  Embarrassingly Simple Unsupervised Aspect Extraction , 2020, ACL.

[12]  Jianfei Yu,et al.  Unified Feature and Instance Based Domain Adaptation for Aspect-Based Sentiment Analysis , 2020, EMNLP.

[13]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[14]  Sebastian Stabinger,et al.  Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification , 2020, LREC.

[15]  Jiaxin Pei,et al.  Targeted Sentiment Analysis: A Data-Driven Categorization , 2019, ArXiv.

[16]  Tieyun Qian,et al.  Relation-Aware Collaborative Learning for Unified Aspect-Based Sentiment Analysis , 2020, ACL.

[17]  John G. Breslin,et al.  A Hierarchical Model of Reviews for Aspect-based Sentiment Analysis , 2016, EMNLP.

[18]  Tao Jiang,et al.  Attentional Encoder Network for Targeted Sentiment Classification , 2019, ICANN.

[19]  Hwee Tou Ng,et al.  Effective Attention Modeling for Aspect-Level Sentiment Classification , 2018, COLING.

[20]  Lu Xu,et al.  Position-Aware Tagging for Aspect Sentiment Triplet Extraction , 2020, EMNLP.

[21]  Bo Pang,et al.  Thumbs up? Sentiment Classification using Machine Learning Techniques , 2002, EMNLP.

[22]  Xiaokui Xiao,et al.  Recursive Neural Conditional Random Fields for Aspect-based Sentiment Analysis , 2016, EMNLP.

[23]  Suresh Manandhar,et al.  SemEval-2014 Task 4: Aspect Based Sentiment Analysis , 2014, *SEMEVAL.

[24]  Lu Xu,et al.  Aspect Based Sentiment Analysis with Aspect-Specific Opinion Spans , 2020, EMNLP.

[25]  Xiaocheng Feng,et al.  Effective LSTMs for Target-Dependent Sentiment Classification , 2015, COLING.

[26]  Kathleen M. Carley,et al.  Aspect Level Sentiment Classification with Attention-over-Attention Neural Networks , 2018, SBP-BRiMS.

[27]  Shuai Wang,et al.  Deep learning for sentiment analysis: A survey , 2018, WIREs Data Mining Knowl. Discov..

[28]  Philipp Koehn,et al.  Synthesis Lectures on Human Language Technologies , 2016 .

[29]  Akbar Karimi,et al.  Adversarial Training for Aspect-Based Sentiment Analysis with BERT , 2020, ArXiv.

[30]  Jun Yang,et al.  Multi-Entity Aspect-Based Sentiment Analysis With Context, Entity and Aspect Memory , 2018, AAAI.

[31]  Lidong Bing,et al.  Exploiting BERT for End-to-End Aspect-based Sentiment Analysis , 2019, EMNLP.

[32]  Xinyan Xiao,et al.  SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis , 2020, ACL.

[33]  Lei Zhang,et al.  Sentiment Analysis and Opinion Mining , 2017, Encyclopedia of Machine Learning and Data Mining.

[34]  Xiaokui Xiao,et al.  Coupled Multi-Layer Attentions for Co-Extraction of Aspect and Opinion Terms , 2017, AAAI.

[35]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[36]  Jiawei Han,et al.  Opinosis: A Graph Based Approach to Abstractive Summarization of Highly Redundant Opinions , 2010, COLING.

[37]  Kim Schouten,et al.  Survey on Aspect-Level Sentiment Analysis , 2016, IEEE Transactions on Knowledge and Data Engineering.

[38]  Ming Zhou,et al.  Adaptive Recursive Neural Network for Target-dependent Twitter Sentiment Classification , 2014, ACL.

[39]  Jianfei Yu,et al.  Global Inference for Aspect and Opinion Terms Co-Extraction Based on Multi-Task Neural Networks , 2019, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[40]  Xin Li,et al.  Aspect Term Extraction with History Attention and Selective Transformation , 2018, IJCAI.

[41]  Heng Yang,et al.  LCF: A Local Context Focus Mechanism for Aspect-Based Sentiment Classification , 2019, Applied Sciences.

[42]  Guillaume Bouchard,et al.  SentiHood: Targeted Aspect Based Sentiment Analysis Dataset for Urban Neighbourhoods , 2016, COLING.

[43]  Benjamin Van Durme,et al.  Open Domain Targeted Sentiment , 2013, EMNLP.

[44]  Heng Yang,et al.  A Multi-task Learning Model for Chinese-oriented Aspect Polarity Classification and Aspect Term Extraction , 2021, Neurocomputing.

[45]  Luo Si,et al.  Knowing What, How and Why: A Near Complete Solution for Aspect-based Sentiment Analysis , 2019, AAAI.