An improved aspect-category sentiment analysis model for text sentiment analysis based on RoBERTa

The aspect-category sentiment analysis can provide more and deeper information than the document-level sentiment analysis, because it aims to predict the sentiment polarities of different aspect categories in the same text. The main challenge of aspect-category sentiment analysis is that different aspect categories may present different polarities in the same text. Previous studies combine the Long Short-Term Memory (LSTM) and attention mechanism to predict the sentiment polarity of the given aspect category, but the LSTM-based methods are not really bidirectional text feature extraction methods. In this paper, we propose a multi-task aspect-category sentiment analysis model based on RoBERTa (Robustly Optimized BERT Pre-training Approach). Treating each aspect category as a subtask, we employ the RoBERTa based on deep bidirectional Transformer to extract features from both text and aspect tokens, and apply the cross-attention mechanism to guide the model to focus on the features most relevant to the given aspect category. According to the experimental results, the proposed model outperforms other models for comparison in aspect-category sentiment analysis. Furthermore, the influencing factors of our proposed model are also analyzed.

[1]  Yuan Luo,et al.  Graph Convolutional Networks for Text Classification , 2018, AAAI.

[2]  Yue Zhang,et al.  Attention Modeling for Targeted Sentiment , 2017, EACL.

[3]  Saif Mohammad,et al.  NRC-Canada-2014: Detecting Aspects and Sentiment in Customer Reviews , 2014, *SEMEVAL.

[4]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[5]  Lillian Lee,et al.  Opinion Mining and Sentiment Analysis , 2008, Found. Trends Inf. Retr..

[6]  Flavius Frasincar,et al.  Supervised and Unsupervised Aspect Category Detection for Sentiment Analysis with Co-occurrence Data , 2018, IEEE Transactions on Cybernetics.

[7]  Ao Feng,et al.  Target-Dependent Sentiment Classification With BERT , 2019, IEEE Access.

[8]  Christopher D. Manning,et al.  Baselines and Bigrams: Simple, Good Sentiment and Topic Classification , 2012, ACL.

[9]  Giuseppe De Pietro,et al.  Best Practices of Convolutional Neural Networks for Question Classification , 2020, Applied Sciences.

[10]  Haipeng Yao,et al.  A novel sentence similarity model with word embedding based on convolutional neural network , 2018, Concurr. Comput. Pract. Exp..

[11]  Delip Rao,et al.  Semi-Supervised Polarity Lexicon Induction , 2009, EACL.

[12]  Hamido Fujita,et al.  Word Sense Disambiguation: A comprehensive knowledge exploitation framework , 2020, Knowl. Based Syst..

[13]  Reda Alhajj,et al.  Emotion and sentiment analysis from Twitter text , 2019, J. Comput. Sci..

[14]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[15]  Mohammed Benkhalifa,et al.  WordNet based Implicit Aspect Sentiment Analysis for Crime Identification from Twitter , 2018 .

[16]  Li Zhao,et al.  Attention-based LSTM for Aspect-level Sentiment Classification , 2016, EMNLP.

[17]  Li-xia Luo.  Network text sentiment analysis method combining LDA text representation and GRU-CNN , 2018, Personal and Ubiquitous Computing.

[18]  Ting Liu,et al.  Attention-over-Attention Neural Networks for Reading Comprehension , 2016, ACL.

[19]  Tao Li,et al.  Aspect Based Sentiment Analysis with Gated Convolutional Networks , 2018, ACL.

[20]  Hwee Tou Ng,et al.  Effective Attention Modeling for Aspect-Level Sentiment Classification , 2018, COLING.

[21]  Wenchao Xu,et al.  Aspect based fine-grained sentiment analysis for online reviews , 2019, Inf. Sci..

[22]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[23]  Kyung-shik Shin,et al.  Attention-based long short-term memory network using sentiment lexicon embedding for aspect-level sentiment analysis in Korean , 2019, Inf. Process. Manag..

[24]  Giuseppe De Pietro,et al.  Hybrid query expansion using lexical resources and word embeddings for sentence retrieval in question answering , 2020, Inf. Sci..

[25]  Tang Da-quan.  Word’s semantic similarity computation method based on Hownet , 2010 .

[26]  Philip S. Yu,et al.  BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis , 2019, NAACL.

[27]  Ning Liu,et al.  Aspect-based sentiment analysis with gated alternate neural network , 2020, Knowl. Based Syst..

[28]  Xianghua Fu,et al.  Multi-aspect sentiment analysis for Chinese online social reviews based on topic modeling and HowNet lexicon , 2013, Knowl. Based Syst..

[29]  Doaa Mohey El Din Mohamed Hussein,et al.  A survey on sentiment analysis challenges , 2016, Journal of King Saud University - Engineering Sciences.

[30]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[31]  Muhammad Usman,et al.  Tourism Mobile App With Aspect-Based Sentiment Classification Framework for Tourist Reviews , 2019, IEEE Transactions on Consumer Electronics.

[32]  Ting Liu,et al.  Aspect Level Sentiment Classification with Deep Memory Network , 2016, EMNLP.

[33]  Mahmoud Al-Ayyoub,et al.  Using long short-term memory deep neural networks for aspect-based sentiment analysis of Arabic reviews , 2018, International Journal of Machine Learning and Cybernetics.

[34]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[35]  Wenji Mao,et al.  A Co-Memory Network for Multimodal Sentiment Analysis , 2018, SIGIR.

[36]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[37]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[38]  Dongyan Zhao,et al.  Multi-grained Attention Network for Aspect-Level Sentiment Classification , 2018, EMNLP.

[39]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[40]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.