SGPT: Semantic Graphs based Pre-training for Aspect-based Sentiment Analysis

Previous studies show effective of pre-trained language models for sentiment analysis. However, most of these studies ignore the importance of sentimental information for pretrained models. Therefore, we fully investigate the sentimental information for pretrained models and enhance pre-trained language models with semantic graphs for sentiment analysis. In particular, we introduce Semantic Graphs based Pre-training(SGPT) using semantic graphs to obtain synonym knowledge for aspect-sentiment pairs and similar aspect/sentiment terms. We then optimize the pre-trained language model with the semantic graphs. Empirical studies on several downstream tasks show that proposed model outperforms strong pre-trained baselines. The results also show the effectiveness of proposed semantic graphs for pre-trained model.

[1]  Christopher Potts,et al.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , 2013, EMNLP.

[2]  Guokun Lai,et al.  RACE: Large-scale ReAding Comprehension Dataset From Examinations , 2017, EMNLP.

[3]  Jianfei Yu,et al.  Recurrent Neural Networks with Auxiliary Labels for Cross-Domain Opinion Target Extraction , 2017, AAAI.

[4]  Claire Cardie,et al.  Annotating Expressions of Opinions and Emotions in Language , 2005, Lang. Resour. Evaluation.

[5]  Maosong Sun,et al.  ERNIE: Enhanced Language Representation with Informative Entities , 2019, ACL.

[6]  Anette Frank,et al.  SRL4ORL: Improving Opinion Role Labeling Using Multi-Task Learning with Semantic Role Labeling , 2017, NAACL.

[7]  Navneet Kaur,et al.  Opinion mining and sentiment analysis , 2016, 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom).

[8]  Xin Liu,et al.  A Variational Approach to Weakly Supervised Document-Level Multi-Aspect Sentiment Classification , 2019, NAACL.

[9]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[10]  Yue Zhang,et al.  Target-Dependent Twitter Sentiment Classification with Rich Automatic Features , 2015, IJCAI.

[11]  Philipp Koehn,et al.  Synthesis Lectures on Human Language Technologies , 2016 .

[12]  Ruslan Salakhutdinov,et al.  Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks , 2016, ICLR.

[13]  Tianyu Gao,et al.  KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation , 2019, ArXiv.

[14]  Shuai Wang,et al.  Deep learning for sentiment analysis: A survey , 2018, WIREs Data Mining Knowl. Discov..

[15]  Roy Schwartz,et al.  Knowledge Enhanced Contextual Word Representations , 2019, EMNLP/IJCNLP.

[16]  Xiaoyan Zhu,et al.  Linguistically Regularized LSTMs for Sentiment Classification , 2016, ArXiv.

[17]  Maite Taboada,et al.  Lexicon-Based Methods for Sentiment Analysis , 2011, CL.

[18]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[19]  Xuanjing Huang,et al.  K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters , 2020, FINDINGS.

[20]  Xiang Zhang,et al.  Character-level Convolutional Networks for Text Classification , 2015, NIPS.

[21]  TaboadaMaite,et al.  Lexicon-based methods for sentiment analysis , 2011 .

[22]  Peter D. Turney.  Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews , 2002, ACL.

[23]  Suresh Manandhar,et al.  SemEval-2014 Task 4: Aspect Based Sentiment Analysis , 2014, *SEMEVAL.

[24]  Owen Rambow,et al.  Sentiment Analysis of Twitter Data , 2011 .

[25]  Xinyan Xiao,et al.  SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis , 2020, ACL.

[26]  Theresa Wilson.  Fine-grained subjectivity and sentiment analysis: recognizing the intensity, polarity, and attitudes of private states , 2008 .

[27]  Zhe Zhao,et al.  K-BERT: Enabling Language Representation with Knowledge Graph , 2019, AAAI.

[28]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[29]  Qin Lu,et al.  A Question Answering Approach for Emotion Cause Extraction , 2017, EMNLP.

[30]  Lidong Bing,et al.  A Knowledge Regularized Hierarchical Approach for Emotion Cause Analysis , 2019, EMNLP.

[31]  Lei Zhang,et al.  Sentiment Analysis and Opinion Mining , 2017, Encyclopedia of Machine Learning and Data Mining.

[32]  Xin Li,et al.  Deep Multi-Task Learning for Aspect Term Extraction with Memory Interaction , 2017, EMNLP.

[33]  Minlie Huang,et al.  SentiLARE: Linguistic Knowledge Enhanced Language Representation for Sentiment Analysis , 2019, EMNLP.

[34]  Bonggun Shin,et al.  Lexicon Integrated CNN Models with Attention for Sentiment Analysis , 2016, WASSA@EMNLP.