Bidirectional LSTM with self-attention mechanism and multi-channel features for sentiment classification

Abstract There are a lot of linguistic knowledge and sentiment resources nowadays, but in the current research with deep learning framework, these kinds of unique sentiment information are not fully used in sentiment analysis tasks. Moreover, the sentiment analysis task can be seen as a sequence model, and the sequence model has a problem: the model will decode the input file sequences into a specific length vector. If the length of the vectors is set too short, the input text information will be lost, and finally the text will be misjudged. To solve these problems, we propose a bidirectional LSTM model with self-attention mechanism and multi-channel features (SAMF-BiLSTM). The method models the existing linguistic knowledge and sentiment resources in sentiment analysis tasks to form different feature channels, and uses self-attention mechanism to enhance the sentiment information. SAMF-BiLSTM model can fully exploit the relationship between target words and sentiment polarity words in a sentence, and does not rely on manually organized sentiment lexicon. In addition, we propose the SAMF-BiLSTM-D model based on SAMF-BiLSTM model for document-level text classification tasks. The method obtains the representation of all sentences in the document through SAMF-BiLSTM training, then integrates BiLSTM to learn the representation of all sentences, and further obtains the sentiment feature information of the entire document. Finally, we evaluate experiment results under five datasets. The results show that SAMF-BiLSTM and SAMF-BiLSTM-D are superior to other advanced methods in classification accuracy in most cases.

[1]  Matthew D. Zeiler.  ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.

[2]  Ming Zhou,et al.  Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification , 2014, ACL.

[3]  Houfeng Wang,et al.  Interactive Attention Networks for Aspect-Level Sentiment Classification , 2017, IJCAI.

[4]  Paolo Gastaldo,et al.  Bayesian network based extreme learning machine for subjectivity detection , 2017, J. Frankl. Inst..

[5]  Quan Pan,et al.  Learning Word Representations for Sentiment Analysis , 2017, Cognitive Computation.

[6]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[7]  Jürgen Schmidhuber,et al.  Framewise phoneme classification with bidirectional LSTM and other neural network architectures , 2005, Neural Networks.

[8]  Swapna Somasundaran,et al.  Discourse Level Opinion Interpretation , 2008, COLING.

[9]  Xuanjing Huang,et al.  Cached Long Short-Term Memory Neural Networks for Document-Level Sentiment Classification , 2016, EMNLP.

[10]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[11]  Diego Reforgiato Recupero,et al.  Sentilo: Frame-Based Sentiment Analysis , 2014, Cognitive Computation.

[12]  Jürgen Schmidhuber,et al.  LSTM: A Search Space Odyssey , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[13]  Andrea Esuli,et al.  SENTIWORDNET: A Publicly Available Lexical Resource for Opinion Mining , 2006, LREC.

[14]  Min Yang,et al.  Feature-enhanced attention network for target-dependent sentiment classification , 2018, Neurocomputing.

[15]  Yunming Ye,et al.  Sentiment analysis through critic learning for optimizing convolutional neural networks with rules , 2019, Neurocomputing.

[16]  Christopher Potts,et al.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , 2013, EMNLP.

[17]  Liang Bin,et al.  Chinese Micro-Blog Sentiment Analysis Based on Multi-Channels Convolutional Neural Networks , 2018 .

[18]  Bowen Zhou,et al.  A Structured Self-attentive Sentence Embedding , 2017, ICLR.

[19]  Rohini K. Srihari,et al.  Using Verbs and Adjectives to Automatically Classify Blog Sentiment , 2006, AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs.

[20]  Min Yang,et al.  Investigating Capsule Networks with Dynamic Routing for Text Classification , 2018, EMNLP.

[21]  Raymond Y. K. Lau,et al.  A Probabilistic Generative Model for Mining Cybercriminal Networks from Online Social Media , 2014, IEEE Computational Intelligence Magazine.

[22]  Christopher D. Manning,et al.  Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks , 2015, ACL.

[23]  Erik Cambria,et al.  SenticNet 5: Discovering Conceptual Primitives for Sentiment Analysis by Means of Context Embeddings , 2018, AAAI.

[24]  Dipankar Das,et al.  A Practical Guide to Sentiment Analysis , 2017 .

[25]  Xiaoyan Zhu,et al.  Sentiment Analysis by Capsules , 2018, WWW.

[26]  Yang Liu,et al.  A method for multi-class sentiment classification based on an improved one-vs-one (OVO) strategy and the support vector machine (SVM) algorithm , 2017, Inf. Sci..

[27]  Carlo Strapparava,et al.  WordNet Affect: an Affective Extension of WordNet , 2004, LREC.

[28]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[29]  Zhiyuan Liu,et al.  Neural Sentiment Classification with User and Product Attention , 2016, EMNLP.

[30]  Regina Barzilay,et al.  Molding CNNs for text: non-linear, non-consecutive convolutions , 2015, EMNLP.

[31]  Phil Blunsom,et al.  A Convolutional Neural Network for Modelling Sentences , 2014, ACL.

[32]  Li Zhao,et al.  Attention-based LSTM for Aspect-level Sentiment Classification , 2016, EMNLP.

[33]  Jeffrey Pennington,et al.  Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions , 2011, EMNLP.

[34]  Steven Skiena,et al.  Building Sentiment Lexicons for All Major Languages , 2014, ACL.

[35]  Yi Liu,et al.  A Multi-sentiment-resource Enhanced Attention Network for Sentiment Classification , 2018, ACL.

[36]  Min Yang,et al.  Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications , 2019, ACL.

[37]  Kathleen M. Carley,et al.  Aspect Level Sentiment Classification with Attention-over-Attention Neural Networks , 2018, SBP-BRiMS.

[38]  Mauro Dragoni,et al.  A Fuzzy System for Concept-Level Sentiment Analysis , 2014, SemWebEval@ESWC.

[39]  Delip Rao,et al.  Semi-Supervised Polarity Lexicon Induction , 2009, EACL.

[40]  Erik Cambria,et al.  Sentiment Data Flow Analysis by Means of Dynamic Linguistic Patterns , 2015, IEEE Computational Intelligence Magazine.

[41]  Yue Zhang,et al.  Context-Sensitive Lexicon Features for Neural Sentiment Analysis , 2016, EMNLP.

[42]  Qiao Liu,et al.  Content Attention Model for Aspect Based Sentiment Analysis , 2018, WWW.

[43]  Xiaoyan Zhu,et al.  Linguistically Regularized LSTM for Sentiment Classification , 2016, ACL.

[44]  Qiang Ye,et al.  Sentiment classification of online reviews to travel destinations by supervised machine learning approaches , 2009, Expert Syst. Appl..

[45]  Erik Cambria,et al.  Word Polarity Disambiguation Using Bayesian Model and Opinion-Level Features , 2014, Cognitive Computation.

[46]  Pushpak Bhattacharyya,et al.  C-Feel-It: A Sentiment Analyzer for Micro-blogs , 2011, ACL.

[47]  Marie-Francine Moens,et al.  A machine learning approach to sentiment analysis in multilingual Web texts , 2009, Information Retrieval.

[48]  Sameena Shah,et al.  Learning Stock Market Sentiment Lexicon and Sentiment-Oriented Word Vector from StockTwits , 2017, CoNLL.

[49]  Liu Quan,et al.  Aspect-Based Sentiment Analysis Based on Multi-Attention CNN , 2017 .

[50]  Ting Liu,et al.  Learning Semantic Representations of Users and Products for Document Level Sentiment Classification , 2015, ACL.

[51]  Yoshua Bengio,et al.  Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach , 2011, ICML.

[52]  Erik Cambria,et al.  Jumping NLP Curves: A Review of Natural Language Processing Research [Review Article] , 2014, IEEE Computational Intelligence Magazine.

[53]  Claire Cardie,et al.  Annotating Expressions of Opinions and Emotions in Language , 2005, Lang. Resour. Evaluation.

[54]  Hongyu Guo,et al.  Long Short-Term Memory Over Recursive Structures , 2015, ICML.

[55]  Quoc V. Le,et al.  Distributed Representations of Sentences and Documents , 2014, ICML.

[56]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[57]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[58]  Erik Cambria,et al.  SenticNet 3: A Common and Common-Sense Knowledge Base for Cognition-Driven Sentiment Analysis , 2014, AAAI.

[59]  Erik Cambria,et al.  Targeted Aspect-Based Sentiment Analysis via Embedding Commonsense Knowledge into an Attentive LSTM , 2018, AAAI.

[60]  Gang Liu,et al.  Bidirectional LSTM with attention mechanism and convolutional layer for text classification , 2019, Neurocomputing.

[61]  Xiaojun Wan,et al.  Attention-based LSTM Network for Cross-Lingual Sentiment Classification , 2016, EMNLP.

[62]  Erik Cambria,et al.  Affective Computing and Sentiment Analysis , 2016, IEEE Intelligent Systems.

[63]  Christopher D. Manning,et al.  Baselines and Bigrams: Simple, Good Sentiment and Topic Classification , 2012, ACL.

[64]  Yuichi Nakamura,et al.  Approximation of dynamical systems by continuous time recurrent neural networks , 1993, Neural Networks.

[65]  Erik Cambria,et al.  A review of affective computing: From unimodal analysis to multimodal fusion , 2017, Inf. Fusion.

[66]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[67]  David E. Losada,et al.  An empirical study of sentence features for subjectivity and polarity classification , 2014, Inf. Sci..