Exploring Different Dimensions of Attention for Uncertainty Detection

Neural networks with attention have proven effective for many natural language processing tasks. In this paper, we develop attention mechanisms for uncertainty detection. In particular, we generalize standardly used attention mechanisms by introducing external attention and sequence-preserving attention. These novel architectures differ from standard approaches in that they use external resources to compute attention weights and preserve sequence information. We compare them to other configurations along different dimensions of attention. Our novel architectures set the new state of the art on a Wikipedia benchmark dataset and perform similar to the state-of-the-art model on a biomedical benchmark which uses a large set of linguistic features.

[1]  Qun Liu,et al.  Encoding Source Language with Convolutional Neural Network for Machine Translation , 2015, ACL.

[2]  Jason Weston,et al.  A Neural Attention Model for Abstractive Sentence Summarization , 2015, EMNLP.

[3]  Wei Gao,et al.  Detecting Semantic Uncertainty by Learning Hedge Cues in Sentences Using an HMM , 2014, SMIR@SIGIR.

[4]  Peng Zhou,et al.  Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling , 2016, COLING.

[5]  James Pustejovsky,et al.  FactBank: a corpus annotated with event factuality , 2009, Lang. Resour. Evaluation.

[6]  Iryna Gurevych,et al.  Cross-Genre and Cross-Domain Detection of Semantic Uncertainty , 2012, CL.

[7]  Maite Taboada,et al.  A machine‐learning approach to negation and speculation detection for sentiment analysis , 2016, J. Assoc. Inf. Sci. Technol..

[8]  Mihai Surdeanu,et al.  The Stanford CoreNLP Natural Language Processing Toolkit , 2014, ACL.

[9]  Robert Stalnaker,et al.  Presuppositions of Compound Sentences , 2008 .

[10]  Yuxin Peng,et al.  The application of two-level attention models in deep convolutional neural network for fine-grained image classification , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Bowen Zhou,et al.  Classifying Relations by Ranking with Convolutional Neural Networks , 2015, ACL.

[12]  Maria Georgescul,et al.  A Hedgehop over a Max-Margin Framework Using Hedge Cues , 2010, CoNLL Shared Task.

[13]  Xiaodong He,et al.  Character-Level Question Answering with Attention , 2016, EMNLP.

[14]  Phil Blunsom,et al.  Reasoning about Entailment with Neural Attention , 2015, ICLR.

[15]  Christopher Potts,et al.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , 2013, EMNLP.

[16]  Long Chen,et al.  Combining Feature-Based and Instance-Based Transfer Learning Approaches for Cross-Domain Hedge Detection with Multiple Sources , 2015, SMP.

[17]  János Csirik,et al.  The CoNLL-2010 Shared Task: Learning to Detect Hedges and their Scope in Natural Language Text , 2010, CoNLL Shared Task.

[18]  Jason Weston,et al.  Natural Language Processing (Almost) from Scratch , 2011, J. Mach. Learn. Res..

[19]  Charles A. Sutton,et al.  A Convolutional Attention Network for Extreme Summarization of Source Code , 2016, ICML.

[20]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[21]  Chris Callison-Burch,et al.  Modality and Negation in SIMT Use of Modality and Negation in Semantically-Informed Syntactic MT , 2012, CL.

[22]  James Pustejovsky,et al.  Are You Sure That This Happened? Assessing the Factuality Degree of Events in Text , 2012, CL.

[23]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[24]  Zhiyuan Liu,et al.  Relation Classification via Multi-Level Attention CNNs , 2016, ACL.

[25]  Yoshua Bengio,et al.  A Neural Probabilistic Language Model , 2003, J. Mach. Learn. Res..

[26]  Bowen Zhou,et al.  ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs , 2015, TACL.

[27]  Christopher Potts,et al.  Did It Happen? The Pragmatic Complexity of Veridicality Assessment , 2012, CL.

[28]  James Pustejovsky,et al.  Annotating, Extracting and Reasoning About Time and Events , 2005, Annotating, Extracting and Reasoning about Time and Events.

[29]  P CruzNoa,et al.  A machine-learning approach to negation and speculation detection for sentiment analysis , 2016 .

[30]  Dong Wang,et al.  Relation Classification via Recurrent Neural Network , 2015, ArXiv.

[31]  Diyi Yang,et al.  Hierarchical Attention Networks for Document Classification , 2016, NAACL.

[32]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[33]  Wenpeng Yin,et al.  Multichannel Variable-Size Convolution for Sentence Classification , 2015, CoNLL.

[34]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[35]  János Csirik,et al.  The BioScope corpus: biomedical texts annotated for uncertainty, negation and their scopes , 2008, BMC Bioinformatics.

[36]  Phil Blunsom,et al.  A Convolutional Neural Network for Modelling Sentences , 2014, ACL.

[37]  Michael Strube,et al.  Finding Hedges by Chasing Weasels: Hedge Detection Using Wikipedia Tags and Shallow Linguistic Features , 2009, ACL.

[38]  Kam-Fai Wong,et al.  Towards Neural Network-based Reasoning , 2015, ArXiv.

[39]  Wei Xu,et al.  ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering , 2015, ArXiv.

[40]  Phil Blunsom,et al.  Teaching Machines to Read and Comprehend , 2015, NIPS.

[41]  Jürgen Schmidhuber,et al.  Deep Networks with Internal Selective Attention through Feedback Connections , 2014, NIPS.

[42]  Wei Gao,et al.  An Empirical Study on Uncertainty Identification in Social Media Context , 2013, ACL.

[43]  Veronika Vincze,et al.  Uncertainty Detection in Hungarian Texts , 2014, COLING.

[44]  Lukás Burget,et al.  Recurrent neural network based language model , 2010, INTERSPEECH.

[45]  Vincze Veronika,et al.  Uncertainty Detection in Natural Language Texts , 2015 .

[46]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[47]  Stephan Oepen,et al.  Speculation and Negation: Rules, Rankers, and the Role of Syntax , 2012, CL.

[48]  Xiaolong Wang,et al.  A Cascade Method for Detecting Hedges and their Scope in Natural Language Text , 2010, CoNLL Shared Task.

[49]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[50]  Jun Zhao,et al.  Relation Classification via Convolutional Deep Neural Network , 2014, COLING.