Less Is More: Attention Supervision with Counterfactuals for Text Classification

We aim to leverage human and machine intelligence together for attention supervision. Specifically, we show that human annotation cost can be kept reasonably low, while its quality can be enhanced by machine self-supervision. Specifically, for this goal, we explore the advantage of counterfactual reasoning, over associative reasoning typically used in attention supervision. Our empirical results show that this machine-augmented human attention supervision is more effective than existing methods requiring a higher annotation cost, in text classification tasks, including sentiment analysis and news categorization.

[1]  Anna Rumshisky,et al.  A Primer in BERTology: What We Know About How BERT Works , 2020, Transactions of the Association for Computational Linguistics.

[2]  Zhiyuan Liu,et al.  Neural Sentiment Classification with User and Product Attention , 2016, EMNLP.

[3]  Yuval Pinter,et al.  Attention is not not Explanation , 2019, EMNLP.

[4]  Diyi Yang,et al.  Hierarchical Attention Networks for Document Classification , 2016, NAACL.

[5]  Quan Z. Sheng,et al.  Generating Textual Adversarial Examples for Deep Learning Models: A Survey , 2019, ArXiv.

[6]  Noah A. Smith,et al.  Is Attention Interpretable? , 2019, ACL.

[7]  Joachim Bingel,et al.  Sequence Classification with Human Attention , 2018, CoNLL.

[8]  Andrea Esuli,et al.  SENTIWORDNET: A Publicly Available Lexical Resource for Opinion Mining , 2006, LREC.

[9]  Mani B. Srivastava,et al.  Generating Natural Language Adversarial Examples , 2018, EMNLP.

[10]  Jie Yang,et al.  YEDDA: A Lightweight Collaborative Text Span Annotation Tool , 2017, ACL.

[11]  Hanqiu Sun,et al.  Human vs Machine Attention in Neural Networks: A Comparative Study , 2019, ArXiv.

[12]  Xiangnan Kong,et al.  Human Attention Maps for Text Classification: Do Humans and Neural Networks Focus on the Same Words? , 2020, ACL.

[13]  Christopher Potts,et al.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , 2013, EMNLP.

[14]  Zachary Chase Lipton,et al.  Born Again Neural Networks , 2018, ICML.

[15]  Omer Levy,et al.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding , 2018, BlackboxNLP@EMNLP.

[16]  Regina Barzilay,et al.  Deriving Machine Attention from Human Rationales , 2018, EMNLP.

[17]  Alexandros Potamianos,et al.  Attention-based Conditioning Methods for External Knowledge Integration , 2019, ACL.

[18]  Chandra Bhagavatula,et al.  Semi-supervised sequence tagging with bidirectional language models , 2017, ACL.

[19]  Andrew McCallum,et al.  Linguistically-Informed Self-Attention for Semantic Role Labeling , 2018, EMNLP.

[20]  Seungtaek Choi,et al.  Counterfactual Attention Supervision , 2019, 2019 IEEE International Conference on Data Mining (ICDM).

[21]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[22]  Byron C. Wallace,et al.  Attention is not Explanation , 2019, NAACL.

[23]  Yue Zhao,et al.  Document Embedding Enhanced Event Detection with Hierarchical and Supervised Attention , 2018, ACL.

[24]  Graham Neubig,et al.  Learning to Deceive with Attention-Based Explanations , 2020, ACL.

[25]  Balaraman Ravindran,et al.  Towards Transparent and Explainable Attention Models , 2020, ACL.

[26]  Jiebo Luo,et al.  Progressive Self-Supervised Attention Learning for Aspect-Level Sentiment Analysis , 2019, ACL.

[27]  Christopher Potts,et al.  Learning Word Vectors for Sentiment Analysis , 2011, ACL.

[28]  Xuanjing Huang,et al.  A Lexicon-Based Supervised Attention Model for Neural Sentiment Analysis , 2018, COLING.

[29]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[30]  Martin Tutek,et al.  Staying True to Your Word: (How) Can Attention Become Explanation? , 2020, RepL4NLP@ACL.

[31]  Young-Bum Kim,et al.  Supervised Domain Enablement Attention for Personalized Domain Classification , 2018, EMNLP.

[32]  Uri Shalit,et al.  Learning Representations for Counterfactual Inference , 2016, ICML.

[33]  Thomas Lukasiewicz,et al.  e-SNLI: Natural Language Inference with Natural Language Explanations , 2018, NeurIPS.

[34]  Chenxi Liu,et al.  Attention Correctness in Neural Image Captioning , 2016, AAAI.

[35]  Licheng Yu,et al.  Hierarchically-Attentive RNN for Album Summarization and Storytelling , 2017, EMNLP.

[36]  Ye Zhang,et al.  Rationale-Augmented Convolutional Neural Networks for Text Classification , 2016, EMNLP.

[37]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[38]  Yang Liu,et al.  On Identifiability in Transformers , 2020, ICLR.

[39]  Kathleen McKeown,et al.  Fine-grained Sentiment Analysis with Faithful Attention , 2019, ArXiv.

[40]  Le Song,et al.  L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data , 2018, ICLR.

[41]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[42]  Han-Yu Chen,et al.  Adversarial Attack on Sentiment Classification , 2019, WNLP@ACL.