SubjQA: A Dataset for Subjectivity and Review Comprehension

Subjectivity is the expression of internal opinions or beliefs which cannot be objectively observed or verified, and has been shown to be important for sentiment analysis and word-sense disambiguation. Furthermore, subjectivity is an important aspect of user-generated data. In spite of this, subjectivity has not been investigated in contexts where such data is widespread, such as in question answering (QA). We therefore investigate the relationship between subjectivity and QA, while developing a new dataset. We compare and contrast with analyses from previous work, and verify that findings regarding subjectivity still hold when using recently developed NLP architectures. We find that subjectivity is also an important feature in the case of QA, albeit with more intricate interactions between subjectivity and QA performance. For instance, a subjective question may or may not be associated with a subjective answer. We release an English QA dataset (SubjQA) based on customer reviews, containing subjectivity annotations for questions and answer spans across 6 distinct domains.

[1]  Percy Liang,et al.  Know What You Don’t Know: Unanswerable Questions for SQuAD , 2018, ACL.

[2]  Isabelle Augenstein,et al.  Unsupervised Evaluation for Question Answering with Transformers , 2020, BLACKBOXNLP.

[3]  Shiliang Sun,et al.  A review of natural language processing techniques for opinion mining systems , 2017, Inf. Fusion.

[4]  Joachim Bingel,et al.  Identifying beneficial task relations for multi-task learning in deep neural networks , 2017, EACL.

[5]  Erik Cambria,et al.  Aspect extraction for opinion mining with a deep convolutional neural network , 2016, Knowl. Based Syst..

[6]  Bing Liu,et al.  Mining and summarizing customer reviews , 2004, KDD.

[7]  Philip S. Yu,et al.  BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis , 2019, NAACL.

[8]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[9]  Rada Mihalcea,et al.  Word Sense and Subjectivity , 2006, ACL.

[10]  Chen Chen,et al.  Sampo: Unsupervised Knowledge Base Construction for Opinions and Implications , 2020, AKBC.

[11]  Julian J. McAuley,et al.  Addressing Complex and Subjective Product-Related Queries with Customer Reviews , 2015, WWW.

[12]  Julien Perez,et al.  ReviewQA: a relational aspect-based opinion reading dataset , 2018, ArXiv.

[13]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[14]  Eunsol Choi,et al.  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension , 2017, ACL.

[15]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[16]  Miao Fan,et al.  Reading Customer Reviews to Answer Product-related Questions , 2019, SDM.

[17]  Jinfeng Li,et al.  Subjective Databases , 2019, Proc. VLDB Endow..

[18]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[19]  Janyce Wiebe,et al.  Development and Use of a Gold-Standard Data Set for Subjectivity Classifications , 1999, ACL.

[20]  Isabelle Augenstein,et al.  Multi-Task Learning of Pairwise Sequence Classification Tasks over Disparate Label Spaces , 2018, NAACL.

[21]  Mansi Gupta,et al.  AmazonQA: A Review-Based Question Answering Task , 2019, IJCAI.

[22]  Andrew McCallum,et al.  Relation Extraction with Matrix Factorization and Universal Schemas , 2013, NAACL.

[23]  Samaneh Moghaddam,et al.  Aspect-based opinion mining in online reviews , 2013 .

[24]  Jan Svartvik,et al.  A __ comprehensive grammar of the English language , 1988 .

[25]  Philip S. Yu,et al.  Review Conversational Reading Comprehension , 2019, ArXiv.

[26]  Sebastian Riedel,et al.  MLQA: Evaluating Cross-lingual Extractive Question Answering , 2019, ACL.

[27]  Yue Lu,et al.  Latent aspect rating analysis on review text data: a rating regression approach , 2010, KDD.

[28]  Dirk Weissenborn,et al.  FastQA: A Simple and Efficient Neural Architecture for Question Answering , 2017, ArXiv.

[29]  Tat-Seng Chua,et al.  Answering Opinion Questions on Products by Exploiting Hierarchical Organization of Consumer Reviews , 2012, EMNLP.

[30]  Philip Bachman,et al.  NewsQA: A Machine Comprehension Dataset , 2016, Rep4NLP@ACL.

[31]  K. Durkin,et al.  Polysemy and the subjective lexicon: Semantic relatedness and the salience of intraword senses , 1989, Journal of psycholinguistic research.

[32]  Joachim Bingel,et al.  Latent Multi-Task Architecture Learning , 2017, AAAI.

[33]  Johannes Bjerva,et al.  Will my auxiliary tagging task help? Estimating Auxiliary Tasks Effectivity in Multi-Task Learning , 2017, NODALIDA.

[34]  Christian Hansen,et al.  MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims , 2019, EMNLP.

[35]  Rada Mihalcea,et al.  Multilingual Sentiment and Subjectivity Analysis , 2011 .

[36]  Johannes Bjerva,et al.  One Model to Rule them all: Multitask and Multilingual Modelling for Lexical Analysis , 2017, ArXiv.

[37]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[38]  Danqi Chen,et al.  CoQA: A Conversational Question Answering Challenge , 2018, TACL.

[39]  Isabelle Augenstein,et al.  X-WikiRE: A Large, Multilingual Resource for Relation Extraction as Machine Comprehension , 2019, EMNLP.

[40]  Isabelle Augenstein,et al.  Jack the Reader – A Machine Reading Framework , 2018, ACL.

[41]  Isabelle Augenstein,et al.  Parameter sharing between dependency parsers for related languages , 2018, EMNLP.

[42]  Claire Cardie,et al.  Annotating Expressions of Opinions and Emotions in Language , 2005, Lang. Resour. Evaluation.

[43]  Maite Taboada,et al.  Evaluative Language Beyond Bags of Words: Linguistic Insights and Computational Applications , 2017, CL.

[44]  Bo Pang,et al.  A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts , 2004, ACL.

[45]  Sasha Blair-Goldensohn,et al.  Building a Sentiment Summarizer for Local Service Reviews , 2008 .

[46]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[47]  Rada Mihalcea,et al.  Multilingual Subjectivity: Are More Languages Better? , 2010, COLING.

[48]  Ari Kobren,et al.  Constructing High Precision Knowledge Bases with Subjective and Factual Attributes , 2019, KDD.

[49]  Janyce Wiebe,et al.  Subjectivity Word Sense Disambiguation , 2009, EMNLP.

[50]  Ali Farhadi,et al.  Bidirectional Attention Flow for Machine Comprehension , 2016, ICLR.