Towards Understanding and Answering Multi-Sentence Recommendation Questions on Tourism

We introduce the first system towards the novel task of answering complex multisentence recommendation questions in the tourism domain. Our solution uses a pipeline of two modules: question understanding and answering. For question understanding, we define an SQL-like query language that captures the semantic intent of a question; it supports operators like subset, negation, preference and similarity, which are often found in recommendation questions. We train and compare traditional CRFs as well as bidirectional LSTM-based models for converting a question to its semantic representation. We extend these models to a semisupervised setting with partially labeled sequences gathered through crowdsourcing. We find that our best model performs semi-supervised training of BiDiLSTM+CRF with hand-designed features and CCM(Chang et al., 2007) constraints. Finally, in an end to end QA system, our answering component converts our question representation into queries fired on underlying knowledge sources. Our experiments on two different answer corpora demonstrate that our system can significantly outperform baselines with up to 20 pt higher accuracy and 17 pt higher recall.

[1]  A R Rodolfo Pazos,et al.  Natural Language Interfaces to Databases: An Analysis of the State of the Art , 2013, HIS 2013.

[2]  Jason Weston,et al.  Reading Wikipedia to Answer Open-Domain Questions , 2017, ACL.

[3]  Andrew Chou,et al.  Semantic Parsing on Freebase from Question-Answer Pairs , 2013, EMNLP.

[4]  Angeliki Metallinou,et al.  Simple Question Answering with Subgraph Ranking and Joint-Scoring , 2019, NAACL.

[5]  Jun Zhao,et al.  IJCNLP-2017 Task 5: Multi-choice Question Answering in Examinations , 2017, IJCNLP.

[6]  Ming-Wei Chang,et al.  Guiding Semi-Supervision with Constraint-Driven Learning , 2007, ACL.

[7]  Oren Etzioni,et al.  Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions , 2016, AAAI.

[8]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[9]  Sebastian Riedel,et al.  Constructing Datasets for Multi-hop Reading Comprehension Across Documents , 2017, TACL.

[10]  Omer Levy,et al.  Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling , 2018, ACL.

[11]  R'emi Louf,et al.  HuggingFace's Transformers: State-of-the-art Natural Language Processing , 2019, ArXiv.

[12]  Rajarshi Das,et al.  Question Answering on Knowledge Bases and Text using Universal Schema and Memory Networks , 2017, ACL.

[13]  Luke S. Zettlemoyer,et al.  Deep Semantic Role Labeling: What Works and What’s Next , 2017, ACL.

[14]  Richard Socher,et al.  A Neural Network for Factoid Question Answering over Paragraphs , 2014, EMNLP.

[15]  Jens Lehmann,et al.  LC-QuAD: A Corpus for Complex Question Answering over Knowledge Graphs , 2017, SEMWEB.

[16]  Ming Zhou,et al.  Answering Questions with Complex Semantic Constraints on Open Knowledge Bases , 2015, CIKM.

[17]  Alexandra Vtyurina.  Complex questions : Let me Google it for you , 2016 .

[18]  Di Wang,et al.  CMU OAQA at TREC 2016 LiveQA: An Attentional Neural Encoder-Decoder Approach for Answer Ranking , 2016, TREC.

[19]  Luke S. Zettlemoyer,et al.  AllenNLP: A Deep Semantic Natural Language Processing Platform , 2018, ArXiv.

[20]  Xuanjing Huang,et al.  Convolutional Neural Tensor Network Architecture for Community-Based Question Answering , 2015, IJCAI.

[21]  Gerhard Weikum,et al.  Automated Template Generation for Question Answering over Knowledge Graphs , 2017, WWW.

[22]  David Vandyke,et al.  Counter-fitting Word Vectors to Linguistic Constraints , 2016, NAACL.

[23]  Oren Etzioni,et al.  Open question answering over curated and extracted knowledge bases , 2014, KDD.

[24]  Jason Weston,et al.  Question Answering with Subgraph Embeddings , 2014, EMNLP.

[25]  Lei Zou,et al.  Question Answering Over Knowledge Graphs: Question Understanding Via Template Decomposition , 2018, Proc. VLDB Endow..

[26]  Yonatan Belinkov,et al.  Neural Attention for Learning to Rank Questions in Community Question Answering , 2016, COLING.

[27]  Jason Weston,et al.  Open Question Answering with Weakly Supervised Embedding Models , 2014, ECML/PKDD.

[28]  Parag Singla,et al.  Entity-balanced Gaussian pLSA for Automated Comparison , 2016, NAACL.

[29]  Nan Jiang,et al.  Word Embedding Based Correlation Model for Question/Answer Matching , 2015, AAAI.

[30]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[31]  Mária Bieliková,et al.  A Comprehensive Survey and Classification of Approaches for Community Question Answering , 2016, ACM Trans. Web.

[32]  Bowen Zhou,et al.  LSTM-based Deep Learning Models for non-factoid answer selection , 2015, ArXiv.

[33]  Tom M. Mitchell,et al.  A Joint Sequential and Relational Model for Frame-Semantic Parsing , 2017, EMNLP.

[34]  Jens Lehmann,et al.  Neural Network-based Question Answering over Knowledge Graphs on Word and Character Level , 2017, WWW.

[35]  Xiao Zhang,et al.  Medical Exam Question Answering with Large-scale Reading Comprehension , 2018, AAAI.

[36]  Luke S. Zettlemoyer,et al.  Learning to map sentences to logical form , 2012, UAI 2012.

[37]  Martha Palmer,et al.  Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017 - System Demonstrations , 2017, EMNLP.

[38]  Eric P. Xing,et al.  Science Question Answering using Instructional Materials , 2016, ACL.

[39]  Di Wang,et al.  CMU OAQA at TREC 2015 LiveQA: Discovering the Right Answer with Clues , 2015, TREC.

[40]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[41]  Gökhan Tür,et al.  Towards Zero-Shot Frame Semantic Parsing for Domain Scaling , 2017, INTERSPEECH.

[42]  Wei Xu,et al.  End-to-end learning of semantic role labeling using recurrent neural networks , 2015, ACL.

[43]  Wei Xu,et al.  Bidirectional LSTM-CRF Models for Sequence Tagging , 2015, ArXiv.

[44]  Zhoujun Li,et al.  Learning Distributed Representations of Data in Community Question Answering for Question Retrieval , 2016, WSDM.

[45]  Mark Steedman,et al.  Transforming Dependency Structures to Logical Forms for Semantic Parsing , 2016, TACL.

[46]  Anagha Kulkarni,et al.  Automated Question Answering System for Community-Based Questions , 2018, AAAI.

[47]  Jonathan Berant,et al.  Semantic Parsing via Paraphrasing , 2014, ACL.

[48]  Eunsol Choi,et al.  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension , 2017, ACL.

[49]  Gabriel Stanovsky,et al.  DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs , 2019, NAACL.

[50]  Ming-Wei Chang,et al.  Open Domain Question Answering via Semantic Enrichment , 2015, WWW.

[51]  Oren Etzioni,et al.  Paraphrase-Driven Learning for Open Question Answering , 2013, ACL.

[52]  Kyunghyun Cho,et al.  Task-Oriented Query Reformulation with Reinforcement Learning , 2017, EMNLP.

[53]  Dongyan Zhao,et al.  Question Answering on Freebase via Relation Extraction and Textual Evidence , 2016, ACL.

[54]  Dan Klein,et al.  Learning Dependency-Based Compositional Semantics , 2011, CL.

[55]  Jennifer Foster,et al.  This is how we do it: Answer Reranking for Open-domain How Questions with Paragraph Vectors and Minimal Feature Engineering , 2016, NAACL.

[56]  Philip Bachman,et al.  NewsQA: A Machine Comprehension Dataset , 2016, Rep4NLP@ACL.

[57]  Elena Paslaru Bontas Simperl,et al.  Using Semantics to Search Answers for Unanswered Questions in Q&A Forums , 2016, WWW.

[58]  Oren Etzioni,et al.  Solving Geometry Problems: Combining Text and Diagram Interpretation , 2015, EMNLP.

[59]  Yoshua Bengio,et al.  HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering , 2018, EMNLP.

[60]  Eunsol Choi,et al.  Scaling Semantic Parsers with On-the-Fly Ontology Matching , 2013, EMNLP.

[61]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[62]  Umar Farooq Minhas,et al.  ATHENA: An Ontology-Driven System for Natural Language Querying over Relational Data Stores , 2016, Proc. VLDB Endow..

[63]  Joemon M. Jose,et al.  A Semantic Graph based Topic Model for Question Retrieval in Community Question Answering , 2016, WSDM.

[64]  Jason Weston,et al.  Large-scale Simple Question Answering with Memory Networks , 2015, ArXiv.

[65]  Luke S. Zettlemoyer,et al.  Question-Answer Driven Semantic Role Labeling: Using Natural Language to Annotate Natural Language , 2015, EMNLP.

[66]  Daniel Gildea,et al.  The Proposition Bank: An Annotated Corpus of Semantic Roles , 2005, CL.

[67]  Percy Liang,et al.  Know What You Don’t Know: Unanswerable Questions for SQuAD , 2018, ACL.

[68]  P. Robinson,et al.  Efficient Estimation of the , 2007 .

[69]  Peter Clark,et al.  Answering Complex Questions Using Open Information Extraction , 2017, ACL.