TWEAC: Transformer with Extendable QA Agent Classifiers

Question answering systems should help users to access knowledge on a broad range of topics and to answer a wide array of different questions. Most systems fall short of this expectation as they are only specialized in one particular setting, e.g., answering factual questions with Wikipedia data. To overcome this limitation, we propose composing multiple QA agents within a meta-QA system. We argue that there exist a wide range of specialized QA agents in literature. Thus, we address the central research question of how to effectively and efficiently identify suitable QA agents for any given question. We study both supervised and unsupervised approaches to address this challenge, showing that TWEAC - Transformer with Extendable Agent Classifiers - achieves the best performance overall with 94% accuracy. We provide extensive insights on the scalability of TWEAC, demonstrating that it scales robustly to over 100 QA agents with each providing just 1000 examples of questions they can answer.

[1]  Alberto Barrón-Cedeño,et al.  A Flexible, Efficient and Accurate Framework for Community Question Answering Pipelines , 2018, ACL.

[2]  Bing Liu,et al.  Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling , 2016, INTERSPEECH.

[3]  Ray Kurzweil,et al.  Multilingual Universal Sentence Encoder for Semantic Retrieval , 2019, ACL.

[4]  Gabriel Stanovsky,et al.  DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs , 2019, NAACL.

[5]  Kevin Gimpel,et al.  Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units , 2016, ArXiv.

[6]  Varvara Logacheva,et al.  DeepPavlov: Open-Source Library for Dialogue Systems , 2018, ACL.

[7]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[8]  Ruslan Salakhutdinov,et al.  Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text , 2018, EMNLP.

[9]  Jie Tang,et al.  A novel classification method for paper-reviewer recommendation , 2018, Scientometrics.

[10]  Zhen Duan,et al.  Reviewer assignment based on sentence pair modeling , 2019, Neurocomputing.

[11]  Preslav Nakov,et al.  SemEval-2017 Task 3: Community Question Answering , 2017, *SEMEVAL.

[12]  W. Bruce Croft,et al.  WikiPassageQA: A Benchmark Collection for Research on Non-factoid Answer Passage Retrieval , 2018, SIGIR.

[13]  Iryna Gurevych,et al.  End-to-End Non-Factoid Question Answering with an Interactive Visualization of Neural Attention Weights , 2017, ACL.

[14]  Iryna Gurevych,et al.  Neural Duplicate Question Detection without Labeled Training Data , 2019, EMNLP.

[15]  Yoshua Bengio,et al.  HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering , 2018, EMNLP.

[16]  Jason Weston,et al.  ParlAI: A Dialog Research Software Platform , 2017, EMNLP.

[17]  Axel-Cyrille Ngonga Ngomo,et al.  7th Open Challenge on Question Answering over Linked Data (QALD-7) , 2017, SemWebEval@ESWC.

[18]  Iryna Gurevych,et al.  Interactive Instance-based Evaluation of Knowledge Base Question Answering , 2018, EMNLP.

[19]  Ming-Wei Chang,et al.  The Value of Semantic Parse Labeling for Knowledge Base Question Answering , 2016, ACL.

[20]  Iryna Gurevych,et al.  MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale , 2020, EMNLP.

[21]  Kevin Gimpel,et al.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations , 2019, ICLR.

[22]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[23]  Jonathan Berant,et al.  MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension , 2019, ACL.

[24]  Young-Bum Kim,et al.  Continuous Learning for Large-scale Personalized Domain Classification , 2019, NAACL.

[25]  Rashmi Gangadharaiah,et al.  Joint Multiple Intent Detection and Slot Labeling for Goal-Oriented Dialog , 2019, NAACL.

[26]  William W. Cohen,et al.  PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text , 2019, EMNLP.

[27]  Danqi Chen,et al.  of the Association for Computational Linguistics: , 2001 .

[28]  Yizhou Sun,et al.  Personalized Question Routing via Heterogeneous Network Embedding , 2019, AAAI.

[29]  Dietrich Klakow,et al.  Linguistically Motivated Question Classification , 2015, NODALIDA.

[30]  Noah Constant,et al.  MultiReQA: A Cross-Domain Evaluation forRetrieval Question Answering Models , 2020, ADAPTNLP.

[31]  Boualem Benatallah,et al.  Expert2Vec: Experts Representation in Community Question Answering for Question Routing , 2019, CAiSE.

[32]  Wen-mei W. Hwu,et al.  PaRe: A Paper-Reviewer Matching Approach Using a Common Topic Space , 2019, EMNLP.

[33]  Iryna Gurevych,et al.  A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking , 2019, CoNLL.

[34]  Yangming Li,et al.  A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding , 2019, EMNLP.

[35]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[36]  Dongyan Zhao,et al.  Hybrid Question Answering over Knowledge Base and Free Text , 2016, COLING.

[37]  Ming-Wei Chang,et al.  BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions , 2019, NAACL.

[38]  Hugo Zaragoza,et al.  The Probabilistic Relevance Framework: BM25 and Beyond , 2009, Found. Trends Inf. Retr..

[39]  Siu Cheung Hui,et al.  Learning to Rank Question Answer Pairs with Holographic Dual LSTM Architecture , 2017, SIGIR.

[40]  Jimmy J. Lin,et al.  End-to-End Open-Domain Question Answering with BERTserini , 2019, NAACL.

[41]  Jason Weston,et al.  Reading Wikipedia to Answer Open-Domain Questions , 2017, ACL.

[42]  Jens Lehmann,et al.  Why Reinvent the Wheel: Let's Build Question Answering Systems Together , 2018, WWW.

[43]  Seung-won Hwang,et al.  KBQA: Learning Question Answering over QA Corpora and Knowledge Bases , 2019, Proc. VLDB Endow..

[44]  Iryna Gurevych,et al.  COALA: A Neural Coverage-Based Approach for Long Answer Selection with Small Data , 2019, AAAI.

[45]  Suresh Manandhar,et al.  Dependency Based Embeddings for Sentence Classification Tasks , 2016, NAACL.