Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing

Interactive semantic parsing based on natural language (NL) feedback, where users provide feedback to correct the parser mistakes, has emerged as a more practical scenario than the traditional one-shot semantic parsing. However, prior work has heavily relied on human-annotated feedback data to train the interactive semantic parser, which is prohibitively expensive and not scalable. In this work, we propose a new task of simulating NL feedback for interactive semantic parsing. We accompany the task with a novel feedback evaluator. The evaluator is specifically designed to assess the quality of the simulated feedback, based on which we decide the best feedback simulator from our proposed variants. On a text-to-SQL dataset, we show that our feedback simulator can generate high-quality NL feedback to boost the error correction ability of a specific parser. In low-data settings, our feedback simulator can help achieve comparable error correction performance as trained using the costly, full set of human annotations.

[1]  Xinyun Chen,et al.  Teaching Large Language Models to Self-Debug , 2023, ArXiv.

[2]  Bodhisattwa Prasad Majumder,et al.  Self-Refine: Iterative Refinement with Self-Feedback , 2023, 2303.17651.

[3]  P. Baldi,et al.  Language Models can Solve Computer Tasks , 2023, ArXiv.

[4]  Xing Han Lu,et al.  Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment , 2022, FINDINGS.

[5]  Ryan J. Lowe,et al.  Training language models to follow instructions with human feedback , 2022, NeurIPS.

[6]  Michael White,et al.  Towards Transparent Interactive Semantic Parsing via Step-by-Step Correction , 2021, FINDINGS.

[7]  Ryan Cotterell,et al.  A Plug-and-Play Method for Controlled Text Generation , 2021, EMNLP.

[8]  Eric P. Xing,et al.  Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation , 2021, EMNLP.

[9]  Weizhe Yuan,et al.  BARTScore: Evaluating Generated Text as Text Generation , 2021, NeurIPS.

[10]  Jacob Andreas,et al.  Compositional Generalization for Neural Semantic Parsing via Span-level Supervised Attention , 2021, NAACL.

[11]  Minsuk Chang,et al.  NeuralWOZ: Learning to Collect Task-Oriented Dialogue via Model-Based Simulation , 2021, ACL.

[12]  Keisuke Sakaguchi,et al.  Improving Neural Model Performance through Natural Language Feedback on Their Explanations , 2021, ArXiv.

[13]  Matthew Richardson,et al.  NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction , 2021, NAACL.

[14]  Brian M. Sadler,et al.  Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases , 2020, WWW.

[15]  Gaurav Pandey,et al.  Simulated Chats for Building Dialog Systems: Learning to Generate Conversations from Instructions , 2020, EMNLP.

[16]  Yuntao Li,et al.  “What Do You Mean by That?” - a Parser-Independent Interactive Approach for Enhancing Text-to-SQL , 2020, EMNLP.

[17]  Richard Socher,et al.  Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing , 2020, FINDINGS.

[18]  Diarmuid Ó Séaghdha,et al.  Conversational Semantic Parsing for Dialog State Tracking , 2020, EMNLP.

[19]  Jacob Andreas,et al.  Task-Oriented Dialogue as Dataflow Synthesis , 2020, Transactions of the Association for Computational Linguistics.

[20]  Christopher D. Manning,et al.  Towards Ecologically Valid Research on Language User Interfaces , 2020, ArXiv.

[21]  Steven C. H. Hoi,et al.  Photon: A Robust Cross-Domain Text-to-SQL System , 2020, ACL.

[22]  Wen-tau Yih,et al.  An Imitation Game for Learning Semantic Parsers from User Interaction , 2020, EMNLP.

[23]  Ahmed Hassan Awadallah,et al.  Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback , 2020, ACL.

[24]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[25]  Kilian Q. Weinberger,et al.  BERTScore: Evaluating Text Generation with BERT , 2019, ICLR.

[26]  Wen-tau Yih,et al.  Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study , 2019, EMNLP.

[27]  Zhou Yu,et al.  How to Build User Simulators to Train RL-based Dialog Systems , 2019, EMNLP.

[28]  Tao Yu,et al.  Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions , 2019, EMNLP.

[29]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[30]  Stefan Riezler,et al.  Self-Regulated Interactive Sequence-to-Sequence Learning , 2019, ACL.

[31]  Jason Weston,et al.  Learning from Dialogue after Deployment: Feed Yourself, Chatbot! , 2019, ACL.

[32]  Brian M. Sadler,et al.  Interactive Semantic Parsing for If-Then Recipes via Hierarchical Reinforcement Learning , 2018, AAAI.

[33]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[34]  Tom M. Mitchell,et al.  Learning to Learn Semantic Parsers from Natural Language Supervision , 2019, EMNLP.

[35]  Sonal Gupta,et al.  Semantic Parsing for Task Oriented Dialog using Hierarchical Representations , 2018, EMNLP.

[36]  Xifeng Yan,et al.  DialSQL: Dialogue Based Structured Query Generation , 2018, ACL.

[37]  Raymond J. Mooney,et al.  Dialog for Language to Code , 2017, IJCNLP.

[38]  Alvin Cheung,et al.  Learning a Neural Semantic Parser from User Feedback , 2017, ACL.

[39]  Jianfeng Gao,et al.  A User Simulator for Task-Completion Dialogues , 2016, ArXiv.

[40]  Ming-Wei Chang,et al.  The Value of Semantic Parse Labeling for Knowledge Base Question Answering , 2016, ACL.

[41]  Christopher D. Manning,et al.  Learning Language Games through Interaction , 2016, ACL.

[42]  Mirella Lapata,et al.  Language to Logical Form with Neural Attention , 2016, ACL.

[43]  Mark Steedman,et al.  Large-scale Semantic Parsing without Question-Answer Pairs , 2014, TACL.

[44]  Alexander Yates,et al.  Semantic Parsing Freebase: Towards Open-domain Semantic Parsing , 2013, *SEMEVAL.

[45]  Ming-Wei Chang,et al.  Driving Semantic Parsing from the World’s Response , 2010, CoNLL.

[46]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[47]  David G. Novick,et al.  Natural-language interfaces , 2000, CHI Extended Abstracts.