NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction

We study semantic parsing in an interactive setting in which users correct errors with natural language feedback. We present NL-EDIT, a model for interpreting natural language feedback in the interaction context to generate a sequence of edits that can be applied to the initial parse to correct its errors. We show that NL-EDIT can boost the accuracy of existing text-to-SQL parsers by up to 20% with only one turn of correction. We analyze the limitations of the model and discuss directions for improvement and evaluation. The code and datasets used in this paper are publicly available at http://aka.ms/NLEdit.

[1]  Dzmitry Bahdanau,et al.  DuoRAT: Towards Simpler Text-to-SQL Models , 2020, NAACL.

[2]  Dong Ryeol Shin,et al.  RYANSQL: Recursively Applying Sketch-based Slot Fillings for Complex Text-to-SQL in Cross-Domain Databases , 2020, CL.

[3]  Yoav Artzi,et al.  Learning to Map Context-Dependent Sentences to Executable Formal Queries , 2018, NAACL.

[4]  Fei Li,et al.  Constructing an Interactive Natural Language Interface for Relational Databases , 2014, Proc. VLDB Endow..

[5]  Tao Yu,et al.  SParC: Cross-Domain Semantic Parsing in Context , 2019, ACL.

[6]  Alex Polozov,et al.  SCoRe: Pre-Training for Context Representation in Conversational Semantic Parsing , 2021, ICLR.

[7]  Monica S. Lam,et al.  AutoQA: From Databases to Q&A Semantic Parsers with Only Synthetic Training Data , 2020, EMNLP.

[8]  David H. D. Warren,et al.  An Efficient Easily Adaptable System for Interpreting Natural Language Queries , 1982, CL.

[9]  Yan Gao,et al.  Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation , 2019, ACL.

[10]  Xiaodong Liu,et al.  RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers , 2020, ACL.

[11]  Rahul Gupta,et al.  DeepFix: Fixing Common C Language Errors by Deep Learning , 2017, AAAI.

[12]  Luke S. Zettlemoyer,et al.  Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions , 2013, TACL.

[13]  Felix Stahlberg,et al.  Seq2Edits: Sequence Transduction Using Span-level Edit Operations , 2020, EMNLP.

[14]  Maya Cakmak,et al.  Power to the People: The Role of Humans in Interactive Machine Learning , 2014, AI Mag..

[15]  Michael Gamon,et al.  Building Natural Language Interfaces to Web APIs , 2017, CIKM.

[16]  Ryen W. White,et al.  Natural Language Interfaces with Fine-Grained User Interaction: A Case Study on Web APIs , 2018, SIGIR.

[17]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[18]  Richard Socher,et al.  Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning , 2018, ArXiv.

[19]  Ming-Wei Chang,et al.  Driving Semantic Parsing from the World’s Response , 2010, CoNLL.

[20]  Xifeng Yan,et al.  DialSQL: Dialogue Based Structured Query Generation , 2018, ACL.

[21]  Jacob Andreas,et al.  Task-Oriented Dialogue as Dataflow Synthesis , 2020, Transactions of the Association for Computational Linguistics.

[22]  Graham Neubig,et al.  TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data , 2020, ACL.

[23]  Alvin Cheung,et al.  Learning a Neural Semantic Parser from User Feedback , 2017, ACL.

[24]  Richard Shin,et al.  Encoding Database Schemas with Relation-Aware Self-Attention for Text-to-SQL Parsers , 2019, ArXiv.

[25]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[26]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[27]  Henry A. Kautz,et al.  Towards a theory of natural language interfaces to databases , 2003, IUI '03.

[28]  Aliaksei Severyn,et al.  Encode, Tag, Realize: High-Precision Text Editing , 2019, EMNLP.

[29]  Wen-tau Yih,et al.  Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study , 2019, EMNLP.

[30]  Junyi Jessy Li,et al.  Learning to Update Natural Language Comments Based on Code Changes , 2020, ACL.

[31]  Tao Yu,et al.  SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-Domain Text-to-SQL Task , 2018, EMNLP.

[32]  Ahmed Hassan Awadallah,et al.  Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback , 2020, ACL.

[33]  Mirella Lapata,et al.  Building a Neural Semantic Parser from a Domain Ontology , 2018, ArXiv.

[34]  Tao Yu,et al.  GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing , 2021, ICLR.

[35]  Tao Yu,et al.  Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task , 2018, EMNLP.

[36]  Jason Weston,et al.  Curriculum learning , 2009, ICML '09.

[37]  Been Kim,et al.  Towards A Rigorous Science of Interpretable Machine Learning , 2017, 1702.08608.

[38]  Luyao Chen,et al.  CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases , 2019, EMNLP.

[39]  Graham Neubig,et al.  Reranking for Neural Semantic Parsing , 2019, ACL.

[40]  Thomas Muller,et al.  TaPas: Weakly Supervised Table Parsing via Pre-training , 2020, ACL.

[41]  Jonathan Berant,et al.  Building a Semantic Parser Overnight , 2015, ACL.

[42]  Tao Yu,et al.  Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions , 2019, EMNLP.

[43]  Ashish Vaswani,et al.  Self-Attention with Relative Position Representations , 2018, NAACL.