Measuring and Improving Compositional Generalization in Text-to-SQL via Component Alignment

In text-to-SQL tasks — as in much of NLP — 001 compositional generalization is a major chal- 002 lenge: neural networks struggle with compo- 003 sitional generalization where training and test 004 distributions differ. However, most recent at- 005 tempts to improve this are based on word-level 006 synthetic data or specific dataset splits to gen- 007 erate compositional biases. In this work, we 008 propose a clause-level compositional example 009 generation method. We first split the sentences 010 in the Spider text-to-SQL dataset into sub- 011 sentences, annotating each sub-sentence with 012 its corresponding SQL clause, resulting in a 013 new dataset Spider-SS. We then construct a fur- 014 ther dataset, Spider-CG, by composing Spider- 015 SS sub-sentences in different combinations, to 016 test the ability of models to generalize com- 017 positionally. Experiments show that existing 018 models suffer significant performance degra- 019 dation when evaluated on Spider-CG, even 020 though every sub-sentence is seen during train- 021 ing. To deal with this problem, we modify a 022 number of state-of-the-art models to train on 023 the segmented data of Spider-SS, and we show 024 that this method improves the generalization 025 performance. 1 026

[1]  Matthew Purver,et al.  Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization , 2021, EMNLP.

[2]  John R. Woodward,et al.  Natural SQL: Making SQL Easier to Infer from Natural Language Specifications , 2021, EMNLP.

[3]  Kenny Smith,et al.  Meta-Learning to Compositionally Generalize , 2021, ACL.

[4]  Mirella Lapata,et al.  Structured Reordering for Modeling Latent Alignments in Sequence Transduction , 2021, NeurIPS.

[5]  Jacob Andreas,et al.  Compositional Generalization for Neural Semantic Parsing via Span-level Supervised Attention , 2021, NAACL.

[6]  Jun Wang,et al.  Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training , 2020, AAAI.

[7]  Ming-Wei Chang,et al.  Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? , 2020, ACL.

[8]  Dragomir R. Radev,et al.  GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing , 2020, ICLR.

[9]  Jonathan Berant,et al.  Span-based Semantic Parsing for Compositional Generalization , 2020, ACL.

[10]  Jonathan Berant,et al.  Improving Compositional Generalization in Semantic Parsing , 2020, FINDINGS.

[11]  Chen Liang,et al.  Compositional Generalization via Neural-Symbolic Stack Machines , 2020, NeurIPS.

[12]  Marc van Zee,et al.  Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures , 2020, ArXiv.

[13]  Ming-Wei Chang,et al.  Exploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing , 2020, ACL.

[14]  Qian Liu,et al.  Compositional Generalization by Learning Analytical Expressions , 2020, NeurIPS.

[15]  Diyi Yang,et al.  ToTTo: A Controlled Table-To-Text Generation Dataset , 2020, EMNLP.

[16]  Xiaodong Liu,et al.  RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers , 2019, ACL.

[17]  Zachary Chase Lipton,et al.  Learning the Difference that Makes a Difference with Counterfactually-Augmented Data , 2019, ICLR.

[18]  Tatsunori B. Hashimoto,et al.  Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization , 2019, ArXiv.

[19]  Haixun Wang,et al.  SpatialNLI: A Spatial Domain Natural Language Interface to Databases Using Spatial Comprehension , 2019, SIGSPATIAL/GIS.

[20]  Jonathan Berant,et al.  Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing , 2019, ACL.

[21]  Ruixiao Sun,et al.  Transferable Natural Language Interface to Structured Queries Aided by Adversarial Generation , 2018, 2019 IEEE 13th International Conference on Semantic Computing (ICSC).

[22]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[23]  Tao Yu,et al.  SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-Domain Text-to-SQL Task , 2018, EMNLP.

[24]  Tao Yu,et al.  Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task , 2018, EMNLP.

[25]  Dragomir R. Radev,et al.  Improving Text-to-SQL Evaluation Methodology , 2018, ACL.

[26]  Marco Baroni,et al.  Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks , 2017, ICML.

[27]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.