STAR: SQL Guided Pre-Training for Context-dependent Text-to-SQL Parsing

In this paper, we propose a novel SQL guided pre-training framework STAR for context-dependent text-to-SQL parsing, which leverages contextual information to enrich natural language (NL) utterance and table schema representations for text-to-SQL conversations. Concretely, we propose two novel pre-training objectives which respectively explore the context-dependent interactions of NL utterances and SQL queries within each text-to-SQL conversation: (i) schema state tracking (SST) objective that tracks and explores the schema states of context-dependent SQL queries in the form of schema-states by predicting and updating the value of each schema slot during interaction; (ii) utterance dependency tracking (UDT) objective that employs weighted contrastive learning to pull together two semantically similar NL utterances and push away the representations of semantically dissimilar NL utterances within each conversation. In addition, we construct a high-quality large-scale context-dependent text-to-SQL conversation corpus to pre-train STAR. Extensive experiments show that STAR achieves new state-of-the-art performance on two downstream benchmarks (SParC and CoSQL), significantly outperforming previous pre-training methods and ranking first on the leaderboard. We believe the release of the constructed corpus, codebase and pre-trained STAR checkpoints would push forward the research in this area. For reproducibility, we release our code and data at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/star.

[1]  Fei Huang,et al.  SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers , 2022, COLING.

[2]  Fei Huang,et al.  A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions , 2022, ArXiv.

[3]  Bailin Wang,et al.  Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsing , 2022, KDD.

[4]  B. Dong,et al.  HIE-SQL: History Information Enhanced Network for Context-Dependent Text-to-SQL Semantic Parsing , 2022, FINDINGS.

[5]  Yongbin Li,et al.  S^2SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parsers , 2022, FINDINGS.

[6]  Qian Liu,et al.  TAPEX: Table Pre-training via Learning a Neural SQL Executor , 2021, ICLR.

[7]  Jian-Guang Lou,et al.  Awakening Latent Grounding from Pretrained Language Models for Semantic Parsing , 2021, FINDINGS.

[8]  Dzmitry Bahdanau,et al.  PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models , 2021, EMNLP.

[9]  Rui Zhang,et al.  Logic-Consistency Text Generation from Semantic Parses , 2021, FINDINGS.

[10]  Kai Yu,et al.  Decoupled Dialogue Modeling and Semantic Parsing for Multi-Turn Text-to-SQL , 2021, FINDINGS.

[11]  Kai Yu,et al.  LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations , 2021, ACL.

[12]  Pengfei Zhu,et al.  Dynamic Hybrid Relation Exploration Network for Cross-Domain Context-Dependent Semantic Parsing , 2021, AAAI.

[13]  Zhen-Hua Ling,et al.  A Multiple-Integration Encoder for Multi-Turn Text-to-SQL Semantic Parsing , 2021, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[14]  Jun Wang,et al.  Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training , 2020, AAAI.

[15]  Yu Hu,et al.  Tracking Interaction States for Multi-Turn Text-to-SQL Semantic Parsing , 2020, AAAI.

[16]  Ahmed Hassan Awadallah,et al.  Structure-Grounded Pretraining for Text-to-SQL , 2020, NAACL.

[17]  Dragomir R. Radev,et al.  GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing , 2020, ICLR.

[18]  Alex Polozov,et al.  SCoRe: Pre-Training for Context Representation in Conversational Semantic Parsing , 2021, ICLR.

[19]  Xiaojun Wan,et al.  IGSQL: Database Schema Interaction Graph Based Neural Model for Context-Dependent Text-to-SQL Generation , 2020, EMNLP.

[20]  Sida I. Wang,et al.  Grounded Adaptation for Zero-shot Executable Semantic Parsing , 2020, EMNLP.

[21]  Shujian Huang,et al.  Dialogue State Tracking with Explicit Slot Connection Modeling , 2020, ACL.

[22]  Graham Neubig,et al.  TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data , 2020, ACL.

[23]  Thomas Muller,et al.  TaPas: Weakly Supervised Table Parsing via Pre-training , 2020, ACL.

[24]  Quoc V. Le,et al.  ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators , 2020, ICLR.

[25]  Xiaodong Liu,et al.  RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers , 2019, ACL.

[26]  R'emi Louf,et al.  HuggingFace's Transformers: State-of-the-art Natural Language Processing , 2019, ArXiv.

[27]  Luyao Chen,et al.  CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases , 2019, EMNLP.

[28]  Tao Yu,et al.  Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions , 2019, EMNLP.

[29]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[30]  Tao Yu,et al.  SParC: Cross-Domain Semantic Parsing in Context , 2019, ACL.

[31]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[32]  Tao Yu,et al.  Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task , 2018, EMNLP.

[33]  Oriol Vinyals,et al.  Representation Learning with Contrastive Predictive Coding , 2018, ArXiv.

[34]  Roberto Cipolla,et al.  Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[35]  Richard Socher,et al.  Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning , 2018, ArXiv.

[36]  Kurt Mehlhorn,et al.  Weisfeiler-Lehman Graph Kernels , 2011, J. Mach. Learn. Res..

[37]  S. Niwattanakul,et al.  Using of Jaccard Coefficient for Keywords Similarity , 2022 .