Searching for Better Database Queries in the Outputs of Semantic Parsers

Findings of the Association for Computational Linguistics: EACL 2023, pages 2243–2256 May 2-6, 2023 ©2023 Association for Computational Linguistics Searching for Better Database Queries in the Outputs of Semantic Parsers Anton Osokin∗ HSE University Yandex Irina Saparina∗ HSE University Yandex Ramil Yarullin∗ HSE University Yandex Abstract The task of generating a database query from a question in natural language suffers from am- biguity and insufficiently precise description of the goal. The problem is amplified when the system needs to generalize to databases un- seen at training. In this paper, we consider the case when, at the test time, the system has access to an external criterion that evaluates the generated queries. The criterion can vary from checking that a query executes without errors to verifying the query on a set of tests. In this setting, we augment neural autoregressive models with a search algorithm that looks for a query satisfying the criterion. We apply our ap- proach to the state-of-the-art semantic parsers and report that it allows us to find many queries passing all the tests on different datasets. 1 Introduction Generating a database query from a natural- language description of the user’s intent is a long- standing and important task. In the recent years, most of the community focus was on the Spider dataset (Yu et al., 2018), which poses the task in the zero-shot regime, meaning that a method has to gen- eralize to databases unseen at training. The Spider dataset contains English questions and SQL queries. The progress has been remarkable, and the accu- racy has moved from below 30% to above 70%. A part of this success can be attributed to the adoption of pre-trained transformer models like BERT (De- vlin et al., 2019) into most of the pipelines. Given such progress, it is natural to ask whether we are getting closer to solving the problem. Sev- eral recent studies have noted that the task might be harder than it looks. Finegan-Dollak et al. (2018) found that many single-database datasets had iden- tical queries in both train and test sets and showed that using such splits effectively reduced the prob- lem to classifying the queries from the train set. ∗* Equal contribution. Shaw et al. (2021) continued the study in the multi- database setting and showed that the compositional generalization was hard to achieve, and even to measure it, one should be very careful with splits. In a different line of thought, Suhr et al. (2020) examined how the models trained on Spider gen- eralize to other datasets and reported that general- ization was challenging. Even within one dataset, many questions have several interpretations leading to different queries, and annotation policies do not cover these ambiguities or cover them differently. Acknowledging that the zero-shot setting might be too difficult to tackle as is, we aim to better define and simplify the problem to achieve better results in terms of the number of correctly gener- ated queries. In this task, most modern models produce a distribution over …

Content truncated to 500 words