Learning from Executions for Semantic Parsing

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2747–2759 June 6–11, 2021. ©2021 Association for Computational Linguistics 2747 Learning from Executions for Semantic Parsing Bailin Wang, Mirella Lapata and Ivan Titov Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh bailin.wang@ed.ac.uk, {mlap, ititov}@inf.ed.ac.uk Abstract Semantic parsing aims at translating natu- ral language (NL) utterances onto machine- interpretable programs, which can be executed against a real-world environment. The ex- pensive annotation of utterance-program pairs has long been acknowledged as a major bot- tleneck for the deployment of contemporary neural models to real-life applications. In this work, we focus on the task of semi-supervised learning where a limited amount of annotated data is available together with many unlabeled NL utterances. Based on the observation that programs which correspond to NL utterances must be always executable, we propose to en- courage a parser to generate executable pro- grams for unlabeled utterances. Due to the large search space of executable programs, conventional methods that use approximations based on beam-search such as self-training and top-k marginal likelihood training, do not per- form as well. Instead, we view the problem of learning from executions from the perspec- tive of posterior regularization and propose a set of new training objectives. Experimen- tal results on OVERNIGHT and GEOQUERY show that our new objectives outperform con- ventional methods, bridging the gap between semi-supervised and supervised learning. 1 Introduction Semantic parsing is the task of mapping nat- ural language (NL) utterances to meaning repre- sentations (aka programs) that can be executed against a real-world environment such as a knowl- edge base or a relational database. While neural sequence-to-sequence models (Dong and Lapata, 2016; Jia and Liang, 2016a) have achieved much success in this task in recent years, they usually re- quire a large amount of labeled data (i.e., utterance- program pairs) for training. However, annotat- ing utterances with programs is expensive as it requires expert knowledge of meaning representa- tions (e.g., lambda calculus, SQLs) and the envi- list all 3 star rated thai restaurants Program Candidates select restaurant where star_rating = thaiGold Exe   select restaurant where cuisine > 3   select restaurant where star_rating = 3   select restaurant where star_rating = 3 and cuisine = thai   Figure 1: Candidate programs for an utterance can be classified by executability (Exe); note that the gold pro- gram is always in the set of executable programs. We propose to ultilize the weak yet freely available signal of executablility for learning. ronment against which they are executed (e.g., a knowledge base, a relational database). An alter- native to annotation is to collect answers (or deno- tations) of programs, rather than programs them- selves (Liang et al., 2013; Berant et al., 2013). In this work, we focus on the more extreme setting where there are no annotations available for a large number of utterances. This setting resembles a common real-life scenario where massive …

Content truncated to 500 words