Few-Shot Semantic Parsing with Language Models Trained on Code

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5417 - 5425 July 10-15, 2022 ©2022 Association for Computational Linguistics Few-Shot Semantic Parsing with Language Models Trained on Code Richard Shin Microsoft Semantic Machines richard.shin@microsoft.com Benjamin Van Durme Microsoft Semantic Machines ben.vandurme@microsoft.com Abstract Large language models can perform semantic parsing with little training data, when prompted with in-context examples. It has been shown that this can be improved by formulating the problem as paraphrasing into canonical utter- ances, which casts the underlying meaning rep- resentation into a controlled natural language- like representation. Intuitively, such models can more easily output canonical utterances as they are closer to the natural language used for pre-training. Recently, models also pre- trained on code, like OpenAI Codex, have risen in prominence. For semantic parsing tasks where we map natural language into code, such models may prove more adept at it. In this pa- per, we test this hypothesis and find that Codex performs better on such tasks than equivalent GPT-3 models. We evaluate on Overnight and SMCalFlow and find that unlike GPT-3, Codex performs similarly when targeting meaning rep- resentations directly, perhaps because meaning representations are structured similar to code in these datasets. 1 Introduction Semantic parsing is the task of mapping natural language to a target meaning representation. Many approaches have been explored by the community, including a recent focus on the use of large au- toregressive language models (LMs). Such pre- trained LMs can achieve surprising levels of ac- curacy with relatively small numbers of examples. Further gains have come from constraining a de- coder to only consider syntactically valid outputs. Historically, language models have been con- structed using a large collection of natural lan- guage. And yet, the term “language” clearly applies to non-natural languages as well. Very large mod- els have been trained on mixed corpora, explicitly curated to include code (programming language) as well as natural language. Examples include GPT-J (Wang and Komatsuzaki, 2021), MT-NLG (Kharya and Alvi, 2021), and Gopher (Rae et al., 2021), with OpenAI Codex (Chen et al., 2021), PaLM-Coder (Chowdhery et al., 2022), and Austin et al. (2021) particularly focused on code. We revisit few-shot semantic parsing experi- ments from Shin et al. (2021), which used GPT-3 with constrained decoding into a controlled sublan- guage of English (canonical utterances) then trans- lated the canonical utterance output into the mean- ing representation using a synchronous context-free grammar (SCFG). In this work, we perform similar experiments on the Overnight (Wang et al., 2015) and SMCalFlow (Andreas et al., 2020) datasets,1 but using OpenAI Codex instead. As Codex has been trained on code, including natural language comments that explain its intent, we hypothesize that Codex will be more adept at semantic parsing for meaning representations resembling code. In this work, we find that: • Codex substantially narrows the gap in accuracy between predicting meaning representations di- rectly versus canonical utterances, thus obviat- ing the need to define canonical …

Content truncated to 500 words