Constrained Language Models Yield Few-Shot Semantic Parsers

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7699–7715 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7699 Constrained Language Models Yield Few-Shot Semantic Parsers Richard Shin, Christopher H. Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, Benjamin Van Durme Microsoft Semantic Machines sminfo@microsoft.com Abstract We explore the use of large pretrained lan- guage models as few-shot semantic parsers. The goal in semantic parsing is to generate a structured meaning representation given a nat- ural language input. However, language mod- els are trained to generate natural language. To bridge the gap, we use language models to paraphrase inputs into a controlled sublan- guage resembling English that can be automat- ically mapped to a target meaning representa- tion. Our results demonstrate that with only a small amount of data and very little code to convert into English-like representations, our blueprint for rapidly bootstrapping semantic parsers leads to surprisingly effective perfor- mance on multiple community tasks, greatly exceeding baseline methods also trained on the same limited data. 1 Introduction Large pretrained language models (LMs) like GPT- 3 (Brown et al., 2020) have shown increasingly impressive few-shot performance by formulating tasks as text-to-text generation problems (Raffel et al., 2020; Brown et al., 2020). Given only a trained LM and a short textual prompt that de- scribes and/or exemplifies a task, one can produce surprisingly accurate models for a variety of natu- ral language processing problems. However, task- specific semantic parsing does not naturally fit into this paradigm because such parsers typically use custom meaning representations that are unlikely to already exist on the web, let alone exist in large enough quantities to affect the parameters of these LMs. We leverage two key insights to overcome this barrier: (1) since LMs excel at generating natu- ral language, we should formulate semantic parsing as paraphrasing into a controlled sublanguage (Be- rant and Liang, 2014; Marzoev et al., 2020) and (2) autoregressive LMs can be efficiently constrained to search over only valid paraphrases, so the sub- language does not need to be learned from scratch. In particular, following Berant and Liang (2014), we envision a developer for some new domain first writing a synchronous context-free grammar (SCFG) that defines the space of supported (and well-formed) meaning representations along with canonical natural language constructions that ex- press them. Such a grammar maps between canon- ical natural language forms and domain-specific meaning representations, so that a separate LM- based system can focus entirely on mapping an unconstrained utterance u to a canonical (but still natural) form c. Furthermore, the grammar can be used to constrain this LM-based system so that the LM is only allowed to generate canonical ut- terances (i.e., utterances that correspond to well- formed meaning representations). Given such a grammar, an LM, and a handful of examples for priming the LM for the task of in- terest, our approach immediately yields a working semantic parser. While we do not expect the accu- …

Content truncated to 500 words