On The Ingredients of an Effective Zero-shot Semantic Parser

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1455 - 1474 May 22-27, 2022 c⃝2022 Association for Computational Linguistics On The Ingredients of an Effective Zero-shot Semantic Parser Pengcheng Yin♠∗ John Wieting♣ Avirup Sil♦ Graham Neubig♠ ♠Carnegie Mellon University ♣Google Research ♦IBM Research AI {pcyin,gneubig}@cs.cmu.edu jwieting@google.com avi@us.ibm.com Abstract Semantic parsers map natural language utter- ances into meaning representations (e.g. pro- grams). Such models are typically bottlenecked by the paucity of training data due to the labo- rious annotation efforts. Recent studies have performed zero-shot learning by synthesizing training examples of canonical utterances and programs from a grammar, and further para- phrasing these utterances to improve linguistic diversity. However, such synthetic examples cannot fully capture patterns in real data. In this paper we analyze zero-shot parsers through the lenses of the language and logical gaps (Herzig and Berant, 2019), which quantify the discrepancy of language and programmatic pat- terns between the synthetic canonical examples and real-world user-issued ones. We propose bridging these gaps using improved grammars, stronger paraphrasers, and efficient learning methods using canonical examples that most likely reflect real user intents. Our model achieves strong results on the SCHOLAR and GEO benchmarks with zero labeled data.1 1 Introduction Semantic parsers translate natural language (NL) utterances into formal meaning representations. In particular, task-oriented semantic parsers map user-issued utterances (e.g. Find papers in ACL) into machine-executable programs (e.g. a database query), play a key role in providing natural lan- guage interfaces to applications like conversational virtual assistants (Gupta et al., 2018; Andreas et al., 2020), robot instruction following (Artzi and Zettle- moyer, 2013; Fried et al., 2018), as well as querying databases (Li and Jagadish, 2014; Yu et al., 2018) or generating Python code (Yin and Neubig, 2017). Learning semantic parsers typically requires parallel data of utterances annotated with pro- grams, which requires significant expertise and ∗Now at Google Brain. Email to pcyin@google.com. 1https://pcyin.me/zeroshot_parser/ cost (Berant et al., 2013). Thus, the field has explored alternative approaches using supervi- sions cheaper to acquire, such as the execution results (Clarke et al., 2010) or unlabeled utter- ances (Poon, 2013). In particular, the seminal OVERNIGHT approach (Wang et al., 2015) synthe- sizes parallel data by using a synchronous grammar to align programs and their canonical NL expres- sions (e.g. Filter(paper,venue= ? ) ↔ papers in ? and acl↔ACL), then generating examples of compositional utterances (e.g. Papers in ACL) with programs (e.g. Filter(paper,venue=acl)). The synthesized utterances are paraphrased by an- notators, a much easier task than writing programs. Recently, Xu et al. (2020b) build upon OVERNIGHT and develop a zero-shot semantic parser replacing the manual paraphrasing process with an automatic paraphrase generator (§2). While promising, there are still several open challenges. First, such systems are not truly zero-shot — they still require labeled validation data (e.g. to select the best checkpoint at training). Next, to ensure the quality and broad-coverage of synthetic canonical examples, existing models rely on heavily curated grammars (e.g. with 800 production rules), which …

Content truncated to 500 words