Grounded Adaptation for Zero-shot Executable Semantic Parsing

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6869–6882, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6869 Grounded Adaptation for Zero-shot Executable Semantic Parsing Victor Zhong University of Washington Seattle, WA vzhong@cs.washington.edu Mike Lewis Facebook AI Research Seattle, WA mikelewis@fb.com Sida I. Wang Facebook AI Research Seattle, WA sidawang@fb.com Luke Zettlemoyer University of Washington Facebook AI Research Seattle, WA lsz@cs.washington.edu Abstract We propose Grounded Adaptation for Zero- shot Executable Semantic Parsing (GAZP) to adapt an existing semantic parser to new environments (e.g. new database schemas). GAZP combines a forward semantic parser with a backward utterance generator to syn- thesize data (e.g. utterances and SQL queries) in the new environment, then selects cycle- consistent examples to adapt the parser. Un- like data-augmentation, which typically syn- thesizes unverified examples in the train- ing environment, GAZP synthesizes exam- ples in the new environment whose input- output consistency are verified. On the Spider, Sparc, and CoSQL zero-shot semantic parsing tasks, GAZP improves logical form and exe- cution accuracy of the baseline parser. Our analyses show that GAZP outperforms data- augmentation in the training environment, per- formance increases with the amount of GAZP- synthesized data, and cycle-consistency is cen- tral to successful adaptation. 1 Introduction Semantic parsers (Zelle and Mooney, 1996; Zettle- moyer and Collins, 2005; Liang et al., 2011) build executable meaning representations for a range of tasks such as question-answering (Yih et al., 2014), robotic control (Matuszek et al., 2013), and in- telligent tutoring systems (Graesser et al., 2005). However, they are usually engineered for each ap- plication environment. For example, a language- to-SQL parser trained on an university manage- ment database struggles when deployed to a sales database. How do we adapt a semantic parser to new environments where no training data exists? We propose Grounded Adaptation for Zero-shot Executable Semantic Parsing, which adapts exist- ing semantic parsers to new environments by syn- thesizing new, cycle-consistent data. In the previ- ous example, GAZP synthesizes high-quality sales questions and SQL queries using the new sales database, then adapts the parser using the synthe- sized data. This procedure is shown in Figure 1. GAZP is complementary to prior modeling work in that it can be applied to any model architec- ture, in any domain where one can enforce cycle- consistency by evaluating equivalence between logical forms. Compared to data-augmentation, which typically synthesizes unverified data in the training environment, GAZP instead synthesizes consistency-verified data in the new environment. GAZP synthesizes data for consistency-verified adaptation using a forward semantic parser and a backward utterance generator. Given a new envi- ronment (e.g. new database), we first sample log- ical forms with respect to a grammar (e.g. SQL grammar conditioned on new database schema). Next, we generate utterances corresponding to these logical forms using the generator. Then, we parse the generated utterances using the parser, keeping those whose parses are equivalent to the original sampled logical form (more in Section 2.4). Finally, we adapt the parser to the new environment by training …

Content truncated to 500 words