Learning to Synthesize Data for Semantic Parsing

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2760–2766 June 6–11, 2021. ©2021 Association for Computational Linguistics 2760 Learning to Synthesize Data for Semantic Parsing Bailin Wang†∗ Wenpeng Yin‡ Xi Victoria Lin§∗ and Caiming Xiong‡ † University of Edinburgh, ‡ Salesforce Research, § Facebook AI bailin.wang@ed.ac.uk, {wyin,cxiong}@salesforce.com, victorialin@fb.com Abstract Synthesizing data for semantic parsing has gained increasing attention recently. How- ever, most methods require handcrafted (high- precision) rules in their generative process, hindering the exploration of diverse unseen data. In this work, we propose a generative model which features a (non-neural) PCFG that models the composition of programs (e.g., SQL), and a BART-based translation model that maps a program to an utterance. Due to the simplicity of PCFG and pre-trained BART, our generative model can be efficiently learned from existing data at hand. Moreover, explic- itly modeling compositions using PCFG leads to a better exploration of unseen programs, thus generate more diverse data. We evalu- ate our method in both in-domain and out-of- domain settings of text-to-SQL parsing on the standard benchmarks of GEOQUERY and SPI- DER, respectively. Our empirical results show that the synthesized data generated from our model can substantially help a semantic parser achieve better compositional and domain gen- eralization. 1 Introduction Recently, synthesizing data for semantic parsing has gained increasing attention (Yu et al., 2018a, 2020; Zhong et al., 2020). However, these mod- els require handcrafted rules (or templates) to syn- thesize new programs or utterance-program pairs. This can be sub-optimal as fixed rules cannot cap- ture the underlying distribution of programs which usually vary across different domains (Herzig and Berant, 2019). Meanwhile, designing such rules also requires human involvement with expert knowledge. To alleviate this, we propose to learn a generative model from the existing data at hand. Our key observation is that programs (e.g., SQL) ∗Work done at Salesforce Research. Bailin was doing a research internship. Domain PCFG select area where state_name = ‘texas’ select population where state_name = ‘california’ sample what is the area of Texas? how many people live in California? translate SQL utterance Figure 1: A two-stage generative process for synthesiz- ing utterance-SQL pairs. are formal languages that are intrinsically compo- sitional. That is, the underlying grammar of pro- grams is usually known and can be used to model the space of all possible programs effectively. Typ- ically, grammars are used to constrain the program space during decoding of neural parsers (Yin and Neubig, 2018; Krishnamurthy et al., 2017). In this work, we utilize grammars to generate (unseen) programs, which are then used to synthesize more parallel data for semantic parsing. Concretely, we use text-to-SQL as an example task, and propose a generative model to synthe- size utterance-SQL pairs. As illustrated in Fig- ure 1, we first employ a probabilistic context-free grammar (PCFG) to model the distribution of SQL queries. Then with the help of a SQL-to-text trans- lation model, the corresponding utterances of SQL queries are generated subsequently. …

Content truncated to 500 words