Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalization

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10793–10809 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10793 Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalization Inbar Oren1 Jonathan Herzig1 Jonathan Berant1,2 1School of Computer Science, Tel-Aviv University 2Allen Institute for Artificial Intelligence inbaroren@mail.tau.ac.il, {jonathan.herzig,joberant}@cs.tau.ac.il Abstract Modern semantic parsers suffer from two prin- cipal limitations. First, training requires ex- pensive collection of utterance-program pairs. Second, semantic parsers fail to generalize at test time to new compositions/structures that have not been observed during training. Re- cent research has shown that automatic gener- ation of synthetic utterance-program pairs can alleviate the first problem, but its potential for the second has thus far been under-explored. In this work, we investigate automatic gener- ation of synthetic utterance-program pairs for improving compositional generalization in se- mantic parsing. Given a small training set of annotated examples and an “infinite” pool of synthetic examples, we select a subset of syn- thetic examples that are structurally-diverse and use them to improve compositional gener- alization. We evaluate our approach on a new split of the schema2QA dataset, and show that it leads to dramatic improvements in composi- tional generalization as well as moderate im- provements in the traditional i.i.d setup. More- over, structurally-diverse sampling achieves these improvements with as few as 5K exam- ples, compared to 1M examples when sam- pling uniformly at random – a 200x improve- ment in data efficiency. 1 Introduction Semantic parsers map natural language utterances to executable programs (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005). A worrying weak- ness of semantic parsers that has been recently ex- posed, is their inability to generalize at test time to new compositions (Finegan-Dollak et al., 2018; Lake and Baroni, 2018; Keysers et al., 2020; Kim and Linzen, 2020; Gu et al., 2021). For example, a virtual assistant trained on the examples “Show me Thai restaurants that allow pets” and “How many hotels are there in Tokyo”, might not gener- alize to “How many hotels in Tokyo allow pets?”. This type of out-of-domain generalization to new Figure 1: Given a small set of annotated data and a large pool of synthetic examples, we propose to sam- ple a diverse training set of synthetic examples which includes a myriad of structures. Training a semantic parser on the synthetic data and fine-tuning on anno- tated data dramatically improves the parser’s composi- tional generalization. compositions constructed from components seen during training is commonly termed compositional generalization. Two high-level approaches have been consid- ered for tackling compositional generalization: (a) designing models with a stronger compositional inductive bias (Liu et al., 2020; Russin et al., 2020; Zheng and Lapata, 2020; Herzig and Berant, 2021), and (b) adding training data that will encourage a compositional solution (Akyürek et al., 2021; Guo et al., 2020c; Wang et al., 2021; Guo et al., 2020a). In the latter approach, typically a model is trained from labeled data (Jia and Liang, 2016; Yu et al., …

Content truncated to 500 words