Improving Compositional Generalization in Semantic Parsing

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2482–2495 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2482 Improving Compositional Generalization in Semantic Parsing Inbar Oren1 Jonathan Herzig∗,1 Nitish Gupta∗,2 Matt Gardner3 Jonathan Berant1,3 1School of Computer Science, Tel-Aviv University 2University of Pennsylvania 3Allen Institute for AI {inbaroren@mail, jonathan.herzig@cs, joberant@cs}.tau.ac.il, nitishg@seas.upenn.edu, mattg@allenai.org Abstract Generalization of models to out-of- distribution (OOD) data has captured tremendous attention recently. Specifically, compositional generalization, i.e., whether a model generalizes to new structures built of components observed during training, has sparked substantial interest. In this work, we investigate compositional generalization in semantic parsing, a natural test-bed for com- positional generalization, as output programs are constructed from sub-components. We analyze a wide variety of models and propose multiple extensions to the attention module of the semantic parser, aiming to improve compositional generalization. We find that the following factors improve compositional generalization: (a) using contextual repre- sentations, such as ELMO and BERT, (b) informing the decoder what input tokens have previously been attended to, (c) training the decoder attention to agree with pre-computed token alignments, and (d) downsampling examples corresponding to frequent program templates. While we substantially reduce the gap between in-distribution and OOD general- ization, performance on OOD compositions is still substantially lower. 1 Introduction Neural models trained on large datasets have re- cently shown great performance on data sampled from the training distribution. However, gener- alization to out-of-distribution (OOD) scenarios has been dramatically lower (Sagawa et al., 2019; Gardner et al., 2020; Kaushik et al., 2020). A par- ticularly interesting case of OOD generalization is compositional generalization, the ability to sys- tematically generalize to test examples composed of components seen during training. For exam- ple, we expect a model that observed the questions “What is the capital of France?” and “What is the ∗ The authors contributed equally. population of Spain?” at training time to general- ize to questions such as “What is the population of the capital of Spain?”. While humans gener- alize systematically to such compositions (Fodor et al., 1988), models often fail to capture the struc- ture underlying the problem, and thus miserably fail (Atzmon et al., 2016; Lake and Baroni, 2018; Loula et al., 2018; Bahdanau et al., 2019b; Ruis et al., 2020). Semantic parsing, mapping natural language ut- terances to structured programs, is a task where compositional generalization is expected, as sub- structures in the input utterance and output program often align. For example, in “What is the capital of the largest US state?”, the span “largest US state” might correspond to an argmax clause in the out- put program. Nevertheless, prior work (Finegan- Dollak et al., 2018; Herzig and Berant, 2019; Key- sers et al., 2020) has shown that data splits that require generalizing to new program templates re- sult in drastic loss of performance. However, past work did not investigate how different modeling choices interact with compositional generalization. In this paper, we thoroughly analyze the impact of different modeling choices on compositional generalization in 5 …

Content truncated to 500 words