Improving Compositional Generalization with Latent Structure and Data Augmentation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4341 - 4362 July 10-15, 2022 ©2022 Association for Computational Linguistics Improving Compositional Generalization with Latent Structure and Data Augmentation Linlu Qiu1∗ † Peter Shaw1∗ Panupong Pasupat1 Paweł Krzysztof Nowak1 Tal Linzen1,2 Fei Sha1 Kristina Toutanova1 1 Google Research 2 New York Univeristy {linluqiu,petershaw,ppasupat,pawelnow,linzen,fsha,kristout}@google.com Abstract Generic unstructured neural networks have been shown to struggle on out-of-distribution compositional generalization. Compositional data augmentation via example recombina- tion has transferred some prior knowledge about compositionality to such black-box neu- ral models for several semantic parsing tasks, but this often required task-specific engineer- ing or provided limited gains. We present a more powerful data recombi- nation method using a model called Compo- sitional Structure Learner (CSL). CSL is a generative model with a quasi-synchronous context-free grammar backbone, which we in- duce from the training data. We sample recom- bined examples from CSL and add them to the fine-tuning data of a pre-trained sequence-to- sequence model (T5). This procedure effec- tively transfers most of CSL’s compositional bias to T5 for diagnostic tasks, and results in a model even stronger than a T5-CSL en- semble on two real world compositional gen- eralization tasks. This results in new state-of- the-art performance for these challenging se- mantic parsing tasks requiring generalization to both natural language variation and novel compositions of elements. 1 Introduction Compositional generalization refers to the ability to generalize to novel combinations of previously ob- served atoms.1 For example, we may ask a model to interpret the instruction “jump twice”, when the atoms “jump” and “twice” were each observed sep- arately during training but never in combination with each other (Lake and Baroni, 2018). Improving compositional generalization is seen as important for approaching human-like language understanding (Lake et al., 2017; Battaglia et al., ∗Equal contribution. †Work done as part of the Google AI Residency program. 1Also commonly referred to as elements or concepts. ORIGINAL EXAMPLES Add an event for next Tuesday to my calendar → (CreateEvent(date= NextDOW(dow=TUE) )) Create a meeting with Alice tomorrow → (CreateEvent(attendees= "Alice" , date= TMRW )) Who is on Bob ’s team → ( FindTeamOf(person= "Bob" ) ) ... Train CSL, a generative model with latent compositional structure CSL Sample synthetic examples from CSL SYNTHETIC EXAMPLES Create a meeting with Alice ’s team next Tuesday → (CreateEvent(attendees= FindTeamOf(person= "Alice" ) , date= NextDOW(dow=TUE) )) ... Train T5 on original and synthetic examples Augmented T5 Figure 1: An overview of our method for compositional data augmentation with CSL, a generative model with a QCFG backbone, which is automatically induced from the training data. We show a notional set of original and synthetic examples mapping utterances to programs. 2018) and is practically significant for real world applications, where models deployed in the wild often need to interpret new combinations of ele- ments not well-covered by expensive and poten- tially skewed annotated training data (Herzig and Berant, 2019; Yin et al., 2021). Generic neural …

Content truncated to 500 words