Meta-Learning to Compositionally Generalize

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3322–3335 August 1–6, 2021. ©2021 Association for Computational Linguistics 3322 Meta-Learning to Compositionally Generalize Henry Conklin1∗, Bailin Wang1∗, Kenny Smith1 and Ivan Titov1,2 1University of Edinburgh 2University of Amsterdam {henry.conklin, bailin.wang, kenny.smith}@ed.ac.uk, ititov@inf.ed.ac.uk Abstract Natural language is compositional; the mean- ing of a sentence is a function of the mean- ing of its parts. This property allows hu- mans to create and interpret novel sentences, generalizing robustly outside their prior ex- perience. Neural networks have been shown to struggle with this kind of generalization, in particular performing poorly on tasks de- signed to assess compositional generalization (i.e. where training and testing distributions differ in ways that would be trivial for a com- positional strategy to resolve). Their poor per- formance on these tasks may in part be due to the nature of supervised learning which as- sumes training and testing data to be drawn from the same distribution. We implement a meta-learning augmented version of super- vised learning whose objective directly op- timizes for out-of-distribution generalization. We construct pairs of tasks for meta-learning by sub-sampling existing training data. Each pair of tasks is constructed to contain relevant examples, as determined by a similarity metric, in an effort to inhibit models from memorizing their input. Experimental results on the COGS and SCAN datasets show that our similarity- driven meta-learning can improve generaliza- tion performance. 1 Introduction Compositionality is the property of human lan- guage that allows for the meaning of a sentence to be constructed from the meaning of its parts and the way in which they are combined (Cann, 1993). By decomposing phrases into known parts we can generalize to novel sentences despite never having encountered them before. In practice this allows us to produce and interpret a functionally limitless number of sentences given finite means (Chomsky, 1965). ∗Equal contribution. Whether or not neural networks can generalize in this way remains unanswered. Prior work as- serts that there exist fundamental differences be- tween cognitive and connectionist architectures that makes compositional generalization by the lat- ter unlikely (Fodor and Pylyshyn, 1988). However, recent work has shown these models’ capacity for learning some syntactic properties. Hupkes et al. (2018) show how some architectures can handle hierarchy in an algebraic context and generalize in a limited way to unseen depths and lengths. Work looking at the latent representations learned by deep machine translation systems show how these models seem to extract constituency and syntactic class information from data (Blevins et al., 2018; Belinkov et al., 2018). These results, and the more general fact that neural models perform a variety of NLP tasks with high fidelity (eg. Vaswani et al., 2017; Dong and Lapata, 2016), suggest these mod- els have some sensitivity to syntactic structure and by extension may be able to learn to generalize compositionally. Recently there have been a number of datasets designed to more formally assess connectionist models’ …

Content truncated to 500 words