Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11778 - 11793 December 7-11, 2022 ©2022 Association for Computational Linguistics Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality Yichen Jiang∗ Xiang Zhou∗ Mohit Bansal UNC Chapel Hill {yichenj, xzh, mbansal}@cs.unc.edu Abstract Recent datasets expose the lack of the system- atic generalization ability in standard sequence- to-sequence models. In this work, we analyze this behavior of seq2seq models and identify two contributing factors: a lack of mutual ex- clusivity bias (one target sequence can only be mapped to one source sequence), and the tendency to memorize whole examples rather than separating structures from contents. We propose two techniques to address these two is- sues respectively: Mutual Exclusivity Training that prevents the model from producing seen generations when facing novel examples via an unlikelihood-based loss, and prim2primX data augmentation that automatically diversi- fies the arguments of every syntactic function to prevent memorizing and provide a compo- sitional inductive bias without exposing test- set data. Combining these two techniques, we show substantial empirical improvements using standard sequence-to-sequence models (LSTMs and Transformers) on two widely-used compositionality datasets: SCAN and COGS. Finally, we provide analysis characterizing the improvements as well as the remaining chal- lenges, and provide detailed ablations of our method.1 1 Introduction Human intelligence demonstrates systematic com- positionality, the algebraic capacity to understand and produce a potentially infinite number of novel combinations of known components (Chomsky, 1957; Montague, 1970), in comprehending and generating natural language. However, a number of recent datasets, e.g., SCAN (Lake and Baroni, 2018), COGS (Kim and Linzen, 2020), etc., pro- vide clear evidence of the lack of systematic com- positionality in state-of-the-art neural networks. By ∗Equal contribution. 1Our code is available at https://github.com/ owenzx/met-primaug analogy with what humans do in meaningful learn- ing (Ausubel, 1963; Shi et al., 2022), methods to solve this problem can be categorized into deduc- tive learning and inductive learning methods. In de- ductive learning, the learner is first introduced to a general rule and then followed by specific examples where the rule is applied (Thornbury, 1999), while in inductive learning, the learner is first presented with abundant examples so that the learner can auto- matically infer the general rule by itself. Under the context of compositional generalization, deductive learning approaches directly inject useful priors (Li et al., 2019; Russin et al., 2020; Liu et al., 2020, 2021) to the model; while inductive learning ap- proaches utilize data augmentation(Andreas, 2020; Akyürek et al., 2021) to provide more accurate examples that facilitate the learning of composi- tionality. In this work, we focus on designing meth- ods that do not require any specific architectural changes, so it is applicable to standard sequence- to-sequence (seq2seq) models, and propose both deductive and inductive learning paradigms. Our deductive method is derived from an ob- servation that when facing compositionally novel examples, models have a tendency to generate old patterns seen during training, hence leading to nu- merous mistakes. Despite being a major challenge for models, …

Content truncated to 500 words