Inducing Transformer’s Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6253–6265 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6253 Inducing Transformer’s Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks Yichen Jiang and Mohit Bansal UNC Chapel Hill {yichenj, mbansal}@cs.unc.edu Abstract Systematic compositionality is an essential mechanism in human language, allowing the recombination of known parts to create novel expressions. However, existing neural models have been shown to lack this basic ability in learning symbolic structures. Motivated by the failure of a Transformer model on the SCAN compositionality challenge (Lake and Baroni, 2018), which requires parsing a command into actions, we propose two auxiliary se- quence prediction tasks as additional training supervision. These automatically-generated sequences are more representative of the un- derlying compositional symbolic structures of the input data. During inference, the model jointly predicts the next action and the next to- kens in the auxiliary sequences at each step. Experiments on the SCAN dataset show that our method encourages the Transformer to un- derstand compositional structures of the com- mand, improving its accuracy on multiple chal- lenging splits from ≤ 10% to 100%. With only 418 (5%) training instances, our approach still achieves 97.8% accuracy on the MCD1 split. Therefore, we argue that compositionality can be induced in Transformers given minimal but proper guidance. We also show that a better re- sult is achieved using less contextualized vec- tors as the attention’s query, providing insights into architecture choices in achieving system- atic compositionality. Finally, we show pos- itive generalization results on the grounded- SCAN task (Ruis et al., 2020). 1 1 Introduction Human intelligence, including natural languages, demonstrates systematic compositionality, the al- gebraic capacity to understand and produce a po- tentially infinite number of novel combinations of known components (Chomsky, 1957; Montague, 1Our code is publicly available at https://github.com/jiangycTarheel/ compositional-auxseq 1970). For example, we know the usage of words “walk,” “twice,” “and”; once we learn a new verb “dax”, we can immediately understand or produce utterances like “dax twice and walk.” This type of compositionality is central to the human abil- ity of making strong generalizations from limited data (Lake et al., 2017). However, there have been arguments that neural networks are associative de- vices that cannot capture systematic composition- ality (Fodor and Pylyshyn, 1988; Marcus, 1998; Fodor and Lepore, 2002; Marcus, 2003; Calvo and Symons, 2014). Supporting this view, it has been shown that general neural models, like RNNs and Transformers (Vaswani et al., 2017), generalize poorly to the development set’s unseen combina- tion of components seen in training set (Lake and Baroni, 2018; Liu et al., 2020). However, recent works have equipped recurrent neural networks (RNNs) with separate primitive and functional embeddings of the input tokens (Li et al., 2019; Russin et al., 2020). On the SCAN dataset (Lake and Baroni, 2018) that requires pars- ing a command into actions, these models can ef- fectively parse “jump thrice" when only trained on how to “walk thrice”, “walk”, and “jump”. In this work, we first …

Content truncated to 500 words