Making Transformers Solve Compositional Tasks

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3591 - 3607 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Making Transformers Solve Compositional Tasks Santiago Ontañón, Joshua Ainslie, Vaclav Cvicek, Zachary Fisher Google Research {santiontanon, jainslie, vcvicek, zachfisher}@google.com Abstract Several studies have reported the inability of Transformer models to generalize composi- tionally, a key type of generalization in many NLP tasks such as semantic parsing. In this paper we explore the design space of Trans- former models showing that the inductive bi- ases given to the model by several design deci- sions significantly impact compositional gen- eralization. We identified Transformer config- urations that generalize compositionally signif- icantly better than previously reported in the literature in many compositional tasks. We achieve state-of-the-art results in a semantic parsing compositional generalization bench- mark (COGS), and a string edit operation com- position benchmark (PCFG). 1 Introduction Although modern neural network architectures reach state-of-the-art performance in many chal- lenging natural language tasks, they seem to exhibit a low amount of “compositional generalization”, i.e., the ability to learn a set of basic primitives and combine them in more complex ways than those seen during training (Hupkes et al., 2020). For ex- ample, suppose a system has learned the meaning of “jump” and that “jump twice” means that the action “jump” has to be repeated two times. Upon learning the meaning of the action “jax”, it should be able to infer what “jax twice” means. Compo- sitional generalization is a key aspect of natural language and many other tasks we might want ma- chine learning models to learn. While both humans and classical AI techniques (such as grammars or search-based systems) can handle compositional tasks with relative ease, it seems that modern deep learning techniques do not possess this ability. A key question is thus: Can we build deep learning architectures that can also solve compositional tasks? In this paper we focus on Transformers (Vaswani et al., 2017), which have been shown in the literature to exhibit poor com- positional generalization (see Section 2). Through an empirical study, we show that this can be im- proved. With the goal of creating general models that generalize compositionally in a large range of tasks, we show that several design decisions, such as position encodings, decoder type, weight shar- ing, model hyper-parameters, and formulation of the target task result in different inductive biases, with significant impact for compositional general- ization1. We use a collection of twelve datasets designed to measure compositional generalization. In addition to six standard datasets commonly used in the literature (such as SCAN (Lake and Baroni, 2018), PCFG (Hupkes et al., 2020), CFQ (Keysers et al., 2019) and COGS (Kim and Linzen, 2020)), we also use a set of basic algorithmic tasks (such as addition, duplication, or set intersection) that although not directly involving natural language, are useful to obtain insights into what can and can- not be learned with different Transformer models. We also include tasks where …

Content truncated to 500 words