Equivariant Transduction through Invariant Alignment

Proceedings of the 29th International Conference on Computational Linguistics, pages 4651–4663 October 12–17, 2022. 4651 Equivariant Transduction through Invariant Alignment Jennifer WhiteD Ryan CotterellQ DUniversity of Cambridge QETH Zürich jw2088@cam.ac.uk ryan.cotterell@inf.ethz.ch Abstract The ability to generalize compositionally is key to understanding the potentially infinite number of sentences that can be constructed in a human language from only a finite number of words. Investigating whether NLP models possess this ability has been a topic of interest: SCAN (Lake and Baroni, 2018) is one task specifically proposed to test for this property. Previous work has achieved impressive empirical results using a group-equivariant neural network that naturally encodes a useful inductive bias for SCAN (Gordon et al., 2020). Inspired by this, we introduce a novel group-equivariant architecture that incorporates a group-invariant hard alignment mechanism. We find that our network’s struc- ture allows it to develop stronger equivariance properties than existing group-equivariant approaches. We additionally find that it out- performs previous group-equivariant networks empirically on the SCAN task. Our results suggest that integrating group-equivariance into a variety of neural architectures is a potentially fruitful avenue of research, and demonstrate the value of careful analysis of the theoretical properties of such architectures. https://github.com/rycolab/equiv ariant-transduction 1 Introduction Humans painlessly process sentences they have never heard before. This feat is possible be- cause they can construct the meaning of a sen- tence by composing the meaning of its parts. This phenomenon is known as compositional general- ization in the computational linguistics literature (Lake and Baroni, 2018; Hupkes et al., 2020) and it is what enables the understanding of an infinite number of novel sentences from only a finite set of words (Chomsky, 1957; Montague, 1970). For example, without knowing what action the verb to blick describes, we know that blick twice indicates that this action should be performed two times. It is natural that we would like for neural network walk right → RTURN WALK walk and jump twice → WALK JUMP JUMP jump left after walk → WALK LTURN JUMP Figure 1: Examples of SCAN inputs and outputs models of language to be endowed with this ability as well—and, indeed, if they are to achieve human- like performance, it is likely to be necessary. There have been multiple proposals for methods of assessing these abilities in neural models (Bahdanau et al., 2019; Hupkes et al., 2020). One popular benchmark is the SCAN task (Lake and Baroni, 2018) and its derivatives (Ruis et al., 2020). SCAN involves translating natural language instruc- tions into a sequence of actions executable by an agent navigating in an environment. Examples of these commands are shown in Figure 1. In order to test generalization capabilities, there are various test–train splits which deliberately hold out specific words. For example, the training set for the Add Jump split only contains jump in the simple input–output pair (jump, JUMP), while the test set includes commands where it is combined with modifiers, such as jump twice and jump left. Since the model has seen the effect …

Content truncated to 500 words