Can Transformers Process Recursive Nested Constructions, Like Humans?

Proceedings of the 29th International Conference on Computational Linguistics, pages 3226–3232 October 12–17, 2022. 3226 Can Transformers Process Recursive Nested Constructions, Like Humans? Yair Lakretz Cognitive Neuroimaging Unit NeuroSpin Gif-sur-Yvette, France Théo Desbordes Cognitive Neuroimaging Unit NeuroSpin, France Meta AI, Paris, France Dieuwke Hupkes Meta AI Paris, France Stanislas Dehaene Cognitive Neuroimaging Unit, NeuroSpin, Gif-sur-Yvette Collège de France, Paris, France Abstract Recursive processing is considered a hallmark of human linguistic abilities. A recent study evaluated recursive processing in recurrent neu- ral language models (RNN-LMs) and showed that such models perform below chance level on embedded dependencies within nested con- structions – a prototypical example of recur- sion in natural language. Here, we study if state-of-the-art Transformer LMs do any bet- ter. We test eight different Transformer LMs on two different types of nested constructions, which differ in whether the embedded (inner) dependency is short or long range. We find that Transformers achieve near-perfect performance on short-range embedded dependencies, sig- nificantly better than previous results reported for RNN-LMs and humans. However, on long- range embedded dependencies, Transformers’ performance sharply drops below chance level. Remarkably, the addition of only three words to the embedded dependency caused Transform- ers to fall from near-perfect to below-chance performance. Taken together, our results re- veal how brittle syntactic processing is in Trans- formers, compared to humans. 1 introduction One of the fundamental principles of contemporary linguistics states that language processing requires the ability to deal with nested structures. Recur- sion, a specific type of computation that involves repeatedly applying a function to its own output, is suggested to be at the core of this ability (Hauser et al., 2002). The strongest evidence for recur- sion in human language processing arises from the tree-like nested structure of sentences in natural language, in which phrases of a particular type (i.e. NPs) can be embedded in other phrases of that same type (Figure 1). Humans, it is argued, are endowed with a unique competence for recursive processing, which allows them to represent and pro- cess such nested tree structures (Chomsky, 2000; Hauser et al., 2002; Dehaene et al., 2015). S VP PP .. are NP NP RC VP holds NP PP NP cabinet the near NP man the that keys DP The Figure 1: A tree-structure representation of a recursive structure with two long-range dependencies, one nested within the other one. In recent years, neural language models (NLMs) have shown tremendous advances on a variety of linguistic tasks, such as next-word prediction, trans- lation or semantic inference. Furthermore, evalua- tions of their syntactic abilities have shown promis- ing results, with similar or even above-human per- formance on a variety of different tasks (Marvin and Linzen, 2018; Goldberg, 2019; Jumelet et al., 2021; Giulianelli et al., 2018). However, nega- tive results were recently also presented (Warstadt et al., 2020; Hu et al., 2020). In particular, when it comes to recursive processing, Lakretz et al. (2021b) showed that while recurrent neural network language models (RNN-LMs) perform well on long-range dependencies, such …

Content truncated to 500 words