Do Transformers Need Deep Long-Range Memory?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7524–7529 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7524 Do Transformers Need Deep Long-Range Memory? Jack W. Rae DeepMind & UCL London, UK jwrae@google.com Ali Razavi DeepMind London, UK alirazavi@google.com Abstract Deep attention models have advanced the mod- elling of sequential data across many do- mains. For language modelling in particu- lar, the Transformer-XL — a Transformer aug- mented with a long-range memory of past ac- tivations — has been shown to be state-of- the-art across a variety of well-studied bench- marks. The Transformer-XL incorporates a long-range memory at every layer of the net- work, which renders its state to be thousands of times larger than RNN predecessors. How- ever it is unclear whether this is necessary. We perform a set of interventions to show that comparable performance can be obtained with 6X fewer long range memories and better per- formance can be obtained by limiting the range of attention in lower layers of the network. 1 Introduction When we read a book, we maintain representations of the characters and events in the text that help us understand the story. We do this with a selective memorisation process; most of the finer details of the text are quickly forgotten and we retain a rela- tively compact representation of the book’s details. Early models of natural language used recurrent neural networks (RNNs) such as the Long Short- Term Memory (Hochreiter and Schmidhuber, 1997) which emulated this selective memory approach by modelling the past in a compact state vector. The model learns to store relevant information within its state implicitly in order to optimise the task loss. The LSTM has reigned as a state-of-the-art lan- guage model for over two decades since its incep- tion in the ’90s (Melis et al., 2017) and is arguably the most ubiquitous neural sequence model. Un- like human memory systems, however, the LSTM struggles to reason over long-range contexts when reading text. This has been observed in multi- ple contexts. In the carefully curated LAMBADA benchmark (Paperno et al., 2016) which tests lan- guage model predictions on sections of book text that have long term structure as decided by human raters, LSTMs completely fail. Namely LSTMs guess the correct word 0% of the time, where hu- mans are considered to be above 70% accuracy. For regular language modelling, Daniluk et al. (2017) observed that an LSTM augmented with attention would rarely attend beyond seven preceding words of context. Samples from LSTMs language models quickly devolve into generic text devoid of an over- all theme. This has lead many to wonder whether there is any non-negligible long-range signal in the task of language modelling. Recently we have seen that deep attention mod- els can draw long-range signal from text, even when the objective is as simple as next-word predic- tion. With the advent of the Transformer (Vaswani et al., 2017), significant gains in language mod- elling performance can be obtained by …

Content truncated to 500 words