∞-former: Infinite Memory Transformer

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5468 - 5485 May 22-27, 2022 c⃝2022 Association for Computational Linguistics ∞-former: Infinite Memory Transformer Pedro Henrique Martins� Zita Marinho�� André F. T. Martins��� �Instituto de Telecomunicações �DeepMind �Institute of Systems and Robotics �LUMLIS (Lisbon ELLIS Unit), Instituto Superior Técnico �Unbabel Lisbon, Portugal pedrohenriqueamartins@tecnico.ulisboa.pt, zmarinho@google.com, andre.t.martins@tecnico.ulisboa.pt. Abstract Transformers are unable to model long-term memories effectively, since the amount of com- putation they need to perform grows with the context length. While variations of effi- cient transformers have been proposed, they all have a finite memory capacity and are forced to drop old information. In this paper, we propose the ∞-former, which extends the vanilla transformer with an unbounded long- term memory. By making use of a continuous- space attention mechanism to attend over the long-term memory, the ∞-former’s attention complexity becomes independent of the con- text length, trading off memory length with precision. In order to control where pre- cision is more important, ∞-former main- tains “sticky memories,” being able to model arbitrarily long contexts while keeping the computation budget fixed. Experiments on a synthetic sorting task, language modeling, and document grounded dialogue generation demonstrate the ∞-former’s ability to retain information from long sequences.1 1 Introduction When reading or writing a document, it is impor- tant to keep in memory the information previously read or written. Humans have a remarkable ability to remember long-term context, keeping in mem- ory the relevant details (Carroll, 2007; Kuhbandner, 2020). Recently, transformer-based language mod- els have achieved impressive results by increasing the context size (Radford et al., 2018, 2019; Dai et al., 2019; Rae et al., 2019; Brown et al., 2020). However, while humans process information se- quentially, updating their memories continuously, and recurrent neural networks (RNNs) update a single memory vector during time, transformers do not – they exhaustively query every representation associated to the past events. Thus, the amount 1The code is available at https://github.com/ deep-spin/infinite-former. of computation they need to perform grows with the length of the context, and, consequently, trans- formers have computational limitations about how much information can fit into memory. For exam- ple, a vanilla transformer requires quadratic time to process an input sequence and linear time to attend to the context when generating every new word. Several variations have been proposed to address this problem (Tay et al., 2020b). Some propose using sparse attention mechanisms, either with data-dependent patterns (Kitaev et al., 2020; Vyas et al., 2020; Tay et al., 2020a; Roy et al., 2021; Wang et al., 2021) or data-independent patterns (Child et al., 2019; Beltagy et al., 2020; Zaheer et al., 2020), reducing the self-attention complexity (Katharopoulos et al., 2020; Choromanski et al., 2021; Peng et al., 2021; Jaegle et al., 2021), and caching past representations in a memory (Dai et al., 2019; Rae et al., 2019). These models are able to reduce the attention complexity, and, conse- quently, to scale up to longer contexts. However, as their …

Content truncated to 500 words