Adaptive Attention Span in Transformers

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 331–335 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 331 Adaptive Attention Span in Transformers Sainbayar Sukhbaatar Edouard Grave Piotr Bojanowski Armand Joulin Facebook AI Research {sainbar,egrave,bojanowski,ajoulin}@fb.com Abstract We propose a novel self-attention mechanism that can learn its optimal attention span. This allows us to extend significantly the maximum context size used in Transformer, while main- taining control over their memory footprint and computational time. We show the effec- tiveness of our approach on the task of charac- ter level language modeling, where we achieve state-of-the-art performances on text8 and enwiki8 by using a maximum context of 8k characters. 1 Introduction Language models are at the core of many NLP applications, like machine translation or dialogue. Recently, much progress has been made by a new neural network called Transformer (Vaswani et al., 2017). Part of its success is due to its ability to capture long term dependencies. This is achieved by taking long sequences as inputs and explicitly compute the relations between every token via a mechanism called the “self-attention” layer (Al- Rfou et al., 2019). While this layer allows for information to prop- agate across long distances, it has a computa- tional and memory cost that scales quadratically with the size of the input sequence. As a con- sequence, Transformers hardly scale to sequences of more than a thousand tokens. This is partic- ularly problematic in the case of character level language modeling where dependencies are often spread over a few thousands time steps. In this work, we propose an alternative to the self-attention layer to reduce the computational burden of a Transformer. Our layer learns its op- timal context size, resulting in a network where each attention layer gathers information on their own context. In practice, we observe that this leads to Transformer with small context in the low- level layers and very large ones for the last lay- ers. With this modification, we are able to scale input sequences to more than 8k tokens with no loss of performance, nor additional computational or memory cost. We validate our approach on the task of character level language modeling where we reach state-of-the-art performances while re- ducing the number of FLOPS. The code to repro- duce our results is publicly available1. 2 Approach 2.1 Sequential transformer network Language modeling is the problem of assigning a probability to a sequence of tokens (w1, . . . , wT ): P(w1, . . . , wT ) = T � t=1 P(wt | wt−1, . . . , w1). Recent progress was made with a new auto- regressive model called Sequential Trans- former (Vaswani et al., 2017). A Transformer is made of a sequence of layers that are composed of a block of parallel self-attention layers followed by a feedforward network. We refer to Vaswani et al. (2017) for the details on the structure. In this paper, we make a couple of modifications …

Content truncated to 500 words