Adaptively Sparse Transformers

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2174–2184, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2174 Adaptively Sparse Transformers Gonc¸alo M. Correia� goncalo.correia@lx.it.pt Vlad Niculae� vlad@vene.ro �Instituto de Telecomunicac¸˜oes, Lisbon, Portugal �Unbabel, Lisbon, Portugal Andr´e F.T. Martins� � andre.martins@unbabel.com Abstract Attention mechanisms have become ubiqui- tous in NLP. Recent architectures, notably the Transformer, learn powerful context-aware word representations through layered, multi- headed attention. The multiple heads learn diverse types of word relationships. How- ever, with standard softmax attention, all at- tention heads are dense, assigning a non-zero weight to all context words. In this work, we introduce the adaptively sparse Transformer, wherein attention heads have flexible, context- dependent sparsity patterns. This sparsity is accomplished by replacing softmax with α- entmax: a differentiable generalization of soft- max that allows low-scoring words to receive precisely zero weight. Moreover, we derive a method to automatically learn the α parameter – which controls the shape and sparsity of α- entmax – allowing attention heads to choose between focused or spread-out behavior. Our adaptively sparse Transformer improves inter- pretability and head diversity when compared to softmax Transformers on machine transla- tion datasets. Findings of the quantitative and qualitative analysis of our approach include that heads in different layers learn different sparsity preferences and tend to be more di- verse in their attention distributions than soft- max Transformers. Furthermore, at no cost in accuracy, sparsity in attention heads helps to uncover different head specializations. 1 Introduction The Transformer architecture (Vaswani et al., 2017) for deep neural networks has quickly risen to promi- nence in NLP through its efficiency and perfor- mance, leading to improvements in the state of the art of Neural Machine Translation (NMT; Junczys- Dowmunt et al., 2018; Ott et al., 2018), as well as inspiring other powerful general-purpose models like BERT (Devlin et al., 2019) and GPT-2 (Rad- ford et al., 2019). At the heart of the Transformer The quick brown fox jumps over The quick brown fox jumps over The quick brown fox jumps over head 1 head 4 head 3 head 2 Sparse Transformer Adaptive Span Transformer Adaptively Sparse Transformer (Ours) Figure 1: Attention distributions of different self- attention heads for the time step of the token “over”, shown to compare our model to other related work. While the sparse Transformer (Child et al., 2019) and the adaptive span Transformer (Sukhbaatar et al., 2019) only attend to words within a contiguous span of the past tokens, our model is not only able to obtain differ- ent and not necessarily contiguous sparsity patterns for each attention head, but is also able to tune its support over which tokens to attend adaptively. lie multi-head attention mechanisms: each word is represented by multiple different weighted aver- ages of its relevant context. As suggested by recent works on interpreting attention head roles, sepa- rate attention heads may learn to look for various relationships between tokens …

Content truncated to 500 words