Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 556–568 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 556 Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation Alessandro Raganato, Yves Scherrer and J¨org Tiedemann University of Helsinki {name.surname}@helsinki.fi Abstract Transformer-based models have brought a rad- ical change to neural machine translation. A key feature of the Transformer architecture is the so-called multi-head attention mechanism, which allows the model to focus simultane- ously on different parts of the input. However, recent works have shown that most attention heads learn simple, and often redundant, posi- tional patterns. In this paper, we propose to replace all but one attention head of each en- coder layer with simple fixed – non-learnable – attentive patterns that are solely based on po- sition and do not require any external knowl- edge. Our experiments with different data sizes and multiple language pairs show that fix- ing the attention heads on the encoder side of the Transformer at training time does not im- pact the translation quality and even increases BLEU scores by up to 3 points in low-resource scenarios. 1 Introduction Models based on the Transformer architecture (Vaswani et al., 2017) have led to tremendous per- formance increases in a wide range of downstream tasks (Devlin et al., 2019; Radford et al., 2019), in- cluding Machine Translation (MT) (Vaswani et al., 2017; Ott et al., 2018). One main component of the architecture is the multi-headed attention mechanism that allows the model to capture long- range contextual information. Despite these suc- cesses, the impact of the suggested parametrization choices, in particular the self-attention mechanism with its large number of attention heads distributed over several layers, has been the subject of many studies, following roughly four lines of research. The first line of research focuses on the interpre- tation of the network, in particular on the analysis of attention mechanisms and the interpretability of the weights and connections (Raganato and Tiede- mann, 2018; Tang et al., 2018; Mareˇcek and Rosa, 2019; Voita et al., 2019a; Brunner et al., 2020). A closely related research area attempts to guide the attention mechanism, e.g. by incorporating alignment objectives (Garg et al., 2019), or im- proving the representation through external infor- mation such as syntactic supervision (Pham et al., 2019; Currey and Heafield, 2019; Deguchi et al., 2019). The third line of research argues that Trans- former networks are over-parametrized and learn redundant information that can be pruned in vari- ous ways (Sanh et al., 2019). For example, Voita et al. (2019b) show that a few attention heads do the “heavy lifting” whereas others contribute very little or nothing at all. Similarly, Michel et al. (2019) raise the question whether 16 attention heads are really necessary to obtain competitive performance. Finally, several recent works address the computa- tional challenge of modeling very long sequences and modify the Transformer architecture with atten- tion operations that reduce time complexity (Shen et al., 2018; Wu et al., 2019; Child et al., 2019; Sukhbaatar …

Content truncated to 500 words