Recurrent Attention for Neural Machine Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3216–3225 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3216 Recurrent Attention for Neural Machine Translation Jiali Zeng1∗, Shuangzhi Wu1, Yongjing Yin2, Yufan Jiang1, Mu Li1 1Tencent Cloud Xiaowei, Beijing, China 2Zhejiang University, Westlake University, Zhejiang, China {lemonzeng,garyyfjiang,frostwu,ethanlli}@tencent.com yinyongjing@westlake.edu.cn Abstract Recent research questions the importance of the dot-product self-attention in Transformer models and shows that most attention heads learn simple positional patterns. In this pa- per, we push further in this research line and propose a novel substitute mechanism for self-attention: Recurrent AtteNtion (RAN). RAN directly learns attention weights with- out any token-to-token interaction and further improves their capacity by layer-to-layer in- teraction. Across an extensive set of exper- iments on 10 machine translation tasks, we find that RAN models are competitive and out- perform their Transformer counterpart in cer- tain scenarios, with fewer parameters and in- ference time. Particularly, when apply RAN to the decoder of Transformer, there brings consistent improvements by about +0.5 BLEU on 6 translation tasks and +1.0 BLEU on Turkish-English translation task. In addition, we conduct extensive analysis on the attention weights of RAN to confirm their reasonable- ness. Our RAN is a promising alternative to build more effective and efficient NMT mod- els. 1 Introduction Transformer models have achieved remarkable success in Neural Machine Translation (NMT) (Vaswani et al., 2017; Freitag and Firat, 2020; Fan et al., 2020). One of the most crucial component of Transformer is the dot-product multi-head self- attention, which is essential to learn relationships between words as well as complex structural repre- sentations. However, many studies have shown that the pairwise self-attention is over-parameterized and leads to a costly inference (Sanh et al., 2019; Correia et al., 2019; Xiao et al., 2019). Based on these observations, various improved networks are proposed by either pruning negligible heads (Voita ∗Corresponding author. et al., 2019; Michel et al., 2019) or replacing self- attention with more efficient one (Xu et al., 2019; Wu et al., 2019; Kitaev et al., 2020; Beltagy et al., 2020). More recently, several researches take this direc- tion to an extreme by replacing dot-product self- attention with fixed or trainable global position- based attentions (Zhang et al., 2018; Tay et al., 2020; You et al., 2020; Raganato et al., 2020). For example, You et al. (2020) roughly modeled atten- tion weights as hard-coded Gaussian distributions, based on the observation that most heads only focus their attention on a local neighborhood. Another more representative method is Random Synthesizer proposed by Tay et al. (2020). Dif- ferent from You et al. (2020), they simply treat attention weights of all heads in each layer as train- able parameters. At inference time, the attention weights are directly retrieved based on the index of the query without dot-product operation. However, these variants are not the ideal alternatives of self- attention due to the unsatisfactory performance. In this paper, we go further along this research line and show that self-attention is empirically re- …

Content truncated to 500 words