Modeling Recurrence for Transformer

Proceedings of NAACL-HLT 2019, pages 1198–1207 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1198 Modeling Recurrence for Transformer Jie Hao Florida State University haoj8711@gmail.com Xing Wang Tencent AI Lab brightxwang@tencent.com Baosong Yang University of Macau nlp2ct.baosong@gmail.com Longyue Wang Tencent AI Lab vinnylywang@tencent.com Jinfeng Zhang Florida State University jinfeng@stat.fsu.edu Zhaopeng Tu∗ Tencent AI Lab zptu@tencent.com Abstract Recently, the Transformer model (Vaswani et al., 2017) that is based solely on attention mechanisms, has advanced the state-of-the-art on various machine translation tasks. How- ever, recent studies reveal that the lack of re- currence hinders its further improvement of translation capacity (Chen et al., 2018; De- hghani et al., 2019). In response to this problem, we propose to directly model recur- rence for Transformer with an additional re- currence encoder. In addition to the stan- dard recurrent neural network, we introduce a novel attentive recurrent network to lever- age the strengths of both attention and recur- rent networks. Experimental results on the widely-used WMT14 English⇒German and WMT17 Chinese⇒English translation tasks demonstrate the effectiveness of the proposed approach. Our studies also reveal that the proposed model benefits from a short-cut that bridges the source and target sequences with a single recurrent layer, which outperforms its deep counterpart. 1 Introduction Recently, Transformer (Vaswani et al., 2017) – a new network architecture based solely on atten- tion mechanisms, has advanced the state-of-the-art on various translation tasks across language pairs. Compared with the conventional recurrent neu- ral network (RNN) (Schuster and Paliwal, 1997) based model that leverages recurrence as the ba- sic building module (Sutskever et al., 2014; Bah- danau et al., 2015; Chen et al., 2018), Transformer replaces RNN with self-attention network (SAN) to model the dependencies among input elements. One appealing strength of SAN is that it breaks ∗ Zhaopeng Tu is the corresponding author of the paper. This work was conducted when Jie Hao and Baosong Yang were interning at Tencent AI Lab. down the sequential assumption to obtain the abil- ity of highly parallel computation: input elements interact with each other simultaneously without regard to their distance. However, prior studies empirically show that the lack of recurrence modeling hinders Trans- former from further improvement of translation quality (Dehghani et al., 2019). Modeling re- currence is crucial for capturing several essen- tial properties of input sequence, such as struc- tural representations (Tran et al., 2016) and posi- tional encoding (Shaw et al., 2018), which are ex- actly the weaknesses of SAN (Tran et al., 2018). Recently, Chen et al. (2018) show that the rep- resentations learned by SAN-based and RNN- based encoders are complementary to each other, and merging them can improve translation perfor- mance for RNN-based NMT models. Starting from these findings, we propose to di- rectly model recurrence for Transformer with an additional recurrence encoder. The recurrence en- coder recurrently reads word embeddings of input sequence and outputs a sequence of hidden states, which serves as an additional information source to the Transformer decoder. In addition to …

Content truncated to 500 words