Self-Attentive Residual Decoder for Neural Machine Translation

Proceedings of NAACL-HLT 2018, pages 1366–1379 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Self-Attentive Residual Decoder for Neural Machine Translation Lesly Miculicich Werlen*,†, Nikolaos Pappas*, Dhananjay Ram*,†, Andrei Popescu-Belis‡ *Idiap Research Institute, Switzerland, † ´Ecole polytechnique f´ed´erale de Lausanne (EPFL), Switzerland, ‡HEIG-VD / HES-SO, Switzerland {lmiculicich, npappas, dram}@idiap.ch andrei.popescu-belis@heig-vd.ch Abstract Neural sequence-to-sequence networks with attention have achieved remarkable perfor- mance for machine translation. One of the rea- sons for their effectiveness is their ability to capture relevant source-side contextual infor- mation at each time-step prediction through an attention mechanism. However, the target-side context is solely based on the sequence model which, in practice, is prone to a recency bias and lacks the ability to capture effectively non- sequential dependencies among words. To ad- dress this limitation, we propose a target-side- attentive residual recurrent network for decod- ing, where attention over previous words con- tributes directly to the prediction of the next word. The residual learning facilitates the flow of information from the distant past and is able to emphasize any of the previously translated words, hence it gains access to a wider context. The proposed model outperforms a neural MT baseline as well as a memory and self-attention network on three language pairs. The analysis of the attention learned by the decoder con- firms that it emphasizes a wider context, and that it captures syntactic-like structures. 1 Introduction Neural machine translation (NMT) has recently become the state-of-the-art approach to machine translation (Bojar et al., 2016). Several architec- tures have been proposed for this task (Kalchbren- ner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014; Gehring et al., 2017; Vaswani et al., 2017), but the attention-based NMT model designed by Bahdanau et al. (2015) is still con- sidered the de-facto baseline. This architecture is composed of two recurrent neural networks (RNNs), an encoder and a decoder, and an at- tention mechanism between them for modeling a (a) Baseline NMT decoder (b) Self-attentive residual dec. Figure 1: Comparison between the decoder of the base- line NMT and the proposed decoder with self-attentive residual connections. soft word-alignment. First, the model encodes the complete source sentence, and then decodes one word at a time. The decoder has access to all the context on the source side through the attention mechanism. However, on the target side, the con- textual information is represented only through a fixed-length vector, namely the hidden state of the decoder. As observed by Bahdanau et al. (2015), this creates a bottleneck which hinders the ability of the sequential model to learn longer-term infor- mation effectively. As pointed out by Cheng et al. (2016), sequen- tial models present two main problems for natural language processing. First, the memory of the en- coder is shared across multiple words and is prone to bias towards the recent past. Second, such mod- els do not fully capture the structural composition of language. To address these limitations, several recent models have been proposed, namely mem- ory networks …

Content truncated to 500 words