Look Harder: A Neural Machine Translation Model with Hard Attention

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3037–3043 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3037 Look Harder: A Neural Machine Translation Model with Hard Attention Sathish Indurthi Insoo Chung Sangha Kim Samsung Research, Seoul, South Korea {s.indurthi, insooo.chung, sangha01.kim}@samsung.com Abstract Soft-attention based Neural Machine Trans- lation (NMT) models have achieved promis- ing results on several translation tasks. These models attend all the words in the source se- quence for each target token, which makes them ineffective for long sequence transla- tion. In this work, we propose a hard-attention based NMT model which selects a subset of source tokens for each target token to effec- tively handle long sequence translation. Due to the discrete nature of the hard-attention mechanism, we design a reinforcement learn- ing algorithm coupled with reward shaping strategy to efficiently train it. Experimen- tal results show that the proposed model per- forms better on long sequences and thereby achieves significant BLEU score improvement on English-German (EN-DE) and English- French (EN-FR) translation tasks compared to the soft-attention based NMT. 1 Introduction In recent years, soft-attention based neural ma- chine translation models (Bahdanau et al., 2015; Gehring et al., 2017; Hassan et al., 2018) have achieved state-of-the-art results on different ma- chine translation tasks. The soft-attention mech- anism computes the context (encoder-decoder at- tention) vector for each target token by weight- ing and combining all the tokens of the source se- quence, which makes them ineffective for long se- quence translation (Lawson et al., 2017). More- over, weighting and combining all the tokens of the source sequence may not be required – a few relevant tokens are sufficient for each target token. Different attention mechanisms have been pro- posed to improve the quality of the context vec- tor. For example, Luong et al. (2015); Yang et al. (2018) proposed a local-attention mechanism to selectively focus on a small window of source to- kens to compute the context vector. Even though local-attention has improved the translation qual- ity, it is not flexible enough to focus on relevant to- kens when they fall outside the specified window size. To overcome the shortcomings of the above ap- proaches, we propose a hard-attention mechanism for a deep NMT model (Vaswani et al., 2017). The proposed model solely selects a few relevant to- kens across the entire source sequence for each target token to effectively handle long sequence translation. Due to the discrete nature of the hard-attention mechanism, we design a Reinforce- ment Learning (RL) algorithm with reward shap- ing strategy (Ng et al., 1999) to train it. The pro- posed hard-attention based NMT model consis- tently outperforms the soft-attention based NMT model (Vaswani et al., 2017), and the gap grows as the sequence length increases. 2 Background A typical NMT model based on encoder-decoder architecture generates a target sequence y = {y1, · · · , yn} given a source sequence x {x1, · · · , xm} by modeling the …

Content truncated to 500 words