Incorporating Discrete Translation Lexicons into Neural Machine Translation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1557–1567, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Incorporating Discrete Translation Lexicons into Neural Machine Translation Philip Arthur∗, Graham Neubig∗†, Satoshi Nakamura∗ ∗ Graduate School of Information Science, Nara Institute of Science and Technology † Language Technologies Institute, Carnegie Mellon University philip.arthur.om0@is.naist.jp gneubig@cs.cmu.edu s-nakamura@is.naist.jp Abstract Neural machine translation (NMT) often makes mistakes in translating low-frequency content words that are essential to understand- ing the meaning of the sentence. We propose a method to alleviate this problem by aug- menting NMT systems with discrete transla- tion lexicons that efficiently encode transla- tions of these low-frequency words. We de- scribe a method to calculate the lexicon proba- bility of the next word in the translation candi- date by using the attention vector of the NMT model to select which source word lexical probabilities the model should focus on. We test two methods to combine this probability with the standard NMT probability: (1) using it as a bias, and (2) linear interpolation. Exper- iments on two corpora show an improvement of 2.0-2.3 BLEU and 0.13-0.44 NIST score, and faster convergence time.1 1 Introduction Neural machine translation (NMT, §2; Kalchbren- ner and Blunsom (2013), Sutskever et al. (2014)) is a variant of statistical machine translation (SMT; Brown et al. (1993)), using neural networks. NMT has recently gained popularity due to its ability to model the translation process end-to-end using a sin- gle probabilistic model, and for its state-of-the-art performance on several language pairs (Luong et al., 2015a; Sennrich et al., 2016). One feature of NMT systems is that they treat each word in the vocabulary as a vector of 1Tools to replicate our experiments can be found at http://isw3.naist.jp/~philip-a/emnlp2016/index.html Input: I come from Tunisia. Reference: チュニジア の 出身です。 Chunisia no shusshindesu. (I’m from Tunisia.) System: ノルウェー の 出身です。 Noruue- no shusshindesu. (I’m from Norway.) Figure 1: An example of a mistake made by NMT on low-frequency content words. continuous-valued numbers. This is in contrast to more traditional SMT methods such as phrase-based machine translation (PBMT; Koehn et al. (2003)), which represent translations as discrete pairs of word strings in the source and target languages. The use of continuous representations is a major advan- tage, allowing NMT to share statistical power be- tween similar words (e.g. “dog” and “cat”) or con- texts (e.g. “this is” and “that is”). However, this property also has a drawback in that NMT systems often mistranslate into words that seem natural in the context, but do not reflect the content of the source sentence. For example, Figure 1 is a sentence from our data where the NMT system mistakenly trans- lated “Tunisia” into the word for “Norway.” This variety of error is particularly serious because the content words that are often mistranslated by NMT are also the words that play a key role in determining the whole meaning of the sentence. In contrast, PBMT and other traditional SMT methods tend to rarely make this …

Content truncated to 500 words