Context-Aware Smoothing for Neural Machine Translation

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 11–20, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Context-Aware Smoothing for Neural Machine Translation Kehai Chen1∗, Rui Wang2†, Masao Utiyama2, Eiichiro Sumita2 and Tiejun Zhao1 1Machine Intelligence & Translation Laboratory, Harbin Institute of Technology 2ASTREC, National Institute of Information and Communications Technology (NICT) {khchen and tjzhao}@hit.edu.cn {wangrui, mutiyama and eiichiro.sumita}@nict.go.jp Abstract In Neural Machine Translation (NMT), each word is represented as a low- dimension, real-value vector for encoding its syntax and semantic information. This means that even if the word is in a different sentence context, it is represented as the fixed vector to learn source representation. Moreover, a large number of Out-Of- Vocabulary (OOV) words, which have different syntax and semantic information, are represented as the same vector representation of unk. To alleviate this problem, we propose a novel context- aware smoothing method to dynamically learn a sentence-specific vector for each word (including OOV words) depending on its local context words in a sentence. The learned context-aware representation is integrated into the NMT to improve the translation performance. Empirical results on NIST Chinese-to-English translation task show that the proposed approach achieves 1.78 BLEU improvements on average over a strong attentional NMT, and outperforms some existing systems. 1 Introduction Neural Machine Translation (NMT) (Kalchbren- ner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015), has shown prominent performances in comparison with the conventional Phrase Based Statistical Machine Translation (PBSMT) (Koehn et al., 2003). In NMT, a source sentence is converted into a vector representation by an RNN called encoder, then another RNN ∗Kehai Chen was an internship research fellow at NICT when conducting this work. †Corresponding author. called decoder generates target sentence word by word based on the source representation with attention information and target history. One advantage of NMT systems is that each word is represented as a low-dimension, real- valued vector, instead of storing statistical rules as in PBSMT. This means that even if the word is in a different sentence context, it is represented as the fixed vector to learn source representation. Figure 1 (a) shows two pair of Chinese-to-English parallel sentences in which two Chinese sentences contain the same word “da”. Intuitively, the “da” denotes “beating” in the first Chinese sentence while the “da” denotes “playing” in the second Chinese sentence. It is obvious that the “da” which denotes different meanings in a specific sentence is represented as the same word vector in the encoder of NMT, as show in Figure 1 (b). Although the RNN-based encoder can capture the sentence context for each word, we believe that offering better word vector with context-aware representation might help improve translation quality of NMT. Moreover, a large number of Out-Of- Vocabulary (OOV) words which have different syntax and semantic information are represented as the same vector representation of unk. Actually, this kind of simple approach may cause ambiguity of the sentences since the single unk breaks the structure of sentences, …

Content truncated to 500 words