Topic-Informed Neural Machine Translation

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1807–1817, Osaka, Japan, December 11-17 2016. Topic-Informed Neural Machine Translation Jian Zhang, Liangyou Li, Andy Way, Qun Liu ADAPT Centre School of Computing Dublin City University, Ireland jian.zhang,liangyou.li,andy.way,qun.liu@adaptcentre.ie Abstract In recent years, neural machine translation (NMT) has demonstrated state-of-the-art machine translation (MT) performance. It is a new approach to MT, which tries to learn a set of parameters to maximize the conditional probability of target sentences given source sentences. In this paper, we present a novel approach to improve the translation performance in NMT by conveying topic knowledge during translation. The proposed topic-informed NMT can increase the likelihood of selecting words from the same topic and domain for translation. Experimentally, we demonstrate that topic-informed NMT can achieve a 1.15 (3.3% relative) and 1.67 (5.4% relative) absolute improvement in BLEU score on the Chinese-to-English language pair using NIST 2004 and 2005 test sets, respectively, compared to NMT without topic information. 1 Introduction In statistical machine translation (SMT) (Och and Ney, 2000; Marcu and Wong, 2002; Koehn et al., 2007), several models are trained separately and then linearly integrated using the log-linear model (Och, 2003). Neural machine translation (NMT) (Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015), being a new approach, employs an individual large neural network to maximize the translation probability directly. One observation we obtain from machine translation (MT) training data is that some of the words within the same sentence often belong to the same or similar topic. Examine the example presented in Figure 1, where words Commercial, market, prices and bank have higher probabilities of being in the Financial domain. Intuitively, by allowing the topic information of the source input sentence and previous translated words to be provided to the decoder, we can maintain the same topic in the translations during the decoding phase, and consequently better translations can be produced. Specifically, when a source word has more than one translation option available (e.g. the word bank in Figure 1), it is clear that the translation in the Financial domain is more likely to be selected because many of the source words are from the same topic. Topic models have been applied successfully in many SMT works. For example, similar “topic con- sistent” behaviour is also observed by Su (2015). In his work, a context-aware topic model is integrated into SMT for better lexical selection. Xiao et al. (2012) and Zhang et al. (2014) focus on document translations and propose a topic-similarity model and a topic-sensitivity model for hierarchical phrase- based SMT (Chiang, 2005) on the document level. The topic-similarity model is used to encourage or penalize topic-sensitive rules, and the topic-sensitivity model is applied to balance topic-insensitive rules. However, these approaches cannot be directly used in NMT. In this work, we propose our novel topic-informed NMT model. In topic models, word-topic distribu- tions can be viewed as a vector. In NMT, words are represented as …

Content truncated to 500 words