Neural Machine Translation with Heterogeneous Topic Knowledge Embeddings

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3197–3202 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3197 Neural Machine Translation with Heterogeneous Topic Knowledge Embeddings Weixuan Wang1 ∗, Wei Peng1 ∗, Meng Zhang2, Qun Liu2 1Artificial Intelligence Application Research Center, Huawei Technologies {weixuanwang2,peng.wei1}@huawei.com 2Noah’s Ark Lab, Huawei Technologies {zhangmeng92,qun.liu}@huawei.com Abstract Neural Machine Translation (NMT) has shown a strong ability to utilize local context to dis- ambiguate the meaning of words. However, it remains a challenge for NMT to leverage broader context information like topics. In this paper, we propose heterogeneous ways of em- bedding topic information at the sentence level into an NMT model to improve translation per- formance. Specifically, the topic information can be incorporated as pre-encoder topic em- bedding, post-encoder topic embedding, and decoder topic embedding to increase the likeli- hood of selecting target words from the same topic of the source sentence. Experimental results show that NMT models with the pro- posed topic knowledge embedding outperform the baselines on the English → German and English → French translation tasks. 1 1 Introduction Neural Machine Translation (NMT) utilizes local context captured from the mapping between the bi- texts to disambiguate the meaning of words. While existing NMT models can handle meaning ambigu- ities based on local contexts learned from explicit collocations, it remains a challenge for NMT to produce accurate results for words presented in implicit collocations. The notion of implicit collo- cation is referred to as the circumstance when the meaning of two or more words can not be learned from the available training data; broader context information like topics may be utilized to generate an accurate meaning. For example, in the sentence “he likes bank fishing”, the word “bank fishing” is likely to produce an ill-translated Chinese word “银行钓鱼” due to a lack of collocation of “bank (河岸)” and “fishing (钓鱼)”. An accurate trans- lation “河岸钓鱼” may be approachable when the shared topic (“recreational sport”) is leveraged. *Co-first authors. 1The codes are available at https://github.com/Vicky- Wil/topic-NMT Incorporating topic information into NMT has been explored in Zhang et al. (2016) and Wei et al. (2019) with both studies adapting Latent Dirich- let Allocation (LDA) (Blei et al., 2003) to model topics of source and target languages. Both works utilized the traditional encoder-decoder architec- ture with gated recurrent units (GRU) (Cho et al., 2014). Although Wei et al. (2019) showed that topic knowledge incorporation is also applicable to the Transformer architecture (Vaswani et al., 2017), it is argued that the joint learning of topic model- ing and NMT is not an ideal way. The training of topic models can leverage a large volume of easily accessible monolingual data. Once a topic model is learned, it can be reused in different translation scenarios without retraining NMT models. There- fore decoupling topic modeling and NMT is a more flexible and scalable option. In this paper, we propose heterogeneous ways of incorporating topic information into the Trans- former architecture. Specifically, the topic in- formation can …

Content truncated to 500 words