Improving Domain Robustness in Neural Machine Translation with Fused Topic Knowledge Embeddings

Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 209–221 September 4–8, 2023, Macau SAR, China. ©2023 The authors. This article is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). 209 Improving Domain Robustness in Neural Machine Translation with Fused Topic Knowledge Embeddings Danai Xezonaki1,2 danai.xezonaki1@huawei-partners.com Talaat Khalil1∗ khalil.talaat@gmail.com David Stap2 d.stap@uva.nl Brandon James Denis1 brandon.james.denis@huawei.com 1 Huawei Technologies R&D, Amsterdam, Netherlands 2 Language Technology Lab, University of Amsterdam Abstract Domain robustness is a key challenge for Neural Machine Translation (NMT). Translating text from a different distribution than the training set requires the NMT models to generalize well to unseen domains. In this work we propose a novel way to address domain robustness, by fusing external topic knowledge into the NMT architecture. We employ a pretrained denoising autoencoder and fuse topic information into the system during continued pretraining, and fine- tuning of the model on the downstream NMT task. Our results show that incorporating external topic knowledge, as well as additional pretraining can improve the out-of-domain performance of NMT models. The proposed methodology meets state-of-the-art on out-of-domain perfor- mance. Our analysis shows that a low overlap between the pretraining and finetuning corpora, as well as the quality of topic representations help the NMT systems become more robust under domain shift. 1 Introduction Neural Machine Translation (NMT) has achieved impressive performance over the last few years when trained on large-scale data (Bojar et al., 2018). This success relies heavily on the availability of such data. The use of deep neural models has become the dominant approach for translation systems. However, it is not always possible to obtain neither parallel nor monolin- gual domain-specific data. Most approaches for improving domain robustness in NMT assume that the target domains are known in advance, and a significant amount of data is available from the target domain. In such cases, the dominant approach for addressing domain mismatch is domain adaptation. How- ever, when building translation systems and, as in many real-life scenarios, the target domains cannot always be known a priori. Koehn and Knowles (2017) were the first to identify domain mismatch as one of the main challenges of NMT. It is important therefore to develop translation systems that can generalize to domains unseen during training and thus be robust even under domain shift, as no target-domain data can be seen during training. ∗Work conducted while working in Huawei. 210 Furthermore, even if NMT systems are trained on large-scale data, it is always possible that new topics or domains will emerge over time. These new domains make it difficult to maintain large translation systems, since these would require additional training on the new domains. A typical example is the outbreak of COVID-19, which intruded into everyday life and affected millions of peoples’ lives. Keeping translation models up-to-date with such emerging topics is practically difficult, due to the limited availability of parallel data (Mahdieh et al., 2020). In this work we focus on the problem of improving robustness under domain …

Content truncated to 500 words