Improving Both Domain Robustness and Domain Adaptability in Machine Translation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5191–5204 October 12–17, 2022. 5191 Improving Both Domain Robustness and Domain Adaptability in Machine Translation Wen Lai1 and Jindˇrich Libovický2 and Alexander Fraser1 1 Center for Information and Language Processing, LMU Munich, Germany 2 Faculty of Mathematics and Physics, Charles Univeristy, Prague, Czech Republic {lavine, fraser}@cis.lmu.de libovicky@ufal.mff.cuni.cz Abstract We consider two problems of NMT domain adaptation using meta-learning. First, we want to reach domain robustness, i.e., we want to reach high quality on both domains seen in the training data and unseen domains. Second, we want our systems to be adaptive, i.e., making it possible to finetune systems with just hundreds of in-domain parallel sentences. We study the domain adaptability of meta-learning when im- proving the domain robustness of the model. In this paper, we propose a novel approach, RML- NMT (Robust Meta-Learning Framework for Neural Machine Translation Domain Adapta- tion), which improves the robustness of ex- isting meta-learning models. More specifi- cally, we show how to use a domain classi- fier in curriculum learning and we integrate the word-level domain mixing model into the meta- learning framework with a balanced sampling strategy. Experiments on English→German and English→Chinese translation show that RMLNMT improves in terms of both domain robustness and domain adaptability in seen and unseen domains1. 1 Introduction The success of Neural Machine Translation (NMT; Bahdanau et al., 2015; Vaswani et al., 2017) heav- ily relies on large-scale high-quality parallel data, which is difficult to obtain in some domains. We study two major problems in NMT domain adapta- tion. First, models should work well on both seen domains (the domains in the training data) and un- seen domains (domains which do not occur in the training data). We call this property domain ro- bustness. Second, with just hundreds of in-domain sentences, we want to be able to quickly adapt to a new domain. We call this property domain adapt- ability. Previous work on NMT domain adaptation has usually focused on only one aspect of domain 1Our source code is available at https://github.com/lavine- lmu/RMLNMT adaptation at the expense of the other one, and our motivation is to consider both of the two properties. There are a few works attempting to solve do- main adaptability. The most basic approach is fine-tuning, in which an out-of-domain model is continually trained on in-domain data (Freitag and Al-Onaizan, 2016; Dakwale and Monz, 2017). Al- though fine-tuning is effective, it can suffer from so-called catastrophic forgetting (French, 1999), resulting in deteriorated model performance in gen- eral domains (Thompson et al., 2019). Another efficient method is Meta-Learning (Hospedales et al., 2021), which trains models which can be later rapidly adapted to new scenarios using only a small amount of data. It works for many natural language processing (NLP) tasks (Gu et al., 2018; Qian and Yu, 2019; Yu et al., 2020; Bansal et al., 2020; Wang et al., 2021; Du et al., 2021), espe- cially in low-resource scenarios (Dou et al., 2019; Yin, 2020). As a …

Content truncated to 500 words