Domain Robustness in Neural Machine Translation

Domain Robustness in Neural Machine Translation Mathias M¨uller mmueller@cl.uzh.ch Annette Rios rios@cl.uzh.ch Rico Sennrich sennrich@cl.uzh.ch Department of Computational Linguistics, University of Zurich, Switzerland Abstract Translating text that diverges from the training domain is a key challenge for machine trans- lation. Domain robustness—the generalization of models to unseen test domains—is low for both statistical (SMT) and neural machine translation (NMT). In this paper, we study the per- formance of SMT and NMT models on out-of-domain test sets. We find that in unknown do- mains, SMT and NMT suffer from very different problems: SMT systems are mostly adequate but not fluent, while NMT systems are mostly fluent, but not adequate. For NMT, we identify such hallucinations (translations that are fluent but unrelated to the source) as a key reason for low domain robustness. To mitigate this problem, we empirically compare methods that are re- ported to improve adequacy or in-domain robustness in terms of their effectiveness at improving domain robustness. In experiments on German→English OPUS data, and German→Romansh (a low-resource setting) we find that several methods improve domain robustness. While those methods do lead to higher BLEU scores overall, they only slightly increase the adequacy of translations compared to SMT. 1 Introduction Even though neural models have improved the state-of-the-art in machine translation consid- erably in recent years, they still underperform in specific conditions. One such condition is out-of-domain translation. Koehn and Knowles (2017) found that neural machine translation (NMT) systems perform poorly in such settings and that their poor performance cannot be ex- plained solely by the fact that out-of-domain translation is difficult: non-neural, statistical ma- chine translation (SMT) systems were superior at this task. For this reason, Koehn and Knowles (2017) identified translation of out-of-domain text as a key challenge for NMT. Catastrophic failure to translate out-of-domain text can be viewed as overfitting to the train- ing domain, i.e. systems learn idiosyncrasies of the domain rather than more general features. Our goal is to learn models that generalize well to unseen data distributions, including data from other domains. We will refer to this property of showing good generalization to unseen domains as domain robustness. We consider domain robustness a desirable property of NLP systems, along with other types of robustness, such as robustness against adversarial examples (Goodfellow et al. 2015) or typos in the input (Belinkov and Bisk, 2018). While domain adaptation with small amounts of parallel or monolingual in-domain data has proven very effective for NMT (e.g. Luong and Manning, 2015; Sennrich et al., 2016a; Kobus et al., 2017; Li et al., 2019), the target domain(s) may be unknown when a system is built, and there are language pairs for which training data is only available for limited domains. Hence, domain robustness of systems without any domain adaptation is not only of theoretical interest, but also relevant in practice. Proceedings of the 14th Conference of the Association for Machine Translation in the Americas October 6 - 9, 2020, Volume 1: MT Research Track Page 151 SRC Aber geh subtil …

Content truncated to 500 words