Europeana Translate: Providing multilingual access to digital cultural heritage

Domain Adaptation for Neural Machine Translation Danielle Saunders∗ Department of Engineering University of Cambridge Cambridge, UK ds636@cantab.ac.uk The development of deep learning techniques has allowed Neural Machine Translation (NMT) models to become extremely powerful, given suf- ficient training data and training time. However, such translation models struggle when translating text of a new or unfamiliar domain (Koehn and Knowles, 2017). A domain may be a well-defined topic, text of a specific provenance, text of un- known provenance with an identifiable vocabulary distribution, or language with some other stylo- metric feature. NMT models can achieve good translation per- formance on domain-specific data via simple tun- ing on a representative training corpus. However, such data-centric approaches have negative side- effects, including over-fitting and brittleness on narrow-distribution samples and catastrophic for- getting of previously seen domains. This thesis focuses instead on more robust ap- proaches to domain adaptation for NMT. We con- sider the case where a system is adapted to a spec- ified domain of interest, but may also need to ac- commodate new language, or domain-mismatched sentences. As well, the thesis highlights that lines of MT research other than performance on tradi- tional ‘domains’ can be framed as domain adapta- tion problems. Techniques that are effective for e.g. adapting machine translation to a biomedi- cal domain can also be used when making use of language representations beyond the surface-level, or when encouraging better machine translation of gendered terms. Over the course of the thesis we pose and answer five research questions: ∗Now at RWS Language Weaver ∗© 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. How effective are data-centric approaches to NMT domain adaptation? We find that simply selecting-domain relevant training data and fine- tuning an existing model achieves strong results, especially when a domain-specific data curriculum is used during training. However, we also demon- strate the side-effects of exposure bias and catas- trophic forgetting. Given an adaptation set, what training schemes improve NMT quality? We investigate two varia- tions on the NMT adaptation algorithm, regular- ized tuning including Elastic Weighting Consoli- dation, and a new variant of Minimum Risk Train- ing. We show they can mitigate the pitfalls of data- centric adaptation. Aside from avoiding the fail- ure modes of data-centric methods, we show these methods may also give better model convergence. Can domain adaptation help when the test do- main is unknown? Most approaches to domain adaptation in the literature assume any unseen test data of interest has a known, fixed domain, with a matching set of tuning data. This thesis works to- wards relaxing these assumptions. We show that adapting sequentially across domains with regu- larization can achieve good cross-domain perfor- mance without knowing the specific test domain. We also explore domain adaptive model ensem- bling and automatic model selection. We find this can outperform oracle approaches, which se- lect the best model for inference by using known provenance labels. Can changing data representation have similar effects …

Content truncated to 500 words