Measuring Immediate Adaptation Performance for Neural Machine Translation

Proceedings of NAACL-HLT 2019, pages 2038–2046 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2038 Measuring Immediate Adaptation Performance for Neural Machine Translation Patrick Simianer, Joern Wuebker, John DeNero Lilt <given name>@lilt.com Abstract Incremental domain adaptation, in which a system learns from the correct output for each input immediately after making its predic- tion for that input, can dramatically improve system performance for interactive machine translation. Users of interactive systems are sensitive to the speed of adaptation and how often a system repeats mistakes, despite be- ing corrected. Adaptation is most commonly assessed using corpus-level BLEU- or TER- derived metrics that do not explicitly take adaptation speed into account. We find that these metrics often do not capture immediate adaptation effects, such as zero-shot and one- shot learning of domain-specific lexical items. To this end, we propose new metrics that di- rectly evaluate immediate adaptation perfor- mance for machine translation. We use these metrics to choose the most suitable adaptation method from a range of different adaptation techniques for neural machine translation sys- tems. 1 Introduction Incremental domain adaptation, or online adap- tation, has been shown to improve statistical machine translation and especially neural ma- chine translation (NMT) systems significantly (Turchi et al., 2017; Karimova et al., 2018) (inter-alia). The natural use case is a computer- aided translation (CAT) scenario, where a user and a machine translation system collaborate to translate a document. Each user translation is immediately used as a new training example to adapt the machine translation system to the specific document. Adaptation techniques for MT are typically evaluated by their corpus translation quality, but such evaluations may not capture prominent as- pects of the user experience in a collaborative translation scenario. This paper focuses on di- rectly measuring the speed of lexical acquisition for in-domain vocabulary. To that end, we propose three related metrics that are designed to reflect the responsiveness of adaptation. An ideal system would immediately acquire in- domain lexical items upon observing their transla- tions. Moreover, one might expect a neural system to generalize from one corrected translation to re- lated terms. Once a user translates “bank” to Ger- man “Bank” (institution) instead of “Ufer” (shore) in a document, the system should also correctly translate “banks” to “Banken” instead of “Ufer” (the plural is identical to the singular in German) in future sentences. We measure both one-shot vo- cabulary acquisition for terms that have appeared once in a previous target sentence, as well as zero- shot vocabulary acquisition for terms that have not previously appeared. Our experimental evaluation shows some surprising results. Methods that appear to have comparable performance using corpus quality metrics such as BLEU can differ substantially in zero-shot and one-shot vocabulary acquisition. In addition, we find that fine-tuning a neural model tends to improve one-shot vocabulary recall while degrading zero-shot vocabulary recall. We evaluate several adaptation techniques on a range of online adaptation datasets. Fine tuning applied to all parameters in the NMT model max- imizes …

Content truncated to 500 words