Addressing Exposure Bias With Document Minimum Risk Training: Cambridge at the WMT20 Biomedical Translation Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 862–869 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 862 Addressing Exposure Bias With Document Minimum Risk Training: Cambridge at the WMT20 Biomedical Translation Task Danielle Saunders and Bill Byrne Department of Engineering, University of Cambridge, UK Abstract The 2020 WMT Biomedical translation task evaluated Medline abstract translations. This is a small-domain translation task, meaning limited relevant training data with very distinct style and vocabulary. Models trained on such data are susceptible to exposure bias effects, particularly when training sentence pairs are imperfect translations of each other. This can result in poor behaviour during inference if the model learns to neglect the source sentence. The UNICAM entry addresses this problem during fine-tuning using a robust variant on Minimum Risk Training. We contrast this approach with data-filtering to remove ‘prob- lem’ training examples. Under MRT fine- tuning we obtain good results for both direc- tions of English-German and English-Spanish biomedical translation. In particular we achieve the best English-to-Spanish transla- tion result and second-best Spanish-to-English result, despite using only single models with no ensembling. 1 Introduction Neural Machine Translation (NMT) in the biomedical domain presents challenges in addition to general domain translation. Text often contains specialist vocabulary and follows specific stylistic conventions. For this task fine-tuning generic pre- trained models on smaller amounts of biomedical- specific data can lead to strong performance, as we found in our 2019 biomedical submission (Saun- ders et al., 2019). For our WMT 2020 submis- sion we start with strong single models from that 2019 submission and fine-tune them exclusively on the small Medline abstracts training sets (Baw- den et al., 2019). This allows fast training on very relevant training data, since the test set is also made up of Medline abstracts. However, fine-tuning on relevant but small cor- pora has pitfalls. The small number of training examples exacerbates the effect of any noisy or poorly aligned sentence pairs. We treat this as a form of exposure bias, in that model overconfi- dence in training data results in poor translation hypotheses at test time. Our contributions in this system paper are: • A discussion of exposure bias in the form of imperfect training data, focusing on the biomedical domain. • An exploration of straightforward ways to mitigate exposure bias via data preparation and training objective. • A discussion of our 2020 Biomedical task re- sults for single models fine-tuned on small, domain-specific data sets. 1.1 Exposure bias in the biomedical domain Exposure bias for an autoregressive sequence de- coder refers to a discrepancy between decoder conditioning during training and inference (Ben- gio et al., 2015; Ranzato et al., 2016). During training the decoder generates a hypothesis for the tth output token ˆyt conditioned on y1:t−1, the gold target sequence prefix. During inference, the gold target y is unavailable, and ˆyt is conditioned in- stead on the hypothesis prefix ˆy1:t−1. Previous work has interpreted the risk of expo- sure bias primarily in terms of the model over- …

Content truncated to 500 words