Dynamic Data Selection and Weighting for Iterative Back-Translation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5894–5904, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5894 Dynamic Data Selection and Weighting for Iterative Back-Translation Zi-Yi DouZ Antonios AnastasopoulosD,† Graham NeubigZ Z Language Technologies Institute, Carnegie Mellon University D Department of Computer Science, George Mason University {zdou,gneubig}@cs.cmu.edu antonis@gmu.edu Abstract Back-translation has proven to be an effec- tive method to utilize monolingual data in neural machine translation (NMT), and itera- tively conducting back-translation can further improve the model performance. Selecting which monolingual data to back-translate is crucial, as we require that the resulting syn- thetic data are of high quality and reflect the target domain. To achieve these two goals, data selection and weighting strategies have been proposed, with a common practice being to select samples close to the target domain but also dissimilar to the average general-domain text. In this paper, we provide insights into this commonly used approach and generalize it to a dynamic curriculum learning strategy, which is applied to iterative back-translation models. In addition, we propose weighting strategies based on both the current quality of the sen- tence and its improvement over the previous iteration. We evaluate our models on domain adaptation, low-resource, and high-resource MT settings and on two language pairs. Ex- perimental results demonstrate that our meth- ods achieve improvements of up to 1.8 BLEU points over competitive baselines.1 1 Introduction Back-translation (Sennrich et al., 2016b) is an ef- fective strategy for improving the performance of neural machine translation (NMT) using monolin- gual data, delivering impressive gains over already competitive NMT models (Edunov et al., 2018). The strategy is simple: given monolingual data in the target language, one can use a translation model in the opposite of the desired translation direction to back-translate the monolingual data, effectively synthesizing a parallel dataset, which is in turn †: Work completed while at Carnegie Mellon University. 1Code: https://github.com/zdou0830/ dynamic_select_weight. next epoch next epoch General Domain (Monolingual) Target Domain (Monolingual) Moore-Lewis Ours Figure 1: The Moore and Lewis (2010) data selection strategy for domain adaptation constantly selects the same set of sentences which cannot well represent the target domain. Our approach, instead, selects different subsets of sentences at each epoch and we gradually shift from selecting samples from the general-domain distribution to samples from the target distribution. used to train the final translation model. Further improvements can be obtained by iteratively re- peating this process (Hoang et al., 2018) in both directions. However, not all monolingual data are equally important. An envisioned downstream application is very often characterized by a unique data distribu- tion. In such cases of domain shift, back-translating target domain data can be an effective strategy (Hu et al., 2019) for obtaining a better in-domain trans- lation model. One common strategy is to select samples that are both (1) close to the target distri- bution and (2) dissimilar to the average general- domain text (Moore and Lewis, 2010). However, as depicted in Figure 1, this method is not ideal because …

Content truncated to 500 words