Target Conditioned Sampling: Optimizing Data Selection for Multilingual Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5823–5828 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5823 Target Conditioned Sampling: Optimizing Data Selection for Multilingual Neural Machine Translation Xinyi Wang Language Technologies Institute Carnegie Mellon University xinyiw1@cs.cmu.edu Graham Neubig Language Technologies Institute Carnegie Mellon University gneubig@cs.cmu.edu Abstract To improve low-resource Neural Machine Translation (NMT) with multilingual corpora, training on the most related high-resource lan- guage only is often more effective than us- ing all data available (Neubig and Hu, 2018). However, it is possible that an intelligent data selection strategy can further improve low- resource NMT with data from other auxiliary languages. In this paper, we seek to construct a sampling distribution over all multilingual data, so that it minimizes the training loss of the low-resource language. Based on this for- mulation, we propose an efficient algorithm, Target Conditioned Sampling (TCS), which first samples a target sentence, and then con- ditionally samples its source sentence. Exper- iments show that TCS brings significant gains of up to 2 BLEU on three of four languages we test, with minimal training overhead1. 1 Introduction Multilingual NMT has led to impressive gains in translation accuracy of low-resource lan- guages (LRL) (Zoph et al., 2016; Firat et al., 2016; Gu et al., 2018; Neubig and Hu, 2018; Nguyen and Chiang, 2018). Many real world datasets provide sentences that are multi-parallel, with the same content in a variety of languages. Examples in- clude TED (Qi et al., 2018), Europarl (Koehn, 2005), and many others (Tiedemann, 2012). These datasets open up the tantalizing prospect of train- ing a system on many different languages to improve accuracy, but previous work has found methods that use only a single related (HRL) of- ten out-perform systems trained on all available data (Neubig and Hu, 2018). In addition, be- cause the resulting training corpus is smaller, us- ing a single language is also substantially faster to 1The code can be found at https://github.com/ cindyxinyiwang/TCS. train, speeding experimental cycles (Neubig and Hu, 2018). In this paper, we go a step further and ask the question: can we design an intelligent data selection strategy that allows us to choose the most relevant multilingual data to further boost NMT performance and training speed for LRLs? Prior work has examined data selection from the view of domain adaptation, selecting good train- ing data from out-of-domain text to improve in- domain performance. In general, these methods select data that score above a preset threshold ac- cording to some metric, such as the difference between in-domain and out-of-domain language models (Axelrod et al., 2011; Moore and Lewis, 2010) or sentence embedding similarity (Wang et al., 2017). Other works use all the data but weight training instances by domain similar- ity (Chen et al., 2017), or sample subsets of train- ing data at each epoch (van der Wees et al., 2017). However, none of these methods are trivially ap- plicable to multilingual parallel datasets, which usually contain …

Content truncated to 500 words