Data Selection for Unsupervised Translation of German–Upper Sorbian

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1099–1103 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1099 Data Selection for Unsupervised Translation of German–Upper Sorbian Lukas Edman Antonio Toral Center for Language and Cognition University of Groningen {j.l.edman, a.toral.ruiz, g.j.m.van.noord}@rug.nl Gertjan van Noord Abstract This paper describes the methods behind the systems submitted by the University of Gronin- gen for the WMT 2020 Unsupervised Machine Translation task for German–Upper Sorbian. We investigate the usefulness of data selection in the unsupervised setting. We find that we can perform data selection using a pretrained model and show that the quality of a set of sen- tences or documents can have a great impact on the performance of the unsupervised neural machine translation (UNMT) system trained on it. Furthermore, we show that document- level data selection should be preferred for training the state-of-the-art UNMT model, the XLM model, when possible. Finally, we show that there is a trade-off between quality and quantity of the data used to train UNMT sys- tems. 1 Introduction Unsupervised Neural Machine Translation (UNMT) has recently become the dominant paradigm for unsupervised MT, with the advent of cross-lingual language model pretraining as used in the XLM model (Conneau and Lample, 2019). However, much of the existing research in UNMT assumes that the amount of data available for one language is roughly equivalent to the other. The WMT 2020 Unsupervised Machine Translation task is unique in that monolingual data is abundant for one language (German), with hundreds of millions of sentences available, and sparse for the other (Upper Sorbian), which only has around 750 thousand sentences available. With a wealth of data available on the German side, it is natural to ask: how can we best use this data? Viewing this under the lens of data selection, we break this broad question down into 3 concrete sub-questions, tailored for the unsupervised setting. They are as follows: • How can we determine the quality of training data? • What kinds of data selection are best for train- ing an XLM model? • Is quality or quantity more important when it comes to training data for UNMT? Section 2 describes the general setup pertain- ing to every experiment, including datasets, data processing steps, model architecture, and training details. In Section 3, we detail our individual ex- periments and their corresponding results. Finally, in Section 4, we make our conclusions and discuss paths for future work. 2 Setup For Upper Sorbian, we use the 3 monolingual datasets provided by the Sorbian Institute, the Witaj Sprachzentrum, and the web data from CIS, LMU. We also use the Upper Sorbian side of the parallel corpus from train.hsb-de.hsb.gz. For Ger- man, we use monolingual data from News Crawl and Common Crawl. For validation and testing, we use the data provided in devtest.tar.gz. All data is tokenized and truecased using the Moses toolkit (Koehn et al., 2007). For BPE seg- mentation (Sennrich et al., 2016), we apply a joint segmentation for both languages. …

Content truncated to 500 words