CUNI Systems for the Unsupervised and Very Low Resource Translation Task in WMT20

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1123–1128 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1123 CUNI Systems for the Unsupervised and Very Low Resource Translation Task in WMT20 Ivana Kvapil´ıkov´a Tom Kocmi Ondˇrej Bojar Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Malostransk´e n´amˇest´ı 25, 118 00 Prague, Czech Republic <surname>@ufal.mff.cuni.cz Abstract This paper presents a description of CUNI sys- tems submitted to the WMT20 task on unsu- pervised and very low-resource supervised ma- chine translation between German and Upper Sorbian. We experimented with training on synthetic data and pre-training on a related language pair. In the fully unsupervised sce- nario, we achieved 25.5 and 23.7 BLEU trans- lating from and into Upper Sorbian, respec- tively. Our low-resource systems relied on transfer learning from German–Czech parallel data and achieved 57.4 BLEU and 56.1 BLEU, which is an improvement of 10 BLEU points over the baseline trained only on the available small German–Upper Sorbian parallel corpus. 1 Introduction An extensive area of the machine translation (MT) research focuses on training translation systems without large parallel data resources (Artetxe et al., 2018b,a, 2019; Lample et al., 2018a,b). The WMT20 translation competition presents a sepa- rate task on unsupervised and very low-resource supervised MT. The organizers prepared a shared task to explore machine translation on a real-life example of a low- resource language pair of German (de) and Up- per Sorbian (hsb). There are around 60k authen- tic parallel sentences available for this language pair which is not sufficient to train a high-quality MT system in a standard supervised way, and calls for unsupervised pre-training (Conneau and Lam- ple, 2019), data augmentation by synthetically pro- duced sentences (Sennrich et al., 2016a) or transfer learning from different language pairs (Zoph et al., 2016a; Kocmi and Bojar, 2018). The WMT20 shared task is divided into two tracks. In the unsupervised track, the participants are only allowed to use monolingual German and Upper Sorbian corpora to train their systems; the low-resource track permits the usage of auxiliary parallel corpora in other languages as well as a small parallel corpus in German–Upper Sorbian. We participate in both tracks in both translation directions. Section 2 describes our participation in the unsupervised track and section 3 describes our systems from the low-resource track. Section 4 introduces transfer learning via Czech (cs) into our low-resource system. We conclude the paper in section 5. 2 Unsupervised MT Unsupervised machine translation is the task of learning to translate without any parallel data re- sources at training time. Both neural and phrase- based systems were proposed to solve the task (Lample et al., 2018b). In this work, we train several neural systems and compare the effects of different training approaches. 2.1 Methodology The key concepts of unsupervised NMT include a shared encoder, shared vocabulary and model initialization (pre-training). The training relies only on monolingual corpora and switches between de-noising, where the model learns to reconstruct corrupted sentences, and online back-translation, where the …

Content truncated to 500 words