NRC Systems for Low Resource German-Upper Sorbian Machine Translation 2020: Transfer Learning with Lexical Modifications

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1112–1122 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1112 NRC Systems for Low Resource German-Upper Sorbian Machine Translation 2020: Transfer Learning with Lexical Modifications Rebecca Knowles and Samuel Larkin and Darlene Stewart and Patrick Littell National Research Council Canada {Rebecca.Knowles, Samuel.Larkin, Darlene.Stewart, Patrick.Littell}@nrc-cnrc.gc.ca Abstract We describe the National Research Council of Canada (NRC) neural machine translation systems for the German–Upper Sorbian super- vised track of the 2020 shared task on Unsu- pervised MT and Very Low Resource Super- vised MT. Our models are ensembles of Trans- former models, built using combinations of BPE-dropout, lexical modifications, and back- translation. 1 Introduction We describe the National Research Council of Canada (NRC) neural machine translation systems for the shared task on Unsupervised MT and Very Low Resource Supervised MT. We participated in the supervised track of the low resource task, building Upper Sorbian–German neural machine translation (NMT) systems in both translation direc- tions. Upper Sorbian is a minority language spoken in Germany. We built baseline systems (standard Transformer (Vaswani et al., 2017) with a byte-pair encoding vocabulary (BPE; Sennrich et al., 2016b)) trained on all available parallel data (60,000 lines), which resulted in unusually high BLEU scores for a language pair with such limited data. In order to improve upon this baseline, we used transfer learning with modifications to the training lexicon. We did this in two ways: by experiment- ing with the application of BPE-dropout (Provilkov et al., 2020) to the transfer learning setting (Sec- tion 2.3), and by modifying Czech data used for training parent systems with word and character replacements in order to make it more “Upper Sorbian-like” (Section 2.4). Our final systems were ensembles of systems built using transfer learning and these two ap- proaches to lexicon modification, along with it- erative backtranslation. 2 Approaches 2.1 General System Notes In both translation directions, our final systems con- sist of ensembles of multiple systems, built using transfer learning (Section 2.2), BPE-Dropout (Sec- tion 2.3), alternative preprocessing of Czech data (Section 2.4), and backtranslation (Section 2.5). We describe these approaches and related work in the following sections, providing implementation details for reproducibility in Sections 3, 4 and 5. 2.2 Transfer Learning Zoph et al. (2016) proposed a transfer learning approach for neural machine translation, using lan- guage pairs with larger amounts of data to pre-train a parent system, followed by finetuning a child sys- tem on the language pair of interest. Nguyen and Chiang (2017) expand on that, showing improved performance using BPE and shared vocabularies between the parent and child. We follow this ap- proach: we build disjoint source and target BPE models and vocabularies, with one vocabulary for German (DE) and one for the combination of Czech (CS) and Upper Sorbian (HSB); see Section 4. We chose to use Czech–German data as the par- ent language pair due to the task suggestions, rela- tive abundance of data, and the close relationship between Czech and Upper Sorbian (cf. …

Content truncated to 500 words