Unsupervised Translation of German–Lower Sorbian: Exploring Training and Novel Transfer Methods on a Low-Resource Language

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 982–988 November 10–11, 2021. ©2021 Association for Computational Linguistics 982 Unsupervised Translation of German–Lower Sorbian: Exploring Training and Novel Transfer Methods on a Low-Resource Language Lukas Edman Ahmet Üstün Antonio Toral Gertjan van Noord Center for Language and Cognition University of Groningen {j.l.edman, a.ustun, a.toral.ruiz, g.j.m.van.noord}@rug.nl Abstract This paper describes the methods behind the systems submitted by the University of Gronin- gen for the WMT 2021 Unsupervised Ma- chine Translation task for German–Lower Sor- bian (DE–DSB): a high-resource language to a low-resource one. Our system uses a trans- former encoder-decoder architecture in which we make three changes to the standard train- ing procedure. First, our training focuses on two languages at a time, contrasting with a wealth of research on multilingual systems. Second, we introduce a novel method for initializing the vocabulary of an unseen lan- guage, achieving improvements of 3.2 BLEU for DE→DSB and 4.0 BLEU for DSB→DE. Lastly, we experiment with the order in which offline and online back-translation are used to train an unsupervised system, finding that us- ing online back-translation first works better for DE→DSB by 2.76 BLEU. Our submis- sions ranked first (tied with another team) for DSB→DE and third for DE→DSB. 1 Introduction Unsupervised Neural Machine Translation (UNMT) has become increasingly useful in the field of MT, given that monolingual data is easier to gather compared to bilingual (or parallel) data. Such is especially the case for low-resource languages, which constitute the majority of languages in the world. The WMT 2021 Unsupervised MT Task focuses on one such low-resource language: Lower Sorbian (DSB). The task is to translate between German (DE), a high-resource language, and Lower Sor- bian, which is a very low resource language with roughly 150 thousand sentences of monolingual data available for the task at hand. The unsuper- vised task from prior years of WMT focused on German–Czech and German–Upper Sorbian trans- lation. Unique to this year however is the relatively small amount of monolingual data available for DSB, compared to last year in which roughly 750 thousand sentences of Upper Sorbian were avail- able. This makes it increasingly difficult to rely on the ubiquitous state-of-the-art UNMT methods (Lample and Conneau, 2019; Song et al., 2019; Liu et al., 2020), as they typically rely on a large amount of monolingual data available for both lan- guages. To alleviate the difficulty that comes with the lack of monolingual data for DSB, this year’s WMT task allows for the use of monolingual and parallel data outside of DE–DSB. Specifically, all Upper Sorbian (HSB) data from WMT20 and all paral- lel data for German (DE) from WMT and OPUS (Tiedemann and Nygaard, 2004) are made avail- able to use. Additionally, as auxiliary languages related to DSB, monolingual data for Czech (CS) and Polish (PL) is also provided. Given the success of language transfer via multi- lingual models such as mBART (Liu et al., 2020), this fundamentally changes this year’s unsuper- vised task from a …

Content truncated to 500 words