Low-Resource Translation as Language Modeling

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1079–1083 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1079 Low-Resource Translation as Language Modeling Tucker Berckmann and Berkan Hiziroglu Department of Computer Science Brown University tucker berckmann@brown.edu berkan hiziroglu@alumni.brown.edu Abstract We present our submission to the very low resource supervised machine translation task at the Fifth Conference on Machine Transla- tion. The goal of this task is to create a system which translates between German and the low- resource language Upper Sorbian. We use a decoder-only transformer architecture and for- mulate the translation task as language model- ing. To address the low-resource aspect of the problem, we pretrain over a similar language parallel corpus. Then, we employ an interme- diate back-translation step before fine-tuning. Finally, we present an analysis of the system’s performance. 1 Introduction This work describes our system for translating in both directions between German (DE) and the low- resource language Upper Sorbian (HSB). German is a widespread language with tens of millions of speakers; Upper Sorbian is a West Slavic language spoken in Germany, and it is recognized as an en- dangered language by UNESCO (Moseley, 2010). This system constitutes our submission to the shared task on very-low-resource supervised ma- chine translation at WMT20.1 The ultimate goal of the task is to translate a blind test set from Up- per Sorbian into German and vice versa. The task is constrained, meaning that all data sets used for training are selected from a set of corpora provided by the organizers. Our primary contribution is our application of a decoder-only language-modeling architecture to a low-resource translation task, which to our knowl- edge is not well-investigated. In Sections 2 and 3, we discuss related work and our system itself. Sections 4, 5, and 6 describe our architecture. Sections 7 and 8 contain our results and analysis. 1http://www.statmt.org/wmt20 2 Related Work Current approaches to machine translation include neural networks based on encoder-decoder trans- formers (Vaswani et al., 2017) and sequence-to- sequence models using recurrent networks (Chen et al., 2018). In both of these methods, the system learns how to produce an intermediate represen- tation of a text sequence as a basis for the output translation. Language-neutral representations have been explored more deeply in the context of mBert (Libovick`y et al., 2019). In the case of low-resource languages, where there is an absence of adequately sized parallel cor- pora, recent techniques focus on transfer learning (Zoph et al., 2016), relying on monolingual cor- pora (Lample et al., 2018), enriching the input to the system (Irvine and Callison-Burch, 2013), or expanding it through back-translation (Sennrich et al., 2016). Techniques related to back-translation include pseudo-labeling and self-labeling. Pseudo-labeling uses partially accurate data for training (Ratner et al., 2017) generated from knowledge bases, heuristic functions and crowdsourcing. Self- labeling is an area that lies between self-supervised learning and pseudo-labeling (Caron et al., 2018; Asano et al., 2020). The model is used to predict labels for an unlabeled dataset and then is trained on …

Content truncated to 500 words