Low-Resource Unsupervised NMT: Diagnosing the Problem and Providing a Linguistically Motivated Solution

Low-Resource Unsupervised NMT: Diagnosing the Problem and Providing a Linguistically Motivated Solution Lukas Edman, Antonio Toral, Gertjan van Noord Center for Language and Cognition University of Groningen {j.l.edman, a.toral.ruiz, g.j.m.van.noord}@rug.nl Abstract Unsupervised Machine Translation has been advancing our ability to translate without parallel data, but state-of-the-art methods assume an abundance of mono- lingual data. This paper investigates the scenario where monolingual data is lim- ited as well, finding that current unsuper- vised methods suffer in performance un- der this stricter setting. We find that the performance loss originates from the poor quality of the pretrained monolingual em- beddings, and we propose using linguis- tic information in the embedding train- ing scheme. To support this, we look at two linguistic features that may help im- prove alignment quality: dependency in- formation and sub-word information. Us- ing dependency-based embeddings results in a complementary word representation which offers a boost in performance of around 1.5 BLEU points compared to stan- dard WORD2VEC when monolingual data is limited to 1 million sentences per lan- guage. We also find that the inclusion of sub-word information is crucial to improv- ing the quality of the embeddings. 1 Introduction Machine Translation (MT) is a rapidly advancing field of Natural Language Processing, where there is an ever-increasing number of claims of MT sys- tems reaching human parity (Hassan et al., 2018; Barrault et al., 2019). However, most of the fo- cus has been on MT systems under the assumption c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. that there is a large amount of parallel data avail- able, which is only the case for a select number of language pairs. Recently, there have been approaches that do away with this assumption, requiring only mono- lingual data, with the first methods based solely around neural MT (NMT), using aligned pre- trained embeddings to bootstrap the translation process, and refining the translation with a neural model via denoising and back-translation (Artetxe et al., 2017b; Lample et al., 2017). More re- cently, statistical MT (SMT) approaches as well as hybrid approaches, combining SMT and NMT, have proven more successful (Lample et al., 2018; Artetxe et al., 2019). While the unsupervised approaches so far have done away with the assumption of parallel data, they still assume an abundance of monolingual data for the two languages, typically assuming at least 10 million sentences per language. This amount of data is not available for every language, notably languages without much of a digital pres- ence. For example, Fulah is a language spoken in West and Central Africa by over 20 million peo- ple, however there is a scarce amount of data freely available online. This motivates a new paradigm in unsupervised MT: Low-Resource Unsupervised MT (LRUMT). In this paper, we investigate the reasons why current unsupervised NMT methods fail in the low-resource setting, addressing the source of the issue, and we propose a potential solution to make unsupervised NMT more robust to …

Content truncated to 500 words