When Does Unsupervised Machine Translation Work?

Proceedings of the 5th Conference on Machine Translation (WMT), pages 571–583 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 571 When Does Unsupervised Machine Translation Work? Kelly Marchisio and Kevin Duh and Philipp Koehn Johns Hopkins University kmarc@jhu.edu, kevinduh@cs.jhu.edu, phi@jhu.edu Abstract Despite the reported success of unsupervised machine translation, the field has yet to ex- amine the conditions under which the meth- ods succeed and fail. We conduct an exten- sive empirical evaluation using dissimilar lan- guage pairs, dissimilar domains, and diverse datasets. We find that performance rapidly deteriorates when source and target corpora are from different domains, and that stochas- ticity during embedding training can dramati- cally affect downstream results. We advocate for extensive empirical evaluation of unsuper- vised MT systems to highlight failure points and encourage continued research on the most promising paradigms. Towards this goal, we release our preprocessed dataset to stress-test systems under multiple data conditions. 1 Introduction Machine translation (MT) has progressed rapidly since the advent of neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Bah- danau et al., 2015; Sutskever et al., 2014) and is better than ever for languages for which ample high-quality bitext exists. Conversely, MT for low- resource languages remains a great challenge due to a dearth of parallel training corpora and poor quality bitext from esoteric domains. To address this, several authors have proposed unsupervised MT techniques, which rely only on monolingual text for training (e.g., Ravi and Knight, 2011; Yang et al., 2018; Artetxe et al., 2018c; Hoshen and Wolf, 2018; Lample et al., 2018a,b; Artetxe et al., 2018b, 2019). Recent unsupervised MT results appear promis- ing, but they primarily report results for the high- resource languages for which traditional MT al- ready works well. The limits of these methods are so far under-explored. For unsupervised MT to be a viable path for low-resource machine transla- tion, the field must determine (1) if it works out- side highly-controlled environments, and (2) how to effectively evaluate newly-proposed training paradigms to pursue those which are promising for real-world low-resource scenarios. Unsupervised MT methods must work (1) on different scripts and between dissimilar languages, (2) with im- perfect domain alignment between source and target corpora, (3) with a domain mismatch be- tween training data and the test set, and (4) on the low-quality data of real low-resource languages. These factors reflect the real-life challenges of low- resource translation. Our main contribution is an extensive analysis of unsupervised MT with regards to factors (1)-(3) above.1 We find that (a) translation performance rapidly deteriorates when source and target corpora are from different domains, (b) stochasticity during word embedding training can dramatically affect downstream bilingual lexicon induction (BLI) and translation performance, and (c) like in the bilin- gual lexicon induction literature, unsupervised MT performance declines when source and target lan- guages are dissimilar. While (4) is not the focus of this paper, we do observe very low performance on an authentic low-resource language pair, corrobo- rating previous studies (Guzm´an et al., 2019). Finally, as …

Content truncated to 500 words