An Effective Approach to Unsupervised Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 194–203 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 194 An Effective Approach to Unsupervised Machine Translation Mikel Artetxe, Gorka Labaka, Eneko Agirre IXA NLP Group University of the Basque Country (UPV/EHU) {mikel.artetxe, gorka.labaka, e.agirre}@ehu.eus Abstract While machine translation has traditionally re- lied on large amounts of parallel corpora, a re- cent research line has managed to train both Neural Machine Translation (NMT) and Sta- tistical Machine Translation (SMT) systems using monolingual corpora only. In this pa- per, we identify and address several deficien- cies of existing unsupervised SMT approaches by exploiting subword information, develop- ing a theoretically well founded unsupervised tuning method, and incorporating a joint re- finement procedure. Moreover, we use our im- proved SMT system to initialize a dual NMT model, which is further fine-tuned through on- the-fly back-translation. Together, we obtain large improvements over the previous state- of-the-art in unsupervised machine transla- tion. For instance, we get 22.5 BLEU points in English-to-German WMT 2014, 5.5 points more than the previous best unsupervised sys- tem, and 0.5 points more than the (supervised) shared task winner back in 2014. 1 Introduction The recent advent of neural sequence-to-sequence modeling has resulted in significant progress in the field of machine translation, with large improve- ments in standard benchmarks (Vaswani et al., 2017; Edunov et al., 2018) and the first solid claims of human parity in certain settings (Has- san et al., 2018). Unfortunately, these systems rely on large amounts of parallel corpora, which are only available for a few combinations of major languages like English, German and French. Aiming to remove this dependency on paral- lel data, a recent research line has managed to train unsupervised machine translation systems using monolingual corpora only. The first such systems were based on Neural Machine Transla- tion (NMT), and combined denoising autoencod- ing and back-translation to train a dual model ini- tialized with cross-lingual embeddings (Artetxe et al., 2018c; Lample et al., 2018a). Neverthe- less, these early systems were later superseded by Statistical Machine Translation (SMT) based approaches, which induced an initial phrase-table through cross-lingual embedding mappings, com- bined it with an n-gram language model, and fur- ther improved the system through iterative back- translation (Lample et al., 2018b; Artetxe et al., 2018b). In this paper, we develop a more principled ap- proach to unsupervised SMT, addressing several deficiencies of previous systems by incorporat- ing subword information, applying a theoretically well founded unsupervised tuning method, and de- veloping a joint refinement procedure. In addition to that, we use our improved SMT approach to ini- tialize an unsupervised NMT system, which is fur- ther improved through on-the-fly back-translation. Our experiments on WMT 2014/2016 French- English and German-English show the effective- ness of our approach, as our proposed system out- performs the previous state-of-the-art in unsuper- vised machine translation by 5-7 BLEU points in all these datasets and translation directions. Our system also outperforms the supervised WMT …

Content truncated to 500 words