Consistency by Agreement in Zero-Shot Neural Machine Translation

Proceedings of NAACL-HLT 2019, pages 1184–1197 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1184 Consistency by Agreement in Zero-shot Neural Machine Translation Maruan Al-Shedivat∗ Carnegie Mellon University Pittsburgh, PA 15213 alshedivat@cs.cmu.edu Ankur P. Parikh Google AI Language New York, NY 10011 aparikh@google.com Abstract Generalization and reliability of multilingual translation often highly depend on the amount of available parallel data for each language pair of interest. In this paper, we focus on zero-shot generalization—a challenging setup that tests models on translation directions they have not been optimized for at training time. To solve the problem, we (i) reformulate mul- tilingual translation as probabilistic inference, (ii) define the notion of zero-shot consistency and show why standard training often results in models unsuitable for zero-shot tasks, and (iii) introduce a consistent agreement-based training method that encourages the model to produce equivalent translations of parallel sen- tences in auxiliary languages. We test our mul- tilingual NMT models on multiple public zero- shot translation benchmarks (IWSLT17, UN corpus, Europarl) and show that agreement- based learning often results in 2-3 BLEU zero- shot improvement over strong baselines with- out any loss in performance on supervised translation directions. 1 Introduction Machine translation (MT) has made remarkable advances with the advent of deep learning ap- proaches (Bojar et al., 2016; Wu et al., 2016; Crego et al., 2016; Junczys-Dowmunt et al., 2016). The progress was largely driven by the encoder-decoder framework (Sutskever et al., 2014; Cho et al., 2014) and typically supplemented with an attention mech- anism (Bahdanau et al., 2014; Luong et al., 2015b). Compared to the traditional phrase-based sys- tems (Koehn, 2009), neural machine translation (NMT) requires large amounts of data in order to reach high performance (Koehn and Knowles, 2017). Using NMT in a multilingual setting exacer- bates the problem by the fact that given k languages ∗Work done at Google. En De En Fr parallel data FrDe FrEn Loss( , ) agreement on Fr DeEn DeFr Loss( , ) agreement on De Figure 1: Agreement-based training of a multilingual NMT. At training time, given English-French (En ↔ Fr) and English-German (En ↔ De) parallel sentences, the model not only is trained to translate between the pair but also to agree on translations into a third language. translating between all pairs would require O(k2) parallel training corpora (and O(k2) models). In an effort to address the problem, different multilingual NMT approaches have been proposed recently. Luong et al. (2015a); Firat et al. (2016a) proposed to use O(k) encoders/decoders that are then intermixed to translate between language pairs. Johnson et al. (2016) proposed to use a single model and prepend special symbols to the source text to indicate the target language, which has later been extended to other text preprocessing ap- proaches (Ha et al., 2017) as well as language- conditional parameter generation for encoders and decoders of a single model (Platanios et al., 2018). Johnson et al. (2016) also show that a single multilingual system could potentially enable zero- …

Content truncated to 500 words