Universal Neural Machine Translation for Extremely Low Resource Languages

Proceedings of NAACL-HLT 2018, pages 344–354 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Universal Neural Machine Translation for Extremely Low Resource Languages Jiatao Gu†∗ Hany Hassan‡ Jacob Devlin□∗ Victor O.K. Li† †The University of Hong Kong ‡Microsoft Research {jiataogu, vli}@eee.hku.hk hanyh@microsoft.com □Google Research jacobdevlin@google.com Abstract In this paper, we propose a new univer- sal machine translation approach focusing on languages with a limited amount of paral- lel data. Our proposed approach utilizes a transfer-learning approach to share lexical and sentence level representations across multi- ple source languages into one target language. The lexical part is shared through a Univer- sal Lexical Representation to support multi- lingual word-level sharing. The sentence- level sharing is represented by a model of experts from all source languages that share the source encoders with all other languages. This enables the low-resource language to uti- lize the lexical and sentence representations of the higher resource languages. Our approach is able to achieve 23 BLEU on Romanian- English WMT2016 using a tiny parallel cor- pus of 6k sentences, compared to the 18 BLEU of strong baseline system which uses multi- lingual training and back-translation. Further- more, we show that the proposed approach can achieve almost 20 BLEU on the same dataset through fine-tuning a pre-trained multi-lingual system in a zero-shot setting. 1 Introduction Neural Machine Translation (NMT) (Bahdanau et al., 2015) has achieved remarkable translation quality in various on-line large-scale systems (Wu et al., 2016; Devlin, 2017) as well as achieving state-of-the-art results on Chinese-English transla- tion (Hassan et al., 2018). With such large sys- tems, NMT showed that it can scale up to immense amounts of parallel data in the order of tens of millions of sentences. However, such data is not widely available for all language pairs and domains. ∗This work was done while the authors at Microsoft. In this paper, we propose a novel universal multi- lingual NMT approach focusing mainly on low resource languages to overcome the limitations of NMT and leverage the capabilities of multi-lingual NMT in such scenarios. Our approach utilizes multi-lingual neural trans- lation system to share lexical and sentence level representations across multiple source languages into one target language. In this setup, some of the source languages may be of extremely limited or even zero data. The lexical sharing is represented by a universal word-level representation where var- ious words from all source languages share the same underlaying representation. The sharing mod- ule utilizes monolingual embeddings along with seed parallel data from all languages to build the universal representation. The sentence-level shar- ing is represented by a model of language experts which enables low-resource languages to utilize the sentence representation of the higher resource languages. This allows the system to translate from any language even with tiny amount of parallel resources. We evaluate the proposed approach on 3 differ- ent languages with tiny or even zero parallel data. We show that for the simulated “zero-resource" settings, our model can consistently outperform a strong …

Content truncated to 500 words