Multilingual Unsupervised NMT using Shared Encoder and Language-Specific Decoders

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3083–3089 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3083 Multilingual Unsupervised NMT using Shared Encoder and Language-Specific Decoders Sukanta Sen, Kamal Kumar Gupta, Asif Ekbal, Pushpak Bhattacharyya Department of Computer Science and Engineering Indian Institute of Technology Patna Patna, India {sukanta.pcs15,kamal.pcs17,asif,pb}@iitp.ac.in Abstract In this paper, we propose a multilingual un- supervised NMT scheme which jointly trains multiple languages with a shared encoder and multiple decoders. Our approach is based on denoising autoencoding of each language and back-translating between English and multi- ple non-English languages. This results in a universal encoder which can encode any lan- guage participating in training into an inter- lingual representation, and language-specific decoders. Our experiments using only mono- lingual corpora show that multilingual unsu- pervised model performs better than the sepa- rately trained bilingual models achieving im- provement of up to 1.48 BLEU points on WMT test sets. We also observe that even if we do not train the network for all possible translation directions, the network is still able to translate in a many-to-many fashion lever- aging encoder’s ability to generate interlingual representation. 1 Introduction Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015) has be- come a dominant paradigm for machine transla- tion achieving state-of-the-art results on publicly available benchmark datasets. An effective NMT system requires supervision of a huge amount of high-quality parallel data which is not easily avail- able for many language pairs. In absence of such huge amount of parallel data, NMT systems tend to perform poorly (Koehn and Knowles, 2017). However, NMT without using any parallel data such as bilingual translations, bilingual dictionary or comparable translations, has recently become reality and opened up exciting opportunities for future research (Lample et al., 2018; Artetxe et al., 2018; Yang et al., 2018). It completely eliminates the need of any kind of parallel data and depends heavily on cross-lingual embeddings and iterative back-translations (Sennrich et al., 2016) between the source and target language using monolingual corpora. On the architectural point of view, the approaches combine one encoder and one (Lam- ple et al., 2018) or two (Artetxe et al., 2018) de- coders. In supervised NMT settings, combining multiple languages to jointly train an NMT system has been found to be successful in improving the performance (Dong et al., 2015; Firat et al., 2016; Johnson et al., 2017). However, to the best of our knowledge, this is the very first attempt which aims at combining multiple languages in an unsu- pervised NMT training. To translate between many languages using bilingual version of unsupervised NMT, we re- quire an encoder and one (Lample et al., 2018) or two (Artetxe et al., 2018) decoders for each pair of languages. However, we may not need separate decoders depending on the source language. We can train source-independent, target-specific de- coders, wherein each decoder will take an interme- diate …

Content truncated to 500 words