Improving Target-side Lexical Transfer in Multilingual Neural Machine Translation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3560–3566 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3560 Improving Target-side Lexical Transfer in Multilingual Neural Machine Translation Luyu Gao, Xinyi Wang, and Graham Neubig Language Technologies Institute, Carnegie Mellon University {luyug,xinyiw1,gneubig}@cs.cmu.edu Abstract To improve the performance of Neural Ma- chine Translation (NMT) for low-resource lan- guages (LRL), one effective strategy is to lever- age parallel data from a related high-resource language (HRL). However, multilingual data has been found more beneficial for NMT mod- els that translate from the LRL to a target language than the ones that translate into the LRLs. In this paper, we aim to improve the effectiveness of multilingual transfer for NMT models that translate into the LRL, by designing a better decoder word embedding. Extending upon a general-purpose multilin- gual encoding method Soft Decoupled Encod- ing (Wang et al., 2019), we propose DecSDE, an efficient character n-gram based embedding specifically designed for the NMT decoder. Our experiments show that DecSDE leads to consistent gains of up to 1.8 BLEU on transla- tion from English to four different languages.1 1 Introduction The performance of Neural Machine Transla- tion (NMT; Sutskever et al. (2014)) tends to de- grade on low-resource languages (LRL) due to a paucity of parallel data (Koehn and Knowles, 2017; Sennrich and Zhang, 2019). One effective strategy to improve translation in LRLs is through multi- lingual training using parallel data from related high-resource languages (HRL) (Zoph et al., 2016; Neubig and Hu, 2018). The assumption underlying cross-lingual transfer is that by sharing parameters between multiple languages the LRL can benefit from the extra training signal from data in other languages. One of the most popular strategies for multilingual training is to train a single NMT model that translates in many directions by simply append- ing a flag to each source sentence to indicate which 1Open-source code is available at https://github. com/luyug/DecSDE target language to translate into (Ha et al., 2016; Johnson et al., 2017). Many works focus on using multilingual training to improve many-to-one NMT models that translate from both an HRL and an LRL to a single target language (Zoph et al., 2016; Neubig and Hu, 2018; Gu et al., 2018). In this situation, sentences from the HRL-target corpus provide an extra training signal for the decoder language model, on top of cross-lingual transfer on the source side. When training an NMT model that translates into an LRL, however, multilingual data tends to lead to smaller improvements (Lakew et al., 2019; Arivazhagan et al., 2019; Aharoni et al., 2019). In this paper, we aim to improve the effective- ness of multilingual training for NMT models that translate into LRLs. Prior work has found vocabu- lary overlap to be an important indicator of whether data from other languages will be effective in im- proving NMT accuracy (Wang and Neubig, 2019; Lin et al., 2019). Therefore, we hypothesize that one of the main problems limiting multilingual transfer on the target side is …

Content truncated to 500 words