Being Generous with Sub-Words towards Small NMT Children

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3650–3656 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3650 Being Generous with Sub-Words towards Small NMT Children Arne Defauw, Tom Vanallemeersch, Koen Van Winckel, Sara Szoc, Joachim Van den Bogaert CrossLang Kerkstraat 106, 9050 Gentbrugge, Belgium {firstname.lastname}@crosslang.com Abstract In the context of under-resourced neural machine translation (NMT), transfer learning from an NMT model trained on a high resource language pair, or from a multilingual NMT (M-NMT) model, has been shown to boost performance to a large extent. In this paper, we focus on so-called cold start transfer learning from an M-NMT model, which means that the parent model is not trained on any of the child data. Such a set-up enables quick adaptation of M-NMT models to new languages. We investigate the effectiveness of cold start transfer learning from a many-to-many M-NMT model to an under-resourced child. We show that sufficiently large sub-word vocabularies should be used for transfer learning to be effective in such a scenario. When adopting relatively large sub-word vocabularies we observe increases in performance thanks to transfer learning from a parent M-NMT model, both when translating to and from the under-resourced language. Our proposed approach involving dynamic vocabularies is both practical and effective. We report results on two under-resourced language pairs, i.e. Icelandic-English and Irish-English. Keywords: Neural Machine Translation, Multilingual Models, Transfer Learning 1. Introduction In recent years, the performance of machine translation systems has been improving significantly thanks to the shift from statistical to neural machine translation (Bahdanau et al., 2014; Sutskever et al., 2014; Vaswani et al., 2017). However, for under-resourced language pairs, the performance of MT systems can still be disappointing, as pointed out for instance by Koehn et al. (2017). To improve MT quality for under-resourced languages, a potential strategy consists of involving other language pairs during MT training than the one under scrutiny. In such a multilingual scenario, different language pairs, often belonging to closely related languages, are combined into a single translation model (Ha et al., 2016; Johnson et al., 2017). Recently, first efforts towards training massively multilingual (or ‘universal’) models have been undertaken (Aharoni et al., 2019; Arivazhagan et al., 2019), by extending systems towards dozens or even hundreds of language directions, which obviously introduces various bottlenecks. An alternative, although related, strategy to improve NMT performance in an under-resourced scenario is transfer learning (Zoph et al., 2016). It is a concept that has been applied to various subfields of AI, such as computer vision and NLP. In NMT, transfer learning consists of training a (multilingual) parent MT model on one or more language pairs, in order to initialize a child MT model that continues training with data from a (different) set of (under-resourced) language pairs. Transfer learning to an under-resourced language pair, either from a parent model trained on a high resource language pair (Zoph et al. 2016; Kocmi and Bojar, 2018; Lakew et al. 2018) or from a large …

Content truncated to 500 words