On Romanization for Model Transfer Between Scripts in Neural Machine Translation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2461–2469 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2461 On Romanization for Model Transfer Between Scripts in Neural Machine Translation Chantal Amrhein1 and Rico Sennrich1,2 1Department of Computational Linguistics, University of Zurich 2School of Informatics, University of Edinburgh {amrhein,sennrich}@cl.uzh.ch Abstract Transfer learning is a popular strategy to im- prove the quality of low-resource machine translation. For an optimal transfer of the embedding layer, the child and parent model should share a substantial part of the vocab- ulary. This is not the case when transferring to languages with a different script. We ex- plore the benefit of romanization in this sce- nario. Our results show that romanization en- tails information loss and is thus not always superior to simpler vocabulary transfer meth- ods, but can improve the transfer between re- lated languages with different scripts. We com- pare two romanization tools and find that they exhibit different degrees of information loss, which affects translation quality. Finally, we extend romanization to the target side, show- ing that this can be a successful strategy when coupled with a simple deromanization model. 1 Introduction Neural Machine Translation (NMT) has opened up new opportunities in transfer learning from high- resource to low-resource language pairs (Zoph et al., 2016; Kocmi and Bojar, 2018; Lakew et al., 2018). While transfer learning has shown great promise, the transfer between languages with dif- ferent scripts brings additional challenges. For a successful transfer of the embedding layer, both the parent and the child model should use the same or a partially overlapping vocabulary (Aji et al., 2020). It is common to merge the two vocabular- ies by aligning identical subwords and randomly assigning the remaining subwords from the child vocabulary to positions in the parent vocabulary (Lakew et al., 2018, 2019; Kocmi and Bojar, 2020). This works well for transfer between languages that use the same script, but if the child language is written in an unseen script, most vocabulary po- sitions are replaced by random subwords. This significantly reduces the transfer from the embed- ding layer. Gheini and May (2019) argue that ro- manization can improve transfer to languages with unseen scripts. However, romanization can also introduce information loss that might hurt transla- tion quality. In our work, we study the usefulness of romanization for transfer from many-to-many multilingual MT models to low-resource languages with different scripts. Our contributions are the following: - We show that romanized MT is not generally optimal, but can improve transfer between re- lated languages that use different scripts. - We study information loss from different ro- manization tools and its effect on MT quality. - We demonstrate that romanization on the tar- get side can also be effective when combined with a learned deromanization model. 2 Related Work Initial work on transfer learning for NMT has as- sumed that the child language is known in advance and that the parent and child model can use a shared vocabulary (Nguyen and …

Content truncated to 500 words