Transfer Learning with Shallow Decoders: BSC at WMT2021’s Multilingual Low-Resource Translation for Indo-European Languages Shared Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 362–367 November 10–11, 2021. ©2021 Association for Computational Linguistics 362 Transfer Learning with Shallow Decoders: BSC at WMT2021’s Multilingual Low-Resource Translation for Indo-European Languages Shared Task Ksenia Kharitonova, Ona de Gibert Bonet, Jordi Armengol-Estapé, Mar Rodríguez i Alvarez, Maite Melero Text Mining Unit, Barcelona Supercomputing Center {ksenia.kharitonova,ona.degibert,jordi.armengol, mar.rodriguez1,maite.melero}@bsc.es Abstract This paper describes the participation of the BSC team in the WMT2021’s Multilingual Low-Resource Translation for Indo-European Languages Shared Task. The system aims to solve the Subtask 2: Wikipedia cultural her- itage articles, which involves translation in four Romance languages: Catalan, Italian, Oc- citan and Romanian. The submitted system is a multilingual semi- supervised machine translation model. is based on a pre-trained language model,It namely XLM-RoBERTa, that is later fine- tuned with parallel data obtained mostly from OPUS. Unlike other works, we only use XLM to initialize the encoder and randomly initial- ize a shallow decoder. The reported results are robust and perform well for all tested lan- guages. 1 Introduction We present the work carried out by the BSC Team in the context of WMT2021’s first edition of the Multilingual Low-Resource Translation Shared Task. The tasks addresses the issue of multi- linguality in machine translation (MT) for low- resource languages, focusing on two language fam- ilies: North Germanic and Romance. We take part in the Subtask 2, which involves translation in four Romance languages: Catalan, Italian, Occitan and Romanian. 2 Background Machine translation for low-resource languages is characterised by the lack of sufficient parallel data of a given language pair, either because the com- bination is infrequent or because the languages in- volved are themselves low-resource. Several works have attempted to overcome this pitfall, using dif- ferent techniques. A common solution is to employ back-translation (Sennrich et al., 2016), while other research focuses on using other languages as piv- ots to compensate for the lack of data (Firat et al., 2016; Zoph et al., 2016). Artetxe et al. (2018); Lample et al. (2018) make use of monolingual data only. Our approach is based on multilinguality. Pre- vious works such as Vergés Boncompte and Ruiz Costa-Jussà (2020); Tubay and Costa-Jussa (2018) have shown that the use of multilingual MT is bene- ficial, as it generalizes better by sharing parameters among all the languages involved, especially if the languages belong to the same linguistic family. At the same time, training of multilingual MT models from scratch usually requires large parallel corpora and may not be feasible in a low-resource and zero- resource translation scenarios. Pre-training of large language models from scratch on monolingual data and then fine-tuning them for the specific downstream tasks, has proved to be an extremely successful approach for many natural language processing problems. Cross- lingual language models such as XLM and XLM- RoBERTa (Conneau and Lample, 2019; Conneau et al., 2020), that combine unsupervised (mono- lingual data) and supervised (parallel data) train- ing objectives, perform especially well both on cross-lingual NLU tasks and in machine translation. The idea …

Content truncated to 500 words