Findings of the WMT 2021 Shared Task on Efficient Translation

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 117–122 November 10–11, 2021. ©2021 Association for Computational Linguistics 117 The TALP-UPC Participation in WMT21 News Translation Task: an mBART-based NMT Approach Carlos Escolano1, Ioannis Tsiamas1, Christine Basta1 2, Javier Ferrando1, Marta R. Costa-jussà1, José A. R. Fonollosa1 1 TALP Research Center, Universitat Politècnica de Catalunya, Barcelona 2 Institute of Graduate Studies and Research, Alexandria University, Egypt {carlos.escolano,ioannis.tsiamas,christine.basta javier.ferrando.monsonis,marta.ruiz,jose.fonollosa}@upc.edu Abstract This paper describes the submission to the WMT 2021 news translation shared task by the UPC Machine Translation group. The goal of the task is to translate German to French (De- Fr) and French to German (Fr-De). Our sub- mission focuses on fine-tuning a pre-trained model to take advantage of monolingual data. We fine-tune mBART50 using the filtered data, and additionally, we train a Transformer model on the same data from scratch. In the exper- iments, we show that fine-tuning mBART50 results in 31.69 BLEU for De-Fr and 23.63 BLEU for Fr-De, which increases 2.71 and 1.90 BLEU accordingly, as compared to the model we train from scratch. Our final sub- mission is an ensemble of these two models, further increasing 0.3 BLEU for Fr-De. 1 Introduction Monolingual data is usually more abundant than parallel data as it does not need any human pro- cessing. Neural Machine Translation (NMT) has focused traditionally on parallel data between lan- guages and monolingual data as back-translation (Sennrich et al., 2016). This method consists of translating a monolingual corpus with an NMT sys- tem and training the model using the synthetic data. Alternatively, in recent years, pre-trained models have been proposed using monolingual data as a pre-training step with self-supervised learning, be- fore performing task-specific fine-tuning. An exam- ple of this approach is BERT (Devlin et al., 2019), which is a Transformer model (Vaswani et al., 2017), pre-trained on masked language modeling and next sentences prediction on a large unlabeled corpus. While BERT is used primarily for classifi- cation tasks, BART (Lewis et al., 2020) has been proposed for sequence-to-sequence tasks. BART is a Transformer encoder-decoder, pre-trained as a Denoising Autoencoder (DAE) on monolingual unlabeled text. Since BART is pre-trained on mono- lingual data, an additional encoder should be intro- duced during fine-tuning to obtain a bilingual NMT system. mBART overcomes this restriction by be- ing pre-trained on multilingual denoising. mBART (Liu et al., 2020; Tang et al., 2020) is liable to fine- tuning on several translation directions in order to obtain a multilingual NMT system. Our participation to the news translation task at WMT focuses on translating between German (De) and French (Fr) in both directions, De-Fr and Fr- De. To accomplish this, we employ a pre-trained mBART model, and more specifically mBART50 (Tang et al., 2020), which is pre-trained with 50 languages. We fine-tune the mBART50 on both translation directions to obtain a single multilingual model for the task. To measure the importance of the pre-training step, we additionally train a Trans- former with the same architecture and hyperparam- eters but randomly …

Content truncated to 500 words