Examining Large Pre-Trained Language Models for Machine Translation: What You Don’t Know about It

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 908 - 919 December 7-8, 2022 ©2022 Association for Computational Linguistics Examining Large Pre-Trained Language Models for Machine Translation: What You Don’t Know About It Lifeng Han1, Gleb Erofeev2, Irina Sorokina2, Serge Gladkoff2, and Goran Nenadic1 1 The University of Manchester, UK 2 Logrus Global, Translation & Localization lifeng.han, g.nenadic@manchester.ac.uk gleberof, irina.sorokina, serge.gladkoff@logrusglobal.com Abstract Pre-trained language models (PLMs) often take advantage of the monolingual and multilingual dataset that is freely available online to acquire general or mixed domain knowledge before deployment into specific tasks. Extra-large PLMs (xLPLMs) are proposed very recently to claim supreme performances over smaller- sized PLMs such as in machine translation (MT) tasks. These xLPLMs include Meta-AI’s wmt21-dense-24-wide-en-X (2021) and NLLB (2022). In this work, we examine if xLPLMs are absolutely superior to smaller-sized PLMs in fine-tuning toward domain-specific MTs. We use two different in-domain data of different sizes: commercial automotive in-house data and clinical shared task data from the Clin- SpEn2022 challenge at WMT2022. We choose popular Marian Helsinki as smaller sized PLM and two massive-sized Mega-Transformers from Meta-AI as xLPLMs. Our experimental investigation shows that 1) on smaller sized in-domain commercial auto- motive data, xLPLM wmt21-dense-24-wide- en-X indeed shows much better evaluation scores using SACREBLEU and hLEPOR met- rics than smaller-sized Marian, even though its score increase rate is lower than Marian after fine-tuning; 2) on relatively larger-size well prepared clinical data fine-tuning, the xLPLM NLLB tends to lose its advantage over smaller-sized Marian on two sub-tasks (clinical terms and ontology concepts) using ClinSpEn offered metrics METEOR, COMET, and ROUGE-L, and totally lost to Marian on Task-1 (clinical cases) on all official metrics including SACREBLEU and BLEU; 3) met- rics do not always agree with each other on the same tasks using the same model out- puts; 4) clinic-Marian ranked No.2 on Task-1 (via SACREBLEU/BLEU) and Task-3 (via ME- TEOR and ROUGE) among all submissions. 1 Introduction Owing to the recent development of neural ma- chine translations (NMTs) (Kalchbrenner and Blun- som, 2013; Cho et al., 2014; Bahdanau et al., 2014; Akhbardeh et al., 2021; Han, 2022a), especially the self-attention based Transformer learning struc- tures (Devlin et al., 2019; Vaswani et al., 2017), pre- trained language models (PLMs) have been domi- nant in natural language understanding (NLU) and natural language processing (NLP) tasks. These applications include Long-Short Term Memory (LSTM) and BERT (Pre-training of Deep Bidi- rectional Transformers) based models to text min- ing (Dernoncourt et al., 2017; Wu et al., 2022), question-answering (Dong et al., 2021), reading comprehension (Schlegel, 2021), and summarisa- tion (Perez-Beltrachini and Lapata, 2021), etc., in addition to MT (Han et al., 2021a; Han, 2022b; Han and Gladkoff, 2022). PLMs often have a large amount of trainable parameters for downstream applications. For in- stance, in translation task, the popular Marian NMT (Junczys-Dowmunt et al., 2018) pre-trained by Mi- crosoft Translator team 1 on OPUS 2 (Tiedemann, 2012) multilingual corpus has 7.6 million param- eters, which can still be fine-tuned on Google’s …

Content truncated to 500 words