Exploiting large pre-trained models for low-resource neural machine translation

Exploiting large pre-trained models for low-resource neural machine translation Aar´on Galiano-Jim´enez, Felipe S´anchez-Mart´ınez, V´ıctor M. S´anchez-Cartagena, Juan Antonio P´erez-Ortiz Dep. de Llenguatges i Sistemes Inform`atics, Universitat d’Alacant E-03690 Sant Vicent del Raspeig (Spain) aaron.galiano@ua.es, {fsanchez,vmsanchez,japerez}@dlsi.ua.es Abstract Pre-trained models have revolutionized the natural language processing field by lever- aging large-scale language representations for various tasks. Some pre-trained mod- els offer general-purpose representations, while others are specialized in particu- lar tasks, like neural machine translation (NMT). Multilingual NMT-targeted sys- tems are often fine-tuned for specific lan- guage pairs, but there is a lack of evidence- based best-practice recommendations to guide this process. Additionally, deploying these large pre-trained models in computa- tionally restricted environments, typically found in developing regions where low- resource languages are spoken, has be- come challenging. We propose a pipeline to tune the mBART50 pre-trained model to 8 diverse low-resource language pairs, and then distill the resulting system to obtain lightweight and more sustainable NMT models. Our pipeline conveniently exploits back-translation, synthetic corpus filtering, and knowledge distillation to de- liver efficient bilingual translation models that are 13 times smaller, while maintain- ing a close BLEU performance. 1 Introduction In the field of natural language processing (NLP), most of the so called pre-trained or foundation models (Bommasani et al., 2021) fall into one of three categories, based on whether the under- lying architecture corresponds to the encoder of the transformer (Vaswani et al., 2017), the de- coder or both. Encoder-like models consist of a © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. number of bidirectional self-attention layers that learn deep general-purpose representations with self-supervised denoising learning objectives — such as predicting the original token for masked or perturbed tokens in the input— and can then be adapted to a wide range of downstream tasks. Monolingual models such as BERT (Devlin et al., 2019) and cross-lingual variations like mBERT or XLM-R (Conneau et al., 2020) have been obtained this way. Decoder-like pre-trained models —such as GPT-3 (Brown et al., 2020) or LLaMA (Tou- vron et al., 2023)— are trained to auto-regressively predict the next token in the sequence by us- ing causal self-attention layers. Pre-trained mod- els involving the whole encoder-decoder trans- former architecture —e.g. DeltaLM (Ma et al., 2021), BART (Lewis et al., 2020) and its cross- lingual variation mBART (Liu et al., 2020)— are also pre-trained to denoise perturbations in the in- put, and then fine-tuned for particular text-to-text downstream tasks such as neural machine transla- tion (NMT). In addition to models pre-trained to obtain general-purpose neutral representations, there ex- ist a number of multilingual encoder-decoder mod- els specifically pre-trained to translate between many different language pairs. Well-known sys- tems in this group include mBART50 (Tang et al., 2021), or NLLB-200 (NLLB Team et al., 2022). All these pre-trained models attain high translation quality (Tran et al., 2021) because they leverage information from multiple language pairs, thus be- coming an interesting realization of the possibili- ties of transfer …

Content truncated to 500 words