Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3440–3453 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3440 Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation Asa Cooper Stickland♣ Xian Li♠ ♣ University of Edinburgh, ♠ Facebook AI a.cooper.stickland@ed.ac.uk, {xianl,ghazvini}@fb.com Marjan Ghazvininejad♠ Abstract There has been recent success in pre-training on monolingual data and fine-tuning on Ma- chine Translation (MT), but it remains unclear how to best leverage a pre-trained model for a given MT task. This paper investigates the benefits and drawbacks of freezing parame- ters, and adding new ones, when fine-tuning a pre-trained model on MT. We focus on 1) Fine-tuning a model trained only on English monolingual data, BART. 2) Fine-tuning a model trained on monolingual data from 25 languages, mBART. For BART we get the best performance by freezing most of the model pa- rameters, and adding extra positional embed- dings. For mBART we match or outperform the performance of naive fine-tuning for most language pairs with the encoder, and most of the decoder, frozen. The encoder-decoder at- tention parameters are most important to fine- tune. When constraining ourselves to an out- of-domain training set for Vietnamese to En- glish we see the largest improvements over the fine-tuning baseline. 1 Introduction Machine Translation (MT) has recently seen sig- nificant advances, with improvements in model- ing, especially since the advent of neural models (Sutskever et al., 2014; Bahdanau et al., 2015), and the availability of large parallel corpora for train- ing such systems (Smith et al., 2013; Kocmi and Bojar, 2017; Tiedemann, 2012). However, often standard neural systems do not perform well on low-resource language pairs (Koehn and Knowles, 2017), especially when the language pairs are only distantly related. Since these languages are spoken by a large fraction of the world’s population, re- ducing the gap in performance between high and low-resource MT could have a large impact. An explosion of interest in large-scale pre- training in Natural Language Processing has led BART�Encoder (pre-trained) New�Encoder (randomly initialized) � � � � BART�Decoder (pre-trained) </s> A B C A B C D Adapter Adapter Adapter Adapter Adapters (randomly initialized) Figure 1: Schematic diagram showing the components of our system for adapting BART to MT. We learn a new encoder that takes as input the source language, with a potentially different vocabulary to the origi- nal BART system. We freeze most BART parameters (frozen model components are shown in blue). to increased performance on smaller datasets, by simple fine-tuning of large pre-trained models on downstream tasks. The typical approach is to train a large model on text from the web (for example English Wikipedia), with a common objective pre- dicting masked out tokens using the unmasked con- text. For Natural Language Generation (for exam- ple summarization of text), performance can be improved by pre-training a sequence-to-sequence model (Song et al., 2019; Lewis et al., 2019). However previous work has shown that on NLP tasks such as …

Content truncated to 500 words