When does Parameter-Efficient Transfer Learning Work for Machine Translation?

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7919 - 7933 December 7-11, 2022 ©2022 Association for Computational Linguistics When does Parameter-Efficient Transfer Learning Work for Machine Translation? Ahmet Üstün* University of Groningen a.ustun@rug.nl Asa Cooper Stickland* University of Edinburgh a.cooper.stickland@ed.ac.uk Abstract Parameter-efficient fine-tuning methods (PEFTs) offer the promise of adapting large pre-trained models while only tuning a small number of parameters. They have been shown to be competitive with full model fine-tuning for many downstream tasks. However, prior work indicates that PEFTs may not work as well for machine translation (MT), and there is no comprehensive study showing when PEFTs work for MT. We conduct a comprehensive empirical study of PEFTs for MT, considering (1) various parameter budgets, (2) a diverse set of language-pairs, and (3) different pre-trained models. We find that ‘adapters’, in which small feed-forward networks are added after every layer, are indeed on par with full model fine-tuning when the parameter budget corresponds to 10% of total model parameters. Nevertheless, as the number of tuned parameters decreases, the performance of PEFTs decreases. The magni- tude of this decrease depends on the language pair, with PEFTs particularly struggling for distantly related language-pairs. We find that using PEFTs with a larger pre-trained model outperforms full fine-tuning with a smaller model, and for smaller training data sizes, PEFTs outperform full fine-tuning for the same pre-trained model.1 1 Introduction There has been enormous progress on scaling up neural machine translation (NMT) in the recent years, resulting in ‘massively multilingual’ mod- els that are capable of translating across many languages (Bapna et al., 2022). Most success- ful applications rely on sequence-to-sequence pre- training that (1) leverages web-scale monolingual * Both authors contributed the paper equally and the order is determined by coin flip. 1Our code and scripts for reproducing the experiments are available at https://github.com/ahmetustun/ fairseq data with a masking objective to build a multilin- gual backbone (parent) model (Liu et al., 2020; Song et al., 2019), or (2) directly targets a many- to-many NMT system by mining parallel corpora (Fan et al., 2020). Standard practice is to fine-tune every parameter of a particular a pre-trained model to specialize it to a language pair (or domain) of interest (Zoph et al., 2016; Neubig and Hu, 2018). However, if we require specialization to many language pairs or do- mains, the storage and time costs of full fine-tuning may become prohibitive. Moreover, as models grow ever larger, more efficient methods become attractive. As an alternative to full model fine-tuning, several parameter-efficient fine-tuning methods (PEFTs) have been proposed. Such methods only fine-tune a small number of parameters, reducing storage cost, and avoid calculating the gradients for every model parameter, reducing training time and memory cost. Examples include adapters (Houlsby et al., 2019; Bapna and Firat, 2019) and prefix- tuning (Li and Liang, 2021), which introduce a few extra parameters to fine-tune, keeping the pre- trained model fixed. Others like BitFit (Zaken et al., 2021) tune only the bias vectors …

Content truncated to 500 words