Composable Sparse Fine-Tuning for Cross-Lingual Transfer

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1778 - 1796 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Composable Sparse Fine-Tuning for Cross-Lingual Transfer Alan Ansell1 Edoardo Maria Ponti1,2 Anna Korhonen1 Ivan Vuli´c1 1Language Technology Lab, University of Cambridge 2Mila - Quebec AI Institute and McGill University Abstract Fine-tuning the entire set of parameters of a large pretrained model has become the main- stream approach for transfer learning. To in- crease its efficiency and prevent catastrophic forgetting and interference, techniques like adapters and sparse fine-tuning have been de- veloped. Adapters are modular, as they can be combined to adapt a model towards dif- ferent facets of knowledge (e.g., dedicated language and/or task adapters). Sparse fine- tuning is expressive, as it controls the behav- ior of all model components. In this work, we introduce a new fine-tuning method with both these desirable properties. In particular, we learn sparse, real-valued masks based on a sim- ple variant of the Lottery Ticket Hypothesis. Task-specific masks are obtained from anno- tated data in a source language, and language- specific masks from masked language model- ing in a target language. Both these masks can then be composed with the pretrained model. Unlike adapter-based fine-tuning, this method neither increases the number of param- eters at inference time nor alters the original model architecture. Most importantly, it out- performs adapters in zero-shot cross-lingual transfer by a large margin in a series of mul- tilingual benchmarks, including Universal De- pendencies, MasakhaNER, and AmericasNLI. Based on an in-depth analysis, we addition- ally find that sparsity is crucial to prevent both 1) interference between the fine-tunings to be composed and 2) overfitting. We release the code and models at https://github.com/ cambridgeltl/composable-sft. 1 Introduction Fine-tuning of pretrained models (Howard and Ruder, 2018; Devlin et al., 2019, inter alia) is ar- guably the dominant paradigm in NLP at present. Originally, “fine-tuning” involved supervised learn- ing of all the parameters of a model pretrained on unlabeled texts. However, given the size of Transformer-based architectures, this approach is often time- and resource- inefficient, and may result in catastrophic forgetting and interference (Wang et al., 2020) during multiple adaptations. To over- come these limitations, two main alternatives have emerged: 1) through adapters, new parameters can be added to a pretrained model in the form of extra intermediate layers (Rebuffi et al., 2017; Houlsby et al., 2019) and fine-tuned while keeping all the pretrained parameters fixed; 2) sparse fine-tuning (SFT) of a small subset of pretrained model param- eters (Guo et al., 2021; Zaken et al., 2021; Xu et al., 2021b, inter alia). Adapters have proven especially useful in multi- lingual NLP (Bapna and Firat, 2019; Üstün et al., 2020; Pfeiffer et al., 2020b; Vidoni et al., 2020; Pfeiffer et al., 2021b; Ansell et al., 2021) because they exhibit a surprising degree of modularity. This ability to disentangle and recombine orthogonal facets of knowledge in original ways (Ponti et al., 2021; Ponti, 2021) allows for separately …

Content truncated to 500 words