Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fast

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4002–4017 July 9-14, 2023 ©2023 Association for Computational Linguistics Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fast Yiduo Guo1,∗, Yaobo Liang2, Dongyan Zhao1,4,5†, Bing Liu3, Duan Nan2 1Wangxuan Institute of Computer Technology, Peking University 2Microsoft Research Asia 3Department of Computer Science, University of Illinois at Chicago 4National Key Laboratory of General Artificial Intelligence, 5BIGAI, Beijing, China yiduo@stu.pku.edu.cn,yaobo.liang@microsoft.com, zhaody@pku.edu.cn nanduan@microsoft.com,liub@uic.edu Abstract Existing research has shown that a multilingual pre-trained language model fine-tuned with one (source) language also performs well on down- stream tasks for non-source languages, even though no fine-tuning is done on these lan- guages. However, there is a clear gap between the performance of the source language and that of the non-source languages. This paper ana- lyzes the fine-tuning process, discovers when the performance gap changes and identifies which network weights affect the overall per- formance most. Additionally, the paper seeks to answer to what extent the gap can be reduced by reducing forgetting. Based on the analysis results, a method named Fine-tuning slow and fast with four training policies is proposed to address these issues. Experimental results show the proposed method outperforms baselines by a clear margin. 1 Introduction Multilingual pre-trained language models (LMs), such as mBERT (Devlin et al., 2018) and XLM- R (Conneau et al., 2019) have shown strong Zero- Shot Cross-Lingual transfer capabilities. Such a model F is usually pre-trained with unlabeled cor- pora D in multiple languages S to enable the model to learn cross-lingual knowledge Hpre cross. To adapt to a downstream task, the pre-trained LM F is typi- cally fine-tuned with a supervised dataset Dˆs of the downstream task T in one source language ˆs ∈ S due to data scarcity in other languages. When the fine-tuned model F is applied to the test set of the same task in the source language ˆs, it achieves strong performance Pˆs. Interestingly, when F is ap- plied to non-source languages, it can also achieve good performance (Conneau et al., 2019). We denote the average performance on the test sets of other languages than ˆs as PS/ˆs. However, the *Work done during internship at Microsoft Research Asia †Corresponding author Figure 1: The performance gap Pˆs − PS/ˆs every hun- dred updates on the XNLI dataset. ’Original perfor- mance gap’ means that we directly fine-tune the model, and ’Fine-tuning slow’/’fine-tuning fast’/’our method’ means that we use the fine-tuning slow algorithm/fine- tuning fast algorithm/the combination of both algo- rithms respectively to fine-tune the model. gap Pˆs − PS/ˆs is quite large (e.g., 13 percent for the XNLI dataset in Figure 1). One potential rea- son is that: during the fine-tuning of the model, the performance of non-source languages firstly increases with the performance of source language, then the arising of the performance of non-source languages becomes slower than that of the perfor- mance of source language as the forgetting of cross- lingual knowledge, resulting …

Content truncated to 500 words