Evaluating the Values of Sources in Transfer Learning

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5084–5116 June 6–11, 2021. ©2021 Association for Computational Linguistics 5084 Evaluating the Values of Sources in Transfer Learning Md Rizwan Parvez University of California Los Angeles rizwan@cs.ucla.edu Kai-Wei Chang University of California Los Angeles kwchang@cs.ucla.edu Abstract Transfer learning that adapts a model trained on data-rich sources to low-resource targets has been widely applied in natural language processing (NLP). However, when training a transfer model over multiple sources, not every source is equally useful for the target. To better transfer a model, it is essential to understand the values of the sources. In this paper, we develop SEAL-Shap, an efficient source val- uation framework for quantifying the useful- ness of the sources (e.g., domains/languages) in transfer learning based on the Shapley value method. Experiments and comprehensive anal- yses on both cross-domain and cross-lingual transfers demonstrate that our framework is not only effective in choosing useful transfer sources but also the source values match the intuitive source-target similarity. 1 Introduction Transfer learning has been widely used in learning models for low-resource scenarios by leveraging the supervision provided in data-rich source cor- pora. It has been applied to NLP tasks in various settings including domain adaptation (Blitzer et al., 2007; Ruder and Plank, 2017), cross-lingual trans- fer (Täckström et al., 2013; Wu and Dredze, 2019), and task transfer (Liu et al., 2019b; Vu et al., 2020). A common transfer learning setting is to train a model on a set of sources and then evaluate it on the corresponding target (Yao and Doretto, 2010; Yang et al., 2020).1 However, not every source corpus contributes equally to the transfer model. Some of them may even cause a performance drop (Ghorbani and Zou, 2019; Lin et al., 2019). There- fore, it is essential to understand the value of each source in the transfer learning not only to achieve 1In this paper, we focus on two transfer learning scenarios: 1) cross-lingual and 2) cross-domain. We train a model on a set of source corpora and evaluate on a target corpus where each “corpus” refers to the corresponding domain or language. Transfer Performance Transfer Performance Transfer Performance Transfer Performance Transfer Performance Transfer Performance 0.45 0.45 0.95 0.85 0.40 0.80 = 𝟎.𝟒𝟓%𝟎.𝟒𝟓 𝟏 + 𝟎.𝟗𝟓%𝟎.𝟖𝟓 𝟐 + 𝟎.𝟒𝟎%𝟎.𝟖𝟎 𝟐 + 𝟎.𝟒𝟎%𝟎.𝟎𝟓 𝟏 Source Value Small Evaluation Target Corpus Source Values 𝝓 = {𝝓1, 𝝓2 ,..., 𝝓m} Source Value (𝝓1) Source Value (𝝓2) Source Value (𝝓m) Transfer Performance Transfer Performance 0.40 0.05 SEAL-Shap Source Corpora 𝕯 = {𝑫1, 𝑫2 ,..., 𝑫m} Source Corpus (𝑫1) Source Corpus (𝑫2) Source Corpus (𝑫m) Based on Source Values, Select Potential Subset of the Source Corpora High Transfer Performance on Test Target Corpus 0.97 Example Source Corpus for which we elaborate the source value calculation Subsampled from the original large source corpus Notations Figure 1: SEAL-Shap estimates the value of each source corpus by the average marginal contribution of that particular source corpus to every …

Content truncated to 500 words