Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2416–2427 May 2-6, 2023 ©2023 Association for Computational Linguistics Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers William Held wheld3@gatech.edu Diyi Yang diyiy@stanford.edu Abstract Multilingual transformer-based models demon- strate remarkable zero and few-shot trans- fer across languages by learning and reusing language-agnostic features. However, as a fixed-size model acquires more languages, its performance across all languages degrades. Those who attribute this interference phe- nomenon to limited model capacity address the problem by adding additional parameters, de- spite evidence that transformer-based models are overparameterized. In this work, we show that it is possible to reduce interference by in- stead identifying and pruning language-specific attention heads. First, we use Shapley Values, a credit allocation metric from coalitional game theory, to identify attention heads that intro- duce interference. Then, we show that pruning such heads from a fixed model improves perfor- mance for a target language on both sentence classification and structural prediction. Finally, we provide insights on language-agnostic and language-specific attention heads using atten- tion visualization.1 1 Introduction Cross-lingual transfer learning aims to utilize a nat- ural language processing system trained on a source language to improve results for the same task in a different target language. The core goal is to main- tain relevant learned patterns from the source while disregarding those which are inapplicable to the target. Multilingual pretraining of transformer lan- guage models has recently become a widespread method for cross-lingual transfer; demonstrating re- markable zero and few shot performance across lan- guages when finetuned on monolingual data (Pires et al., 2019; Conneau et al., 2019; Xue et al., 2021). However, adding languages beyond a threshold begins to harm cross-lingual transfer in a fixed- size model as shown in prior work (Conneau et al., 2019; Xue et al., 2021). This phenomenon, termed 1We release code to compute Shapley Values on GitHub interference, has been addressed with additional parameters, both language-specific (Pfeiffer et al., 2020) and broadly (Conneau et al., 2019; Xue et al., 2021). Wang et al. (2020) justifies this by showing that competition over limited capacity drives inter- ference. This seems to contradict the lottery ticket hypothesis, which has shown that pretrained lan- guage models are highly overparameterized (Fran- kle and Carbin, 2019; Chen et al., 2020). We offer an alternate hypothesis that interfer- ence is caused by components that are special- ized to language-specific patterns and introduce noise when applied to other languages. To test this hypothesis, we introduce a methodology that selectively removes noisy components to improve language-specific performance without updating or adding additional language-specific parameters. Our work builds on prior research studying mono- lingual models that shows they can be pruned ag- gressively (Michel et al., 2019; Voita et al., 2019). We leverage Shapley Values, the mean marginal contribution of a player to a collaborative reward, to identify attention heads that cause interference. Un- like prior methods, Shapley Values map each head to positive and negative values …

Content truncated to 500 words