Contributions of Transformer Attention Heads in Multi- and Cross-lingual Tasks

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1956–1966 August 1–6, 2021. ©2021 Association for Computational Linguistics 1956 Contributions of Transformer Attention Heads in Multi- and Cross-lingual Tasks Weicheng Ma1*, Kai Zhang2*†, Renze Lou3†, Lili Wang1, and Soroush Vosoughi4 1,4Department of Computer Science, Dartmouth College 2Department of Computer Science and Technology, Tsinghua University 3Department of Computer Science, Zhejiang University City College 1{first.last}.gr@dartmouth.edu 2drogozhang@gmail.com 3marionojump0722@gmail.com 4soroush.vosoughi@dartmouth.edu Abstract This paper studies the relative importance of attention heads in Transformer-based models to aid their interpretability in cross-lingual and multi-lingual tasks. Prior research has found that only a few attention heads are important in each mono-lingual Natural Language Pro- cessing (NLP) task and pruning the remaining heads leads to comparable or improved per- formance of the model. However, the impact of pruning attention heads is not yet clear in cross-lingual and multi-lingual tasks. Through extensive experiments, we show that (1) prun- ing a number of attention heads in a multi- lingual Transformer-based model has, in gen- eral, positive effects on its performance in cross-lingual and multi-lingual tasks and (2) the attention heads to be pruned can be ranked using gradients and identified with a few trial experiments. Our experiments focus on se- quence labeling tasks, with potential applica- bility on other cross-lingual and multi-lingual tasks. For comprehensiveness, we examine two pre-trained multi-lingual models, namely multi-lingual BERT (mBERT) and XLM-R, on three tasks across 9 languages each. We also discuss the validity of our findings and their extensibility to truly resource-scarce lan- guages and other task settings. 1 Introduction Prior research on mono-lingual Transformer-based (Vaswani et al., 2017) models reveals that a subset of their attention heads makes key contributions to each task, and the models perform comparably well (Voita et al., 2019; Michel et al., 2019) or even better (Kovaleva et al., 2019) with the remaining heads pruned 1. While multi-lingual Transformer- ∗Equal contribution. †Work done when interning at the Minds, Machines, and Society Lab at Dartmouth College. 1We regard single-source machine translation as a mono- lingual task since the inputs to the models are mono-lingual. based models, e.g. mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020), are widely ap- plied in cross-lingual and multi-lingual NLP tasks 2 (Wang et al., 2019; Keung et al., 2019; Eskander et al., 2020), no attempt has been made to extend the findings on the aforementioned mono-lingual research to this context. In this paper, we explore the roles of attention heads in cross-lingual and multi-lingual tasks for two reasons. First, better understanding and interpretability of Transformer- based models leads to efficient model designs and parameter tuning. Second, head-pruning makes Transformer-based models more applicable to truly resource-scarce languages if it does not negatively affect model performance significantly. The biggest challenge we face when studying the roles of attention heads in cross-lingual and multi-lingual tasks is locating the heads to prune. Existing research has shown that each attention head is specialized to …

Content truncated to 500 words