Finding the Pillars of Strength for Multi-Head Attention

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14526–14540 July 9-14, 2023 ©2023 Association for Computational Linguistics Finding the Pillars of Strength for Multi-Head Attention Jinjie Ni and Rui Mao and Zonglin Yang and Han Lei and Erik Cambria Nanyang Technological University, Singapore {jinjie001, leih0003}@e.ntu.edu.sg, {rui.mao, zonglin.yang, cambria}@ntu.edu.sg Abstract Recent studies have revealed some issues of Multi-Head Attention (MHA), e.g., redundancy and over-parameterization. Specifically, the heads of MHA were originally designed to at- tend to information from different representa- tion subspaces, whereas prior studies found that some attention heads likely learn simi- lar features and can be pruned without harm- ing performance. Inspired by the minimum- redundancy feature selection, we assume that focusing on the most representative and dis- tinctive features with minimum resources can mitigate the above issues and lead to more ef- fective and efficient MHAs. In particular, we propose Grouped Head Attention, trained with a self-supervised group constraint that group at- tention heads, where each group focuses on an essential but distinctive feature subset. We ad- ditionally propose a Voting-to-Stay procedure to remove redundant heads, thus achieving a transformer with lighter weights. Moreover, our method achieves significant performance gains on three well-established tasks while con- siderably compressing parameters. 1 Introduction Transformers have shown promising performance across various tasks . However, they have some issues, e.g., redundancy and over-parameterization, which is mainly caused by Multi-Head Attention (MHA) (Michel et al., 2019; Voita et al., 2019) and Feed-Forward Network (FFN) (Sukhbaatar et al., 2019; Wu et al., 2019, 2020) of trans- former. We aim to mitigate the redundancy and over-parameterization issues by optimizing the MHA module. The multi-heads were originally designed to attend to different representation sub- spaces of input (Vaswani et al., 2017). However, prior works (Michel et al., 2019; Voita et al., 2019) have shown that the attention heads are highly re- dundant and over-parameterized after training be- cause some heads can be switched off with a negli- gible performance drop. Such an issue is probably caused by their paral- lel design: the heads naturally work in the same way and likely attend to similar features (Cor- donnier et al., 2020). The existing redundancy optimization methods are mainly based on ho- mogenization, diversification, and head signifi- cance. However, they all have some limits. (1) The homogenization-based methods mitigate re- dundancy and over-parameterization by making heads similar and removing unnecessary parame- ters. Cordonnier et al. (2020) homogenized atten- tion heads by sharing most weights between all heads, which reduced the redundant parameters but sacrificed the performance somewhat because of the lack of diversity. (2) The diversification- based methods diversify the heads to enrich fea- tures and reduce the inter-head redundancy. Li et al. (2018) found that diversifying attention heads by adding a regularization could force MHA to reduce inter-head information redundancy, yielding per- formance gains in Machine Translation. However, such strategy that retains all feature subsets is sub- optimal, because it does not address the issue that MHA is …

Content truncated to 500 words