Enlivening Redundant Heads in Multi-head Self-attention for Machine Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3238–3248 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3238 Enlivening Redundant Heads in Multi-head Self-attention for Machine Translation Tianfu Zhang†‡, Heyan Huang†‡, Chong Feng†♣∗, Longbing Cao♠ †Beijing Institute of Technology ‡Intelligent Information Processing and Contents Computing, Key Laboratory of MIIT ♣The Southeast Information Technology Research institute of BIT ♠University of Technology Sydney {tianfuzhang,hhy63,fengchong}@bit.edu.cn LongBing.Cao@uts.edu.au Abstract Multi-head self-attention recently attracts enormous interest owing to its specialized functions, significant parallelizable computa- tion, and flexible extensibility. However, very recent empirical studies show that some self- attention heads make little contribution and can be pruned as redundant heads. This work takes a novel perspective of identifying and then vitalizing redundant heads. We propose a redundant head enlivening (RHE) method to precisely identify redundant heads, and then vitalize their potential by learning syntactic re- lations and prior knowledge in text without sacrificing the roles of important heads. Two novel syntax-enhanced attention (SEA) mech- anisms: a dependency mask bias and a rela- tive local-phrasal position bias, are introduced to revise self-attention distributions for syntac- tic enhancement in machine translation. The importance of individual heads is dynamically evaluated during the redundant heads identifi- cation, on which we apply SEA to vitalize re- dundant heads while maintaining the strength of important heads. Experimental results on WMT14 and WMT16 English→German and English→Czech language machine translation validate the RHE effectiveness. 1 Introduction Recently, self-attention network (SAN) (Lin et al., 2017) has been applied to various natural language processing tasks. Instead of drawing distance- aware dependencies like recurrent neural network (Hochreiter and Schmidhuber, 1997) and convolu- tional neural network (Kim, 2014), SAN captures short- and long-range relations between elements. SAN involves all signals with a weighted averag- ing operation, which may incorporate too many unrelated elements to concentrates on specific rela- tions. Recent work has modified SAN to enhance specific relation learning. For example, in (Shen ∗Corresponding author. Modifying Multi-head SANs Identifying and Pruning Redundant heads Indentifying and Enhancing Redundant heads Figure 1: Rationale of the multi-head SAN. The left and middle parts are two existing SAN methods, the right one illustrates our proposed method. The col- ored circles represent different functions of individual heads. et al., 2018), a directional self-attention network (DiSAN) uses one to multiple positional masks to model the asymmetric attention between two ele- ments and capture context-aware relations for all tokens. (Yang et al., 2018) modeled the local infor- mation by revising the attention distribution with a learnable Gaussian bias to focus on neighboring relations. (Shaw et al., 2018) extended SAN to efficiently consider distinct representations of the relative linear position relations between sequence elements. However, the above approaches consider the multi-head SAN as a whole but ignore unbal- anced contribution distributions between heads. Furthermore, multi-head SAN combines differ- ent attentions from multiple subspaces to construct Transformer (Vaswani et al., 2017) and achieves the state-of-the-art results in recent neural machine translation (NMT) tasks (Hassan et al., 2018). The very recent work (Voita et …

Content truncated to 500 words