A Dynamic Head Importance Computation Mechanism for Neural Machine Translation

Proceedings of Recent Advances in Natural Language Processing, pages 454–462 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_052 454 A Dynamic Head Importance Computation Mechanism for Neural Machine Translation Akshay Goindani Manish Shrivastava International Institute of Information Technology - Hyderabad {akshay.goindani@research.iiit.ac.in, m.shrivastava@iiit.ac.in} Abstract Multiple parallel attention mechanisms that use multiple attention heads facilitate greater performance of the Transformer model for various applications e.g., Neural Machine Translation (NMT), text classification. In multi-head attention mechanism, different heads attend to different parts of the input. However, the limitation is that multiple heads might attend to the same part of the input, resulting in multiple heads being redundant. Thus, the model resources are under-utilized. One approach to avoid this is to prune least important heads based on certain importance score. In this work, we focus on designing a Dynamic Head Importance Computation Mechanism (DHICM) to dynamically calculate the importance of a head with respect to the input. Our insight is to design an additional attention layer together with multi-head attention, and utilize the outputs of the multi-head attention along with the input, to compute the importance for each head. Additionally, we add an extra loss function to prevent the model from assigning same score to all heads, to identify more important heads and improvise performance. We analyzed performance of DHICM for NMT with different languages. Experiments on different datasets show that DHICM outperforms traditional Transformer-based approach by large margin, especially, when less training data is available. 1 Introduction Transformer based NMT systems perform well on multiple translation tasks (Vaswani et al., 2017). Multi-head attention is a very important component of the Transformer model (Vaswani et al., 2017). Multiple heads improve performance compared to a single head, as they allow the model to jointly look at different subspaces, and hence capture enhanced features from sentences. For example, a head can capture positional information by attending to adjacent tokens, or it can capture syntactic information by attending to tokens in a particular syntactic dependency relation (Voita et al., 2019). However, the performance of the transformer-base model with 8 heads at each layer is only 1 BLEU point higher than that of a similar model with just a single head at each layer (Voita et al., 2019). This is due to the fact that majority of the heads learn similar weights, and therefore, multiple heads attend to the same parts of the input. Hence, most of the heads are redundant, leading to an increased computational complexity without improving performance. To avoid this redundancy, one approach is to prune the redundant heads based on certain importance score. In this work, we focus on designing an importance computation method to compute the importance score for each head. Some recent work has analyzed the importance of heads by considering average attention weights of each head at some specific position (Voita et al., 2018). However, average of attention weights is a static measure of the head importance as it does not consider the varying importance of each head with respect to the input. The importance …

Content truncated to 500 words