Attention is Not Only a Weight: Analyzing Transformers with Vector Norms

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7057–7075, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7057 Attention is Not Only a Weight: Analyzing Transformers with Vector Norms Goro Kobayashi1 Tatsuki Kuribayashi1,2 Sho Yokoi1,3 Kentaro Inui1,3 1 Tohoku University 2 Langsmith Inc. 3 RIKEN {goro.koba, kuribayashi, yokoi, inui}@ecei.tohoku.ac.jp Abstract Attention is a key component of Transform- ers, which have recently achieved consider- able success in natural language processing. Hence, attention is being extensively studied to investigate various linguistic capabilities of Transformers, focusing on analyzing the par- allels between attention weights and specific linguistic phenomena. This paper shows that attention weights alone are only one of the two factors that determine the output of atten- tion and proposes a norm-based analysis that incorporates the second factor, the norm of the transformed input vectors. The findings of our norm-based analyses of BERT and a Transformer-based neural machine translation system include the following: (i) contrary to previous studies, BERT pays poor attention to special tokens, and (ii) reasonable word align- ment can be extracted from attention mecha- nisms of Transformer. These findings provide insights into the inner workings of Transform- ers. 1 Introduction Transformers (Vaswani et al., 2017; Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019; Lan et al., 2020) have improved the state-of-the-art in a wide range of natural language processing tasks. The success of the models has not yet been sufficiently explained; hence, substantial research has focused on assessing the linguistic capabilities of these models (Rogers et al., 2020; Clark et al., 2019). One of the main features of Transformers is that they utilize an attention mechanism without the use of recurrent or convolutional layers. The atten- tion mechanism computes an output vector by ac- cumulating relevant information from a sequence of input vectors. Specifically, it assigns attention weights (i.e., relevance) to each input, and sums up input vectors based on their weights. The anal- ysis of correlations between attention weights and various linguistic phenomena (i.e., weight-based analysis) is a prominent research area (Clark et al., 2019; Kovaleva et al., 2019; Reif et al., 2019; Lin et al., 2019; Mareˇcek and Rosa, 2019; Htut et al., 2019; Raganato and Tiedemann, 2018; Tang et al., 2018). This paper first shows that weight-based analy- sis is insufficient to analyze the attention mech- anism. Weight-based analysis is a common ap- proach to analyze the attention mechanism by simply tracking attention weights. The attention mechanism can be expressed as a weighted sum of linearly transformed vectors (Section 2.2); how- ever, the effect of transformed vectors in weight- based analysis is ignored. We propose a norm- based analysis that considers the previously ig- nored factors (Section 3). In this analysis, we mea- sure the norms (lengths) of the vectors that were summed to compute the output vector of the atten- tion mechanism. Using the norm-based analysis of BERT (Sec- tion 4), we interpreted the internal workings of the model in more detail than when weight-based analysis …

Content truncated to 500 words