How Far Does BERT Look At: Distance-based Clustering and Analysis of BERT’s Attention

Proceedings of the 28th International Conference on Computational Linguistics, pages 3853–3860 Barcelona, Spain (Online), December 8-13, 2020 3853 How Far Does BERT Look At: Distance-based Clustering and Analysis of BERT’s Attention Yue Guan1⋆†, Jingwen Leng1⋆†, Chao Li2⋆†, Quan Chen2⋆†, Minyi Guo2⋆† ⋆Shanghai Jiao Tong University †Shanghai Qi Zhi Institute 1{bonboru,leng-jw}@sjtu.edu.cn 2{lichao,chen-quan,guo-my}@cs.sjtu.edu.cn Abstract Recent research on the multi-head attention mechanism, especially that in pre-trained models such as BERT, has shown us heuristics and clues in analyzing various aspects of the mechanism. As most of the research focus on probing tasks or hidden states, previous works have found some primitive patterns of attention head behavior by heuristic analytical methods, but a more system- atic analysis specific on the attention patterns still remains primitive. In this work, we clearly cluster the attention heatmaps into significantly different patterns through unsupervised cluster- ing on top of a set of proposed features, which corroborates with previous observations. We further study their corresponding functions through analytical study. In addition, our proposed features can be used to explain and calibrate different attention heads in Transformer models. 1 Introduction With the rapid development of neural network in NLP tasks these year, the Transformer (Vaswani et al., 2017) that uses multi-head attention (MHA) mechanism is one recent huge leap (Goldberg, 2016). It has become a standard building block of recent NLP models. The Transformer-based BERT (Devlin et al., 2018) model further advances the model accuracy by introducing pre-training and has reached the state-of-the-art performance on many NLP tasks. Beyond the general explanation which attributes the effectiveness of BERT model to its capability of long-range dependency and contextual embeddings, more detailed analysis of the MHA mechanism in BERT still remains an active research topic (Jain and Wallace, 2019; Serrano and Smith, 2019; Etha- yarajh, 2019; Michel et al., 2019). Michael et al. (2020); Brunner et al. (2020); Coenen et al. (2019) explore the linguistic information or importance of BERT by inspecting hidden state embeddings. Wu et al. (2020); Roy et al. (2020) further investigate possible attention mechanism designs. Several previous works on attention interpretability have analyzed the function and behavior of atten- tion heads. Jawahar et al. (2019) finds that attention heads within a same layer tend to function similarly, and vast redundancy could be found in the attention heads. Clark et al. (2019) provides the study on relating the behavior of each head to linguistic merits such as dependency parser. Kovaleva et al. (2019) manually annotates attention heatmaps and trains a CNN to classify attention heads. Given these various analytical findings, from an algorithmic view, we are not aware of any research that provides a clear and reliable method to classify the attention heads without extra human intervention. In this work, we provide a simple set of features that can be used to reliably disentangle the attention heads into different categories through clustering method. Different from heuristic analytics that looks at single inputs, our proposed attention head features are extracted corpus-wide and is thus more compre- hensive in its discovered …

Content truncated to 500 words