Detecting Extraneous Content in Podcasts

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 105–124 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 105 Telling BERT’s Full Story: from Local Attention to Global Aggregation Dami´an Pascual, Gino Brunner, Roger Wattenhofer Department of Electrical Engineering and Information Technology ETH Zurich, Switzerland {dpascual,brunnegi,wattenhofer}@ethz.ch Abstract We take a deep look into the behaviour of self- attention heads in the transformer architecture. In light of recent work discouraging the use of attention distributions for explaining a model’s behaviour, we show that attention distributions can nevertheless provide insights into the lo- cal behaviour of attention heads. This way, we propose a distinction between local patterns re- vealed by attention and global patterns that re- fer back to the input, and analyze BERT from both angles. We use gradient attribution to analyze how the output of an attention head depends on the input tokens, effectively ex- tending the local attention-based analysis to account for the mixing of information through- out the transformer layers. We find that there is a significant mismatch between attention and attribution distributions, caused by the mix- ing of context inside the model. We quantify this discrepancy and observe that interestingly, there are some patterns that persist across all layers despite the mixing. 1 Introduction The inception of the transformer architecture has sparked significant progress across a wide range of language understanding tasks. Vari- ants of transformers currently dominate the pop- ular GLUE (Wang et al., 2019b) and Super- GLUE (Wang et al., 2019a) benchmarks and have even achieved super human performance on mul- tiple tasks. The main innovations behind the transformer architecture are the stacking of self- attention layers into a multi-layer self-attention ar- chitecture, as well as an unsupervised pre-training phase that primes the model to be fine-tuned on a wide range of language tasks. Transformers and other self-attention-based models have been successfully adopted in other areas such as com- puter vision (Parmar et al., 2018), music process- ing (Huang et al., 2019) or protein research (Rao et al., 2019). Their extraordinary empirical success has led researchers to investigate transformers in or- der to better understand the source of this success, but also in an attempt to explain model decisions. Much of the research around interpretability and explainability is focused on analyzing the self- attention operation (Clark et al., 2019). In multi- layer self-attention, every input computes an atten- tion distribution over itself and all other inputs to produce ever more complex feature representations. In the case of language, a word in a sentence attends to itself and to all other words in order to compute an updated contextual representation of itself. It is tempting to directly rely on attention distributions to explain the model’s predictions. The rationale is that if the attention distribution aligns with human intuition, we can conclude that the model learned robust features and obtained a deep understanding of language, in contrast to simply overfitting on spurious patterns. For example, if a transformer …

Content truncated to 500 words