How to Dissect a Muppet: The Structure of Transformer Embedding Spaces

How to Dissect a Muppet: The Structure of Transformer Embedding Spaces Timothee Mickus∗ University of Helsinki, Finland timothee.mickus @helsinki.fi Denis Paperno Utrecht University, The Netherlands d.paperno@uu.nl Mathieu Constant Universit´e de Lorraine, CNRS,ATILF, France Mathieu.Constant @univ-lorraine.fr Abstract Pretrained embeddings based on the Trans- former architecture have taken the NLP community by storm. We show that they can mathematically be reframed as a sum of vector factors and showcase how to use this refram- ing to study the impact of each component. We provide evidence that multi-head atten- tions and feed-forwards are not equally useful in all downstream applications, as well as a quantitative overview of the effects of fine- tuning on the overall embedding space. This approach allows us to draw connections to a wide range of previous studies, from vector space anisotropy to attention weights. 1 Introduction The Transformer architecture (Vaswani et al., 2017) has taken the NLP community by storm. Based on the attention mechanism (Bahdanau et al., 2015; Luong et al., 2015), it was shown to outperform recurrent architectures on a wide variety of tasks. Another step was taken with pretrained language models derived from this ar- chitecture (BERT, Devlin et al., 2019, among others): they now embody the default approach to a vast swath of NLP applications. Success breeds scrutiny; likewise the popularity of these models has fostered research in explainable NLP interested in the behavior and explainability of pretrained language models (Rogers et al., 2020). In this paper, we develop a novel decomposition of Transformer output embeddings. Our approach consists in quantifying the contribution of each network submodule to the output contextual em- bedding, and grouping those into four terms: (i) what relates to the input for a given position, (ii) what pertains to feed-forward submodules, (iii) ∗The work described in the present paper was conducted chiefly while at ATILF. what corresponds to multi-head attention, and (iv) what is due to vector biases. This allows us to investigate Transformer em- beddings without relying on attention weights or treating the entire model as a black box, as is most often done in the literature. The usefulness of our method is demonstrated on BERT: Our case study yields enlightening connections to state-of-the-art work on Transformer explainability, evidence that multi-head attentions and feed-forwards are not equally useful in all downstream applications, as well as an overview of the effects of finetuning on the embedding space. We also provide a simple and intuitive measurement of the importance of any term in this decomposition with respect to the whole embedding. We will provide insights on the Transformer architecture in Section 2, and showcase how these insights can translate into experimental investiga- tions in Sections 3 to 6. We will conclude with connections to other relevant works in Section 7 and discuss future perspectives in Section 8. 2 Additive Structure in Transformers We show that the Transformer embedding et for a token t is as a sum of four terms: et = it + ht + ft + ct (1) where it …

Content truncated to 500 words