Explaining How Transformers Use Context to Build Predictions

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5486–5513 July 9-14, 2023 ©2023 Association for Computational Linguistics Explaining How Transformers Use Context to Build Predictions Javier Ferrando1, Gerard I. Gállego1, Ioannis Tsiamas1, Marta R. Costa-jussà2 1TALP Research Center, Universitat Politècnica de Catalunya 2Meta AI {javier.ferrando.monsonis,gerard.ion.gallego,ioannis.tsiamas}@upc.edu costajussa@meta.com Abstract Language Generation Models produce words based on the previous context. Although exist- ing methods offer input attributions as explana- tions for a model’s prediction, it is still unclear how prior words affect the model’s decision throughout the layers. In this work, we lever- age recent advances in explainability of the Transformer and present a procedure to analyze models for language generation. Using con- trastive examples, we compare the alignment of our explanations with evidence of the lin- guistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines. Then, we investigate the role of MLPs inside the Trans- former and show that they learn features that help the model predict words that are gram- matically acceptable. Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predic- tions. 1 Introduction Language Generation Models, like Transformer- based Language Models (Brown et al., 2020; Zhang et al., 2022a) have recently revolutionized the field of Natural Language Processing (NLP). Despite this, there is still a gap in our understanding of how they are able to produce language that closely resembles that of humans. This means that we are unable to determine the cause of a model’s failure in specific instances, which can result in the generation of hallucinated content or toxic output. The majority of previous work in explainability of NLP model predictions has focused on analyzing them on downstream tasks, generally with a small output space, such as text classification or Natural Language Inference (Atanasova et al., 2020; Bast- ings et al., 2022; Zaman and Belinkov, 2022). This line of research includes a large body of work fo- cusing on the analysis of the attention mechanism Logits Difference: Increase Decrease Model Prediction: has (2.2%), have (0.1%) Logits Difference: logithas−have = 3.1 L12 | A report about the Impressionists has L11 | A report about the Impressionists has L10 | A report about the Impressionists has L9 | A report about the Impressionists has L8 | A report about the Impressionists has L7 | A report about the Impressionists has L6 | A report about the Impressionists has L5 | A report about the Impressionists has L4 | A report about the Impressionists has L3 | A report about the Impressionists has L2 | A report about the Impressionists has L1 | A report about the Impressionists has � | A report about the Impressionists has Table 1: Updates to the (logits) prediction difference between has and have in different layers produced by input tokens. Red indicates an increase in the difference in logits between both predictions. At the bottom, we show the …

Content truncated to 500 words