Where Does Linguistic Information Emerge in Neural Language Models? Measuring Gains and Contributions across Layers

Proceedings of the 29th International Conference on Computational Linguistics, pages 4664–4676 October 12–17, 2022. 4664 Where Does Linguistic Information Emerge in Neural Language Models? Measuring Gains and Contributions Across Layers Jenny Kunz and Marco Kuhlmann Dept. of Computer and Information Science Linköping University jenny.kunz@liu.se and marco.kuhlmann@liu.se Abstract Probing studies have extensively explored where in neural language models linguistic in- formation is located. The standard approach to interpreting the results of a probing classi- fier is to focus on the layers whose representa- tions give the highest performance on the prob- ing task. We propose an alternative method that asks where the task-relevant information emerges in the model. Our framework consists of a family of metrics that explicitly model local information gain relative to the previ- ous layer and each layer’s contribution to the model’s overall performance. We apply the new metrics to two pairs of syntactic probing tasks with different degrees of complexity and find that the metrics confirm the expected ordering only for one of the pairs. Our local metrics show a massive dominance of the first layers, indicating that the features that contribute the most to our probing tasks are not as high-level as global metrics suggest. 1 Introduction Probing neural language models aims at finding evidence of learned linguistic structure in the mod- els’ parameters by empirically testing hypotheses about the learned representations (Hupkes et al., 2018; Alain and Bengio, 2017). This is often done by training a probing classifier on a diagnostic task with the representations at different layers as the input, and comparing task performance across lay- ers. While probes are conceptually simple and widely used, the methodology and in particular the interpretation of the obtained results is subject to ongoing discussion (Belinkov, 2022). A classical pattern we often see when plotting probing accuracy across layers is that for higher- level linguistic tasks, the model aggregates informa- tion over several layers until it reaches its highest performance. Often, the curves start steep, flatten out, and eventually drop again in the final layers. Figure 1: Heatmaps illustrating our results for syntactic parent (P) and grandparent (GP) prediction (BERT-base, en, layers 1–12): Global metrics peak in middle lay- ers. Local contributions are concentrated in early layers. (Darker shades indicate higher values.) In this paper, we zoom in on models’ relative in- formation gains between one layer and the previous one. By this, we aim to turn the focus of probing away from information that is already present in the non-contextualized embedding layer and focus exclusively on information that needs context to be retrieved. We also aim to make the contribution of each layer to the model’s overall linguistic capabil- ities explicit. We argue that under the hypothesis that information in language models is structured as in classical NLP pipelines (Tenney et al., 2019), the depth at which information emerges is as im- portant as the impact of that information on model performance. When we think of language process- ing as a pipeline, we are primarily interested in …

Content truncated to 500 words