Spectral Probing

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7730 - 7741 December 7-11, 2022 ©2022 Association for Computational Linguistics Spectral Probing Max Müller-Eberstein ☼ and Rob van der Goot ☼ and Barbara Plank ☼�� ☼ Department of Computer Science, IT University of Copenhagen, Denmark � Center for Information and Language Processing (CIS), LMU Munich, Germany � Munich Center for Machine Learning (MCML), Munich, Germany mamy@itu.dk, robv@itu.dk, b.plank@lmu.de Abstract Linguistic information is encoded at varying timescales (subwords, phrases, etc.) and com- municative levels, such as syntax and seman- tics. Contextualized embeddings have analo- gously been found to capture these phenomena at distinctive layers and frequencies. Leverag- ing these findings, we develop a fully learnable frequency filter to identify spectral profiles for any given task. It enables vastly more granu- lar analyses than prior handcrafted filters, and improves on efficiency. After demonstrating the informativeness of spectral probing over manual filters in a monolingual setting, we in- vestigate its multilingual characteristics across seven diverse NLP tasks in six languages. Our analyses identify distinctive spectral profiles which quantify cross-task similarity in a lin- guistically intuitive manner, while remaining consistent across languages—highlighting their potential as robust, lightweight task descriptors. 1 Introduction Analyzing the contextualized embedding represen- tations of pre-trained language models (LMs) us- ing lightweight probes (Hewitt and Liang, 2019; Voita and Titov, 2020) has identified latent fea- tures in the untuned encoders which are highly relevant to downstream NLP tasks at various layer depths (Tenney et al., 2019). Orthogonally, lin- guistic phenomena are also encoded at different timescales: i.e., rapidly changing (sub-)word-level information versus slower changing sentence or paragraph-level information. Decomposing contex- tualized embeddings into frequencies with different rates of change has yielded initial insights into the timescales at which these task-specific latent phe- nomena occur (Tamkin et al., 2020). These findings currently rely on handcrafted spectral filters and are limited to English. To enable more efficient analy- ses of finer-grained, continuous frequency spectra in contextualized representations covering more tasks and languages, this work contributes: DCT IDCT 1 2 3 4 Figure 1: Visualization of Spectral Probing. Given a sequence of embedding values 1 , decompose into composite frequency waves using DCT 2 , apply the learned filter γ , retaining a subset of waves 3 , for which IDCT returns the filtered sequence of values 4 . • A fully differentiable spectral probing frame- work for learning which frequencies are rele- vant for specific NLP tasks (Section 2).1 • A multilingual probing study examining timescale characteristics of seven diverse NLP tasks across six languages (Section 3). • An analysis of the relationships between the spectral profiles of different tasks and their consistency across languages (Section 4). 2 Probing for Spectral Profiles Spectral Probing (Figure 1) builds on established signal processing methods (Ahmed et al., 1974) and recent findings on the manual frequency filtering of 1Code at https://github.com/mainlp/spectral-probing. 7730 contextual embeddings (Tamkin et al., 2020). The method automatically learns spectral profiles which measure the relevance of specific frequencies to …

Content truncated to 500 words