The Functional Relevance of Probed Information: A Case Study

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 835–848 May 2-6, 2023 ©2023 Association for Computational Linguistics The Functional Relevance of Probed Information: A Case Study Michael Hanna∗, Roberto Zamparelli†, and David Mareˇcek§ ∗ILLC, University of Amsterdam, The Netherlands: m.w.hanna@uva.nl †CIMeC, University of Trento, Italy: roberto.zamparelli@unitn.it §MFF ÚFAL, Charles University, Czechia: marecek@ufal.mff.cuni.cz Abstract Recent studies have shown that transformer models like BERT rely on number informa- tion encoded in their representations of sen- tences’ subjects and head verbs when perform- ing subject-verb agreement. However, probing experiments suggest that subject number is also encoded in the representations of all words in such sentences. In this paper, we use causal interventions to show that BERT only uses the subject plurality information encoded in its rep- resentations of the subject and words that agree with it in number. We also demonstrate that cur- rent probing metrics are unable to determine which words’ representations contain function- ally relevant information. This both provides a revised view of subject-verb agreement in lan- guage models, and suggests potential pitfalls for current probe usage and evaluation. 1 Introduction The phenomenon of subject-verb agreement has re- ceived significant attention from the NLP commu- nity. In English, this phenomenon is very simple: present tense verbs must agree in number with their subject noun, which is either singular or plural. In the present tense, verbs that agree with 3rd-person singular nouns receive one verb conjugation, gener- ally ending in “-s”; in all other cases, the bare form of the verb is used. The simplicity of this phenomenon, combined with the potential for long-distance subject and verb agreement across intervening adverbs or relative clauses (e.g. “The friend [that probably called my parents] is...”) has made it the object of intense study in humans (Vigliocco et al. (1995) and Franck et al. (2010), inter alia). It also prompted early investigations on the ability of language models to capture it (left-to-right LSTMs in Linzen et al. (2016); Gulordava et al. (2018)). More recently, the popular pre-trained model BERT (Devlin et al., 2019) has been shown to be relatively proficient at subject-verb agreement (Goldberg, 2019), although these abilities depend somewhat on verb frequency and lexical patterns (Newman et al., 2021; Lasri et al., 2022a). Other studies ask not how models behave (e.g. with respect to subject-verb agreement), but how models’ representations support this behav- ior. Such studies often use probing, a technique in which an auxiliary classifier (probe) is trained to extract some property of words or sentences from models’ internal representations thereof (Belinkov and Glass, 2019; Belinkov, 2022). If the probe can extract the property with high accuracy, one concludes that the model has encoded the property in the word’s representation. Klafka and Ettinger (2020) discover that probes can extract the plural- ity of a sentence’s subject from last-layer BERT representations of any word in the sentence. However, probing has received criticism because the information discovered by probes is not always used by models (Ravichander et …

Content truncated to 500 words