Bird’s Eye: Probing for Linguistic Graph Structures with a Simple Information-Theoretic Approach

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1844–1859 August 1–6, 2021. ©2021 Association for Computational Linguistics 1844 Bird’s Eye: Probing for Linguistic Graph Structures with a Simple Information-Theoretic Approach Yifan Hou Department of Computer Science ETH Z¨urich yifan.hou@inf.ethz.ch Mrinmaya Sachan Department of Computer Science ETH Z¨urich mrinmaya.sachan@inf.ethz.ch Abstract NLP has a rich history of representing our prior understanding of language in the form of graphs. Recent work on analyzing contextual- ized text representations has focused on hand- designed probe models to understand how and to what extent do these representations en- code a particular linguistic phenomenon. How- ever, due to the inter-dependence of various phenomena and randomness of training probe models, detecting how these representations encode the rich information in these linguis- tic graphs remains a challenging problem. In this paper, we propose a new information- theoretic probe, Bird’s Eye, which is a fairly simple probe method for detecting if and how these representations encode the informa- tion in these linguistic graphs. Instead of us- ing classifier performance, our probe takes an information-theoretic view of probing and es- timates the mutual information between the linguistic graph embedded in a continuous space and the contextualized word represen- tations. Furthermore, we also propose an ap- proach to use our probe to investigate local- ized linguistic information in the linguistic graphs using perturbation analysis. We call this probing setup Worm’s Eye. Using these probes, we analyze BERT models on their ability to encode a syntactic and a semantic graph structure, and find that these models en- code to some degree both syntactic as well as semantic information; albeit syntactic infor- mation to a greater extent. Our implementa- tion is available in https://github.com/ yifan-h/Graph_Probe-Birds_Eye. 1 Introduction Graphs have served as a predominant represen- tation for various linguistic phenomena in natu- ral language (Marcus et al., 1993; De Marneffe et al., 2006; Hockenmaier and Steedman, 2007; Hajic et al., 2012; Abend and Rappoport, 2013; Ba- narescu et al., 2013; Bos, 2013). These graph based representations have served our intuition for repre- senting both language structure (Chomsky, 1957) as well as meaning (Koller et al., 2019). With the growing popularity of pretrained lan- guage models that build contextualized text repre- sentations (Reid et al., 2020; Devlin et al., 2019, inter alia), various probing models have been in- troduced to understand if and how our linguis- tic intuitions are encoded in these representations. These probes train supervised models to predict pieces of linguistic information such as POS (part- of-speech), morphology, syntactic and semantic relations, and other local or long-range phenom- ena in language (Belinkov et al., 2017; Conneau et al., 2018; Hewitt and Manning, 2019; Tenney et al., 2019b; Jawahar et al., 2019). However, it is still an open question if these representations somehow encode entire linguistic graph structures such as dependency and constituency parse trees or graph structured meaning representations such as AMR (Abstract Meaning Representation), UCCA (Universal Conceptual Cognitive …

Content truncated to 500 words