Linguistic Dependencies and Statistical Dependence

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2941–2963 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2941 Linguistic Dependencies and Statistical Dependence Jacob Louis Hoover1,2, Alessandro Sordoni3, Wenyu Du4, and Timothy J. O’Donnell1,2,5 1McGill University 2Mila Québec AI Institute 3Microsoft Research, Montréal 4The University of Hong Kong jacob.hoover@mail.mcgill.ca,5Canada CIFAR AI Chair, Mila alsordon@microsoft.com, wenyudu@connect.hku.hk, timothy.odonnell@mcgill.ca Abstract Are pairs of words that tend to occur together also likely to stand in a linguistic dependency? This empirical question is motivated by a long history of literature in cognitive science, psy- cholinguistics, and NLP. In this work we con- tribute an extensive analysis of the relationship between linguistic dependencies and statistical dependence between words. Improving on pre- vious work, we introduce the use of large pre- trained language models to compute contextu- alized estimates of the pointwise mutual infor- mation between words (CPMI). For multiple models and languages, we extract dependency trees which maximize CPMI, and compare to gold standard linguistic dependencies. Over- all, we find that CPMI dependencies achieve an unlabelled undirected attachment score of at most ≈ 0.5. While far above chance, and consistently above a non-contextualized PMI baseline, this score is generally comparable to a simple baseline formed by connecting adja- cent words. We analyze which kinds of linguis- tic dependencies are best captured in CPMI de- pendencies, and also find marked differences between the estimates of the large pretrained language models, illustrating how their differ- ent training schemes affect the type of depen- dencies they capture. 1 Introduction A fundamental aspect of natural language struc- ture is the set of dependency relations which hold between pairs of words in a sentence. Such de- pendencies indicate how the sentence is to be in- terpreted and mediate other aspects of its structure, such as agreement. Consider the sentence: Several ravens flew out of their nests to confront the invad- ing mongoose. In this example, there is a depen- dency between the verb flew and its subject ravens, capturing the role this subject plays in the flying event, and how it controls number agreement. All modern linguistic theories recognize the centrality It is impossible to know whether that theory is realistic . It is CPMI impossible to know whether that theory is realistic . It is impossible to know whether that theory is realistic . LM Figure 1: We use models pretrained on masked lan- guage modelling objectives to extract trees which maximize contextualized pointwise mutual informa- tion (CPMI) between words, to examine how linguistic dependencies relate to statistical dependence. of such word-word relationships, despite consid- erable differences in detail in how they are treated (for a review of linguistic dependency grammar lit- erature see de Marneffe and Nivre, 2019). In addition to linguistic dependencies between words, there are also clear and robust statistical re- lationships. A noun like ravens is likely to occur with a verb like flew. In short, the presence or ab- sence of certain words in certain positions in a sen- tence …

Content truncated to 500 words