Language Models Use Monotonicity to Assess NPI Licensing

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4958–4969 August 1–6, 2021. ©2021 Association for Computational Linguistics 4958 Language Models Use Monotonicity to Assess NPI Licensing Jaap Jumelet↑ Milica Deni´c↑ Jakub Szymanik↑ Dieuwke Hupkesλ Shane Steinert-Threlkeld↓ ↑ Institute for Logic, Language and Computation, University of Amsterdam λ Facebook AI Research ↓ Department of Linguistics, University of Washington {j.w.d.jumelet, m.denic, J.K.Szymanik}@uva.nl dieuwkehupkes@fb.com shanest@uw.edu Abstract We investigate the semantic knowledge of lan- guage models (LMs), focusing on (1) whether these LMs create categories of linguistic envi- ronments based on their semantic monotonic- ity properties, and (2) whether these categories play a similar role in LMs as in human lan- guage understanding, using negative polarity item licensing as a case study. We introduce a series of experiments consisting of probing with diagnostic classifiers (DCs), linguistic ac- ceptability tasks, as well as a novel DC rank- ing method that tightly connects the probing results to the inner workings of the LM. By applying our experimental pipeline to LMs trained on various filtered corpora, we are able to gain stronger insights into the semantic gen- eralizations that are acquired by these mod- els.1 1 Introduction Neural language models (LMs) have become pow- erful approximators of human language, making it increasingly important to understand the features and mechanisms underlying their behavior (Linzen et al., 2018, 2019). In the past few years, a sub- stantial number of studies have investigated the lin- guistic capabilities of LMs (Gulordava et al., 2018; Giulianelli et al., 2018; Lakretz et al., 2019; Wu et al., 2020; Ettinger, 2020, i.a.). Such work has focused primarily on syntactic properties, while fewer studies have been done on what kind of for- mal semantic features are encoded by language models. In this paper, we focus explicitly on what LMs learn about a semantic property of sentences, and in what ways their knowledge reflects well- known features of human language processing. As the topic of our studies, we consider mono- tonicity, a semantic property of linguistic envi- 1All code and data can be found at https://github. com/jumelet/monotonicity-npi-lm ronments that plays an important role in human language understanding and inference (Hoeksema, 1986; Valencia, 1991; Van Benthem, 1995; Icard III and Moss, 2014): the monotonicity of a linguistic environment determines whether inferences from a general to a particular term or vice versa are valid in that environment. For example, the fact that the in- ference from “Mary didn’t write a paper” to “Mary didn’t write a linguistics paper” is valid shows us that the position where “a paper” occurs is down- ward monotone: the inference is valid when a more general term (“a paper”) is replaced with a more specific one (“a linguistics paper”). To investigate monotonicity we focus on nega- tive polarity items (NPIs): a class of expressions such as any or ever that are solely acceptable in downward monotone environments (Fauconnier, 1975; Ladusaw, 1979). Psycholinguistic research has confirmed this connection between NPIs and monotonicity: humans judge NPIs acceptable in a linguistic environment if they …

Content truncated to 500 words