Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Items

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 222–231 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 222 Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Items Jaap Jumelet University of Amsterdam jaap.jumelet@student.uva.nl Dieuwke Hupkes ILLC, University of Amsterdam d.hupkes@uva.nl Abstract In this paper, we attempt to link the inner workings of a neural language model to lin- guistic theory, focusing on a complex phe- nomenon well discussed in formal linguis- tics: (negative) polarity items. We briefly dis- cuss the leading hypotheses about the licens- ing contexts that allow negative polarity items and evaluate to what extent a neural language model has the ability to correctly process a subset of such constructions. We show that the model finds a relation between the licensing context and the negative polarity item and ap- pears to be aware of the scope of this context, which we extract from a parse tree of the sen- tence. With this research, we hope to pave the way for other studies linking formal linguistics to deep learning. 1 Introduction In the past decade, we have seen a surge in the de- velopment of neural language models (LMs). As they are more capable of detecting long distance dependencies than traditional n-gram models, they serve as a stronger model for natural language. However, it is unclear what kind of properties of language these models encode. This does not only hinder further progress in the development of new models, but also prevents us from using models as explanatory models and relating them to formal linguistic knowledge of natural language, an as- pect we are particularly interested in in the current paper. Recently, there has been an increasing interest in investigating what kind of linguistic informa- tion is represented by neural models, (see, e.g., Conneau et al., 2018; Linzen et al., 2016; Tran et al., 2018), with a strong focus on their syntac- tic abilities. In particular, (Gulordava et al., 2018) used the ability of neural LMs to detect noun-verb congruence pairs as a proxy for their awareness of syntactic structure, yielding promising results. In this paper, we follow up on this research by study- ing a phenomenon that has received much atten- tion by linguists and for which the model requires – besides knowledge of syntactic structure – also a semantic understanding of the sentence: negative polarity items (NPIs). In short, NPIs are a class of words that bear the special feature that they need to be licensed by a specific licensing context (LC) (a more elaborate linguistic account of NPIs can be found in the next section). A common example of an NPI and LC in English are any and not, respectively: The sen- tence He didn’t buy any books is correct, whereas He did buy any books is not. To properly process an NPI construction, a language model must be able to detect a relationship between a licensing context and an NPI. Following …

Content truncated to 500 words