Transformers in the loop: Polarity in neural models of language

Transformers in the loop: Polarity in neural models of language Lisa Bylinina∗ Bookarang, Amsterdam bylinina@gmail.com Alexey Tikhonov∗ Yandex Technologies GmbH, Berlin altsoph@gmail.com Abstract Representation of linguistic phenomena in com- putational language models is typically as- sessed against the predictions of existing lin- guistic theories of these phenomena. Using the notion of polarity as a case study, we show that this is not always the most adequate set-up. We probe polarity via so-called ‘negative polarity items’ (in particular, English any) in two pre- trained Transformer-based models (BERT and GPT-2). We show that – at least for polarity – metrics derived from language models are more consistent with data from psycholinguistic ex- periments than linguistic theory predictions. Establishing this allows us to more adequately evaluate the performance of language models and also to use language models to discover new insights into natural language grammar beyond existing linguistic theories. This work contributes to establishing closer ties between psycholinguistic experiments and experiments with language models. 1 Introduction Recent Transformer-based language representation models (LRMs) – such as BERT and GPT-2 (De- vlin et al., 2019; Radford et al., 2019) – show im- pressive results on practical text analysis tasks. But do these models have access to complex linguistic notions? The results in this domain are less clear – as well as ways to best approach this question. Instead of asking whether LRMs encode frag- ments of current linguistic theory, we will directly compare metrics derived from LRMs to correspond- ing human judgments obtained in psycholinguistic experiments. The motivation for this is twofold. First, linguistic theories can be inaccurate – so, evaluating a model with respect to predictions of such theories is not informative about the model performance. Second, robust abstract theoretical notions rarely correspond to robust judgments in ∗Equal contribution. Accepted to ACL main conference. humans, and ‘theoretical’ and ‘perceived’ versions of the same phenomenon can be significantly dif- ferent (for instance, see Geurts 2003 on inference judgments; discussed in Section 2). If this is some- thing that LRMs inherit through training on human- produced texts, this makes LRMs an attractive pos- sible component in an experimental pipeline, serv- ing as a source of empirical predictions about hu- man linguistic behaviour (Baroni, 2021; Linzen and Baroni, 2021). As a case study, we focus on polarity: a com- plex property of sentences at the intersection of grammar and semantics. We tackle polarity via the distribution of items that are sensitive to it – namely, so-called negative polarity items (NPIs) like English any. As a basic illustration of NPI sensitivity to polarity, consider a pair of sentences in (1) (* = ungrammaticality): (1) a. Mary didn’t buy any books. b. *Mary bought any books. (1-a) is a negative sentence (has negative polarity), and any is grammatical in it. (1-b) is an affirma- tive sentence (has positive polarity) and any in this sentence is grammatically degraded compared to (1-a). Apart from this paradigmatic contrast, as we discuss below, polarity contrasts are expressed in a variety of ways and …

Content truncated to 500 words