BLiMP: The Benchmark of Linguistic Minimal Pairs for English

BLiMP: The Benchmark of Linguistic Minimal Pairs for English Alex Warstadt1, Alicia Parrish1, Haokun Liu2, Anhad Mohananey2, Wei Peng2, Sheng-FuWang1, Samuel R. Bowman1,2,3 1Department of Linguistics 2Department of Computer Science 3Center for Data Science New York University New York University New York University {warstadt,alicia.v.parrish,haokunliu,anhad, weipeng,shengfu.wang,bowman}@nyu.edu Abstract We introduce The Benchmark of Linguistic Minimal Pairs (BLiMP),1 a challenge set for evaluating the linguistic knowledge of lan- guage models (LMs) on major grammatical phenomena in English. BLiMP consists of 67 individual datasets, each containing 1,000 minimal pairs—that is, pairs of minimally dif- ferent sentences that contrast in grammatical acceptability and isolate specific phenomenon in syntax, morphology, or semantics. We gen- erate the data according to linguist-crafted grammar templates, and human aggregate agreement with the labels is 96.4%. We evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs by observ- ing whether they assign a higher probability to the acceptable sentence in each minimal pair. We find that state-of-the-art models identify morphological contrasts related to agreement reliably, but they struggle with some subtle semantic and syntactic phenomena, such as negative polarity items and extraction islands. 1 Introduction Current neural networks for sentence processing rely on unsupervised pretraining tasks like lan- guage modeling. Still, it is an open question how the linguistic knowledge of state-of-the-art language models (LMs) varies across the lin- guistic phenomena of English. Recent studies (e.g., Linzen et al., 2016; Marvin and Linzen, 2018; Wilcox et al., 2018) have explored this question by evaluating LMs’ preferences between minimal pairs of sentences differing in gramma- ticalacceptability, as in Example 1. However, each 1https://github.com/alexwarstadt/blimp. of these studies uses a different set of metrics, and focuses on a small set of linguistic paradigms, severely limiting any possible big- picture conclusions. (1) a. The cats annoy Tim. (grammatical) b. *The cats annoys Tim. (ungrammatical) We introduce the Benchmark of Linguistic Minimal Pairs (shortened to BLiMP), a linguis- tically motivated benchmark for assessing the sensitivity of LMs to acceptability contrasts across a wide range of English phenomena, covering both previously studied and novel contrasts. BLiMP consists of 67 datasets automatically generated from linguist-crafted grammar templates, each containing 1,000 minimal pairs and organized by phenomenon into 12 categories. Validation with crowdworkers shows that BLiMP faithfully represents human preferences. We use BLiMP to study several pretrained LMs: Transformer-based LMs GPT-2 (Radford et al., 2019) and Transformer-XL (Dai et al., 2019), an LSTM LM trained by Gulordava et al. (2019), and an n-gram LM. We evaluate whether the LM assigns a higher probability to the acceptable sentence in each minimal pair to determine which grammatical distinctions LMs are sensitive to. This gives us indirect evidence about each model’s linguistic knowledge and allows us to compare models in a fine-grained way. We conclude that current neural LMs appear to acquire robust knowledge of morphological agreement and some syntactic phenomena such as ellipsis and control/ raising. They show weaker evidence of knowledge about argument structure, negative polarity item licensing, and the semantic properties of quan- tifiers. All models perform at or …

Content truncated to 500 words