Neural Network Acceptability Judgments

Neural Network Acceptability Judgments Alex Warstadt New York University warstadt@nyu.edu Amanpreet Singh New York University Facebook AI Research∗ amanpreet@nyu.edu Samuel R. Bowman New York University bowman@nyu.edu Abstract This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We intro- duce the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. As baselines, we train several recurrent neural network mod- els on acceptability classification, and find that our models outperform unsupervised models by Lau et al. (2016) on CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et al.’s models and ours learn systematic generalizations like subject-verb- object order. However, all models we test per- form far below human level on a wide range of grammatical constructions. 1 Introduction Artificial neural networks (ANNs) achieve a high degree of competence on many applied natural language understanding tasks, but it does not follow that they have knowledge of grammar. A key property of a human’s linguistic competence is the ability to identify in one’s native language, without formal training in grammar, a contrast in acceptability1 between pairs of sentences like those in (1). Acceptability judgments like these are the primary behavioralmeasure that generative linguists use to observehumans’ grammaticalknowl- edge (Chomsky, 1957; Sch¨utze, 1996). (1) a. What did Betsy paint a picture of? b. *What was a picture of painted by Betsy? ∗ Current affiliation. This work was completed when the author was at New York University. 1Following terminological conventions in linguistics, a sentence’s grammaticality is determined by a grammatical formalism, and its acceptability is determined by introspec- tive judgments of native speakers (Sch¨utze, 1996). We train neural networks to perform accept- ability judgments—following work by Lawrence et al. (2000), Lau et al. (2016), and others—in order to evaluate their acquisition of the kinds of grammatical concepts linguists identify as central to human linguistic competence. This contributes to a growing effort to test ANNs’ ability to make fine-grained grammatical distinctions (Linzen et al., 2016; Adi et al., 2017; Conneau et al., 2018; Ettinger et al., 2018; Marvin and Linzen, 2018). This research programseeks to provide new informative ways to evaluate ANN models popular with engineers. Furthermore, it has the potential to address foundational questions in theoretical linguistics by investigating how well unbiased learners can acquire grammatical knowledge. In this paper we make four concrete contribu- tions: (i) We introduce the Corpus of Linguistic Acceptability (CoLA), a collection of sentences from the linguistics literature with expert accept- ability labels which, at over 10k examples, is by far the largest of its kind. (ii) We train several semi-supervised neural sequence models to do acceptability classification on CoLA and com- pare their performance with unsupervised models from Lau et al. (2016). Our best model outper- forms unsupervised baselines, but falls short of human performance on CoLA by a wide mar- gin. (iii) We analyze the impact of supervised training on …

Content truncated to 500 words