Quantity doesn’t buy quality syntax with neural language models

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5831–5837, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5831 Quantity doesn’t buy quality syntax with neural language models Marten van Schijndel Cornell University mv443@cornell.edu Aaron Mueller Johns Hopkins University amueller@jhu.edu Tal Linzen Johns Hopkins University tal.linzen@jhu.edu Abstract Recurrent neural networks can learn to predict upcoming words remarkably well on average; in syntactically complex contexts, however, they often assign unexpectedly high probabil- ities to ungrammatical words. We investigate to what extent these shortcomings can be miti- gated by increasing the size of the network and the corpus on which it is trained. We find that gains from increasing network size are min- imal beyond a certain point. Likewise, ex- panding the training corpus yields diminishing returns; we estimate that the training corpus would need to be unrealistically large for the models to match human performance. A com- parison to GPT and BERT, Transformer-based models trained on billions of words, reveals that these models perform even more poorly than our LSTMs in some constructions. Our results make the case for more data efficient architectures. 1 Introduction Recurrent neural network language models (LMs) can learn to predict upcoming words with remark- ably low perplexity (Mikolov et al., 2010; Joze- fowicz et al., 2016; Radford et al., 2019). This overall success has motivated targeted paradigms that measure whether the LM’s predictions reflect a correct analysis of sentence structure. One such evaluation strategy compares the probability as- signed by the LM to a minimal pair of sentences differing only in grammaticality (Linzen et al., 2016). In the following example, the LM is ex- pected to assign a higher probability to the sen- tence when the verb agrees in number with the subject (1a) than when it does not (1b): (1) a. The author laughs. b.*The author laugh. RNN LMs have been shown to favor the grammat- ical variant in the vast majority of cases sampled at random from a corpus (Linzen et al., 2016), but their accuracy decreases in the presence of dis- tracting nouns intervening between the head of the subject and the verb, especially when those nouns are in relative clauses (Marvin and Linzen, 2018). Can we hope to address these deficits by train- ing larger and larger networks on larger and larger corpora, relying on the “unreasonable effective- ness” of massive datasets (Halevy et al., 2009) and computational power?1 Or would architectural ad- vances be necessary to improve our LMs’ syntac- tic representations (Kuncoro et al., 2018)? This paper takes a first step towards address- ing this question. We train 125 RNN LMs with long short-term memory (LSTM, Hochreiter and Schmidhuber, 1997) units, systematically varying the size of the training corpus and the dimension- ality of the models’ hidden layer, and track the re- lationship between these parameters and the per- formance of the models on agreement dependen- cies in a range of syntactic constructions …

Content truncated to 500 words