A Systematic Assessment of Syntactic Generalization in Neural Language Models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1725 A Systematic Assessment of Syntactic Generalization in Neural Language Models Jennifer Hu1, Jon Gauthier1, Peng Qian1, Ethan Wilcox2, and Roger P. Levy1 1Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology 2Department of Linguistics, Harvard University {jennhu,pqian,rplevy}@mit.edu jon@gauthiers.net, wilcoxeg@g.harvard.edu Abstract While state-of-the-art neural network models continue to achieve lower perplexity scores on language modeling benchmarks, it remains un- known whether optimizing for broad-coverage predictive performance leads to human-like syntactic knowledge. Furthermore, existing work has not provided a clear picture about the model properties required to produce proper syntactic generalizations. We present a sys- tematic evaluation of the syntactic knowledge of neural language models, testing 20 com- binations of model types and data sizes on a set of 34 English-language syntactic test suites. We find substantial differences in syn- tactic generalization performance by model ar- chitecture, with sequential models underper- forming other architectures. Factorially manip- ulating model architecture and training dataset size (1M–40M words), we find that variabil- ity in syntactic generalization performance is substantially greater by architecture than by dataset size for the corpora tested in our ex- periments. Our results also reveal a dissocia- tion between perplexity and syntactic general- ization performance. 1 Introduction A growing body of work advocates that assess- ment of neural language models should include both information-theoretic metrics, such as per- plexity, as well as targeted linguistic evaluation. Benchmarks such as GLUE (Wang et al., 2019a,b) have demonstrated that neural language models trained on naturalistic corpora for next-word predic- tion learn representations that can yield remarkable performance on many semantic tasks. Targeted syntactic evaluations have shown that these mod- els also implicitly capture many syntactic gener- alizations, ranging from subject–verb agreement Materials and code can be found at https://github. com/cpllab/syntactic-generalization. to long-distance filler–gap dependencies (Linzen et al., 2016; Marvin and Linzen, 2018; Futrell et al., 2018; Wilcox et al., 2019b). This paper aims to bring targeted evaluations of syntactic performance to scale, complementing similar developments in semantic evaluation (McCoy et al., 2019). Because the most widespread currency of evalu- ation for language models is perplexity—how well, on average, a model predicts a word in its context— a primary focus of this paper is the relationship between a model’s perplexity and its performance on targeted syntactic evaluations. As perplexity im- proves, can we expect more human-like syntactic generalization? How do training dataset size and model architecture jointly affect syntactic gener- alization? And what picture of models’ syntactic generalization emerges when evaluation is brought to scale, across dozens of controlled syntactic tests? In this paper we offer initial answers to these questions, systematically assessing the syntactic generalization abilities of neural language models on 34 targeted test suites (33 adapted from pre- viously published work, and 1 novel) covering a wide range of syntactic phenomena. Test suites are written using a standard format that allows for flexible predictions which more closely resemble …

Content truncated to 500 words