Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1979–1990 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1979 Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment Forrest Davis and Marten van Schijndel Department of Linguistics Cornell University {fd252|mv443}@cornell.edu Abstract A standard approach to evaluating language models analyzes how models assign proba- bilities to valid versus invalid syntactic con- structions (i.e. is a grammatical sentence more probable than an ungrammatical sentence). Our work uses ambiguous relative clause at- tachment to extend such evaluations to cases of multiple simultaneous valid interpretations, where stark grammaticality differences are ab- sent. We compare model performance in En- glish and Spanish to show that non-linguistic biases in RNN LMs advantageously overlap with syntactic structure in English but not Spanish. Thus, English models may appear to acquire human-like syntactic preferences, while models trained on Spanish fail to acquire comparable human-like preferences. We con- clude by relating these results to broader con- cerns about the relationship between compre- hension (i.e. typical language model use cases) and production (which generates the training data for language models), suggesting that nec- essary linguistic biases are not present in the training signal at all. 1 Introduction Language modeling is widely used as pretraining for many tasks involving language processing (Pe- ters et al., 2018; Radford et al., 2018; Devlin et al., 2019). Since such pretraining affects so many tasks, effective evaluations to assess model quality are critical. Researchers in the vein of the present study, typically take (pretrained) language models and ask whether those models have learned some linguistic phenomenon (e.g., subject-verb agreement). Of- ten the task is operationalized as: do the models match some human baseline (e.g., acceptability judgments, reading times, comprehension ques- tions) measured as humans experience this linguis- tic phenomenon (e.g., comparing acceptability rat- ings of sentences with grammatical/ungrammatical agreement). This approach tacitly assumes that the necessary linguistic biases are in the training signal and then asks whether the models learn the same abstract representations as humans given this sig- nal. The present study casts doubt on the notion that the necessary linguistic biases are present in the training signal at all. We utilize the, now common, evaluation tech- nique of checking whether a model assigns higher probability to grammatical sentences compared to ungrammatical sentences (Linzen et al., 2016). However, we extend beyond binary grammaticality. Real world applications demand that our models not only know the difference between valid and in- valid sentences; they must also be able to correctly prioritize simultaneous valid interpretations (Lau et al., 2017). In this paper, we investigate whether neural networks can in fact prioritize simultaneous interpretations in a human-like way. In particular, we probe the biases of neural networks for ambigu- ous relative clause (RC) attachments, such as the following: (1) Andrew had dinner yesterday with the nephew of the teacher that was divorced. (from Fern´andez, 2003) In (1), there are two nominals (nephew and teacher) that are available for modification by the …

Content truncated to 500 words