Investigating representations of verb bias in neural language models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4653–4663, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4653 Investigating representations of verb bias in neural language models Robert D. Hawkins∗1, Takateru Yamakoshi∗1,2, Thomas L. Griffiths1, Adele E. Goldberg1 1Princeton University, 2University of Tokyo {rdhawkins,takateru,tomg,adele}@princeton.edu Abstract Languages typically provide more than one grammatical construction to express certain types of messages. A speaker’s choice of con- struction is known to depend on multiple fac- tors, including the choice of main verb – a phenomenon known as verb bias. Here we in- troduce DAIS, a large benchmark dataset con- taining 50K human judgments for 5K distinct sentence pairs in the English dative alternation. This dataset includes 200 unique verbs and sys- tematically varies the definiteness and length of arguments. We use this dataset, as well as an existing corpus of naturally occurring data, to evaluate how well recent neural language models capture human preferences. Results show that larger models perform better than smaller models, and transformer architectures (e.g. GPT-2) tend to out-perform recurrent ar- chitectures (e.g. LSTMs) even under compa- rable parameter and training settings. Addi- tional analyses of internal feature representa- tions suggest that transformers may better in- tegrate specific lexical information with gram- matical constructions. 1 Introduction When we use language, we are often faced with a choice between several possible ways of express- ing the same message. For example, in English, to express an event of intended or actual trans- fer between two animate entities, one option is the double-object (DO) construction, in which two noun phrases follow the verb. Alternatively, the same content can be expressed using the preposi- tional dative (PO) construction. (1) a. Ava gave him something. DO b. Ava gave something to him. PO Speakers’ preferences for one or the other con- struction depend on multiple factors, including the length and definiteness of the arguments (Oehrle, 1976; Arnold et al., 2000; Wasow, 2002; Bresnan, 2007). One particularly subtle factor is the lexical verb bias. While some verbs readily occur in ei- ther construction, others have strong preferences for one over the other (Levin, 1993): (2) a. ?Ava said him something. DO b. Ava said something to him. PO Decades of work in linguistics and psychology has investigated how humans learn these distinc- tions (Gropen et al., 1989; Perfors et al., 2010; Barak et al., 2014; Goldberg, 2019). Yet, as deep neural networks have achieved state-of-the-art per- formance across many tasks in natural language processing, little is known about the extent to which they have acquired similarly fine-grained prefer- ences. Although neural language models robustly capture certain types of grammatical constraints, e.g., subject-verb agreement and long distance de- pendencies (Linzen and Baroni, 2021; Manning et al., 2020), they continue to struggle with other aspects of syntax, including argument structure (e.g. Warstadt et al., 2019). Verb biases provide a partic- ularly interesting testbed. Successfully predicting these psycholinguistic phenomena requires the in- tegration of specific lexical information with repre- sentations of higher-level grammatical structures, …

Content truncated to 500 words