Refining Targeted Syntactic Evaluation of Language Models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3710–3723 June 6–11, 2021. ©2021 Association for Computational Linguistics 3710 Refining Targeted Syntactic Evaluation of Language Models Benjamin Newman Kai-Siang Ang Julia Gong John Hewitt Department of Computer Science Stanford University {blnewman,kaiang,jxgong,johnhew}@cs.stanford.edu Abstract Targeted syntactic evaluation of subject-verb number agreement in English (TSE) evaluates language models’ syntactic knowledge using hand-crafted minimal pairs of sentences that differ only in the main verb’s conjugation. The method evaluates whether language models rate each grammatical sentence as more likely than its ungrammatical counterpart. We iden- tify two distinct goals for TSE. First, evalu- ating the systematicity of a language model’s syntactic knowledge: given a sentence, can it conjugate arbitrary verbs correctly? Second, evaluating a model’s likely behavior: given a sentence, does the model concentrate its proba- bility mass on correctly conjugated verbs, even if only on a subset of the possible verbs? We argue that current implementations of TSE do not directly capture either of these goals, and propose new metrics to capture each goal sep- arately. Under our metrics, we find that TSE overestimates systematicity of language mod- els, but that models score up to 40% better on verbs that they predict are likely in context. 1 Introduction As neural language models have emerged as both broadly useful engineering tools (Devlin et al., 2018; Radford et al., 2019) and potential models of human language processing (Linzen and Leonard, 2018; Ettinger et al., 2018; Futrell et al., 2019), evaluations targeting their syntactic ability have been developed to better understand their capabili- ties. One such method for syntactic evaluation tests models’ knowledge of English subject-verb (S/V) number agreement (Linzen et al., 2016; Gulordava et al., 2018). These studies consider minimal pairs of sentences, such as The keys to the cabinet is/are on the table, that differ only in verb number, and test if models rate grammatical sentences as more probable. The syntactically correct of the two sen- tences is sampled from natural corpora (Linzen et al., 2016; Kuncoro et al., 2018) or constructed from templates. The use of templates, known as Targeted Syntactic Evaluation (TSE), allows for the fine-grained evaluation of models on specific, of- ten rare, syntactic phenomena (Marvin and Linzen, 2018; Ettinger et al., 2018; Warstadt et al., 2020), but (when evaluating S/V number agreement) relies on researchers hand-specifying a small set of verb lemmas that are substituted into each template. In this work, we improve the TSE methodol- ogy by disentangling its broad objective of evalu- ating syntactic ability into two distinct goals, and we introduce two variants of TSE to separately capture each goal. These evaluations demonstrate that neural models do not generally consider well- conjugated verbs more likely than their incorrect conjugations, but instead prefer to correctly conju- gate verbs they deem likely. We argue that the objective of evaluating syn- tactic ability can be decomposed into two goals and that current implementations of TSE do not achieve either of …

Content truncated to 500 words