Do pretrained transformers infer telicity like humans?

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 72–81 November 10–11, 2021. ©2021 Association for Computational Linguistics 72 Do pretrained transformers infer telicity like humans? Yiyun Zhao, Jian Gang Ngui, Lucy Hall Hartley University of Arizona Department of Linguistics Tucson, AZ 85721, USA {yiyunzhao,jgngui,lucyhallhartley,bethard}@email.arizona.edu Steven Bethard University of Arizona School of Information Tucson, AZ 85721, USA Abstract Pretrained transformer-based language models achieve state-of-the-art performance in many NLP tasks, but it is an open question whether the knowledge acquired by the models dur- ing pretraining resembles the linguistic knowl- edge of humans. We present both humans and pretrained transformers with descriptions of events, and measure their preference for telic interpretations (the event has a natural endpoint) or atelic interpretations (the event does not have a natural endpoint). To mea- sure these preferences and determine what fac- tors influence them, we design an English test and a novel-word test that include a va- riety of linguistic cues (noun phrase quan- tity, resultative structure, contextual informa- tion, temporal units) that bias toward certain interpretations. We find that humans’ choice of telicity interpretation is reliably influenced by theoretically-motivated cues, transformer models (BERT and RoBERTa) are influenced by some (though not all) of the cues, and trans- former models often rely more heavily on tem- poral units than humans do. 1 Introduction Large pretrained-language models (ELMo: Peters et al., 2018a, BERT: Devlin et al., 2019, RoBERTa Liu et al., 2019, etc.) keep achieving new states of the art in a variety of NLP tasks, leading to a grow- ing interest in exploring what has been acquired by the pretraining objectives. Many recent works utilize probes: shallow, usu- ally supervised classifiers that try to determine which linguistic phenomena are predictable from the pretrained representations. The linguistic prop- erties studied include syntactic relationships (He- witt and Manning, 2019), morphological informa- tion (Belinkov et al., 2017), and semantic knowl- edge and entailment (Peters et al., 2018b; Good- win et al., 2020), and the representations studied include sentence embeddings (Adi et al., 2017; Conneau et al., 2018; Ettinger et al., 2018) and token/word-level embeddings (Kim et al., 2019; Ethayarajh, 2019). Recent work has also explored unsupervised analysis of attention heads instead of supervised training (Clark et al., 2019; Kovaleva et al., 2019; Zhao and Bethard, 2020). Another approach to understanding pretrained language models is to test their behavior on psy- cholinguistic tasks. Stimuli in psycholinguistic tasks are typically designed to reveal linguistic bias in human behaviors (e.g., grammatical judgments, reading speeds, neural responses). Evaluating pre- trained language models on such tasks can thus pro- vide insights on the linguistic biases acquired by the models. The linguistic properties studied include subject-verb agreement (Linzen et al., 2016; Gulor- dava et al., 2018), filler-gap depedencies (Wilcox et al., 2018), garden-path effects (Futrell et al., 2019), other types of syntactic awareness (Marvin and Linzen, 2018; Hu et al., 2020), and variations of grammatical judgment based on availability of contexts (Lau et al., 2020). Most works focus on syntactic knowledge …

Content truncated to 500 words