Incremental Processing of Principle B: Mismatches Between Neural Models and Humans

Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 144 - 156 December 7-8, 2022 ©2022 Association for Computational Linguistics Incremental Processing of Principle B: Mismatches Between Neural Models and Humans Forrest Davis Department of Linguistics and Philosophy Massachusetts Institute of Technology forrestd@mit.edu Abstract Despite neural language models qualitatively capturing many human linguistic behaviors, re- cent work has demonstrated that they under- estimate the true processing costs of ungram- matical structures. We extend these more fine- grained comparisons between humans and mod- els by investigating the interaction between Principle B and coreference processing. While humans use Principle B to block certain struc- tural positions from affecting their incremental processing, we find that GPT-based language models are influenced by ungrammatical posi- tions. We conclude by relating the mismatch between neural models and humans to prop- erties of training data and suggest that certain aspects of human processing behavior do not directly follow from linguistic data. 1 Introduction Neural models trained on text data alone have been shown to qualitatively capture aspects of a large variety of human linguistic behaviors (e.g., Gulor- dava et al., 2018; Wilcox et al., 2019; Warstadt et al., 2020; Hu et al., 2020; Jumelet et al., 2021). Investigations have evaluated a range of levels of linguistic knowledge, including: i) syntax (Marvin and Linzen, 2018; Warstadt et al., 2019; Wilcox et al., 2019, 2021a), ii) semantics (Pannitto and Herbelot, 2020; Misra et al., 2020), and iii) dis- course structure and pragmatics (Schuster et al., 2020; Davis and van Schijndel, 2020). Recent work has placed increased attention on finer-grained comparisons between neural models and humans (e.g., van Schijndel and Linzen, 2021; Wilcox et al., 2021b; Paape and Vasishth, 2022). The growing consensus is that neural models un- derestimate the processing costs seen with humans, while nonetheless capturing the broad patterns (see Wilcox et al., 2021b). The present study adds to this literature by comparing the incremental processing of coreference in humans and neural models. While coreference, more generally, is modulated by discourse, pragmatics, and information struc- ture (e.g., Arnold, 1998, 2001; Rohde et al., 2006; Hartshorne, 2014; Rohde and Kehler, 2014), there are sentential restrictions on coreference that have immediate effects on human incremental process- ing (e.g., Nicol, 1988; Clifton et al., 1997; Sturt, 2003; Chow et al., 2014). This study finds that, con- trary to humans, autoregressive neural models do not similarly restrict their behavior in coreference processing. In particular, the present study investigated the interaction between the Binding Principles, articu- lated in Chomsky (1981), and incremental process- ing. Binding Principles account for the constrained distribution of pronouns (and anaphora) and their possible linguistic antecedents: (1) Binding Principles PRINCIPLE A An anaphor is bound in its governing category PRINCIPLE B A pronominal is free in its governing category PRINCIPLE C An R-expression is free Roughly, Principle A excludes examples like John thinks that Mike hates himself from meaning that “John thinks that Mike hates John”. Conversely, Principle B excludes examples like John thinks that Mike …

Content truncated to 500 words