Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities

Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 301 - 313 December 7-8, 2022 ©2022 Association for Computational Linguistics Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities Suhas Arehalli Johns Hopkins University suhas@jhu.edu Brian Dillon University of Massachusetts, Amherst brian@linguist.umass.edu Tal Linzen New York University linzen@nyu.edu Abstract Humans exhibit garden path effects: When reading sentences that are temporarily structurally ambigu- ous, they slow down when the structure is disam- biguated in favor of the less preferred alternative. Surprisal theory (Hale, 2001; Levy, 2008), a promi- nent explanation of this finding, proposes that these slowdowns are due to the unpredictability of each of the words that occur in these sentences. Chal- lenging this hypothesis, van Schijndel and Linzen (2021) find that estimates of the cost of word pre- dictability derived from language models severely underestimate the magnitude of human garden path effects. In this work, we consider whether this un- derestimation is due to the fact that humans weight syntactic factors in their predictions more highly than language models do. We propose a method for estimating syntactic predictability from a language model, allowing us to weigh the cost of lexical and syntactic predictability independently. We find that treating syntactic predictability independently from lexical predictability indeed results in larger esti- mates of garden path. At the same time, even when syntactic predictability is independently weighted, surprisal still greatly underestimate the magnitude of human garden path effects. Our results support the hypothesis that predictability is not the only fac- tor responsible for the processing cost associated with garden path sentences. 1 Introduction Readers exhibit garden path effects: When reading a temporarily syntactically ambiguous sentence, they tend to slow down when the sentence is dis- ambiguated in favor of the less preferred parse. For example, a participant who reads the sentence frag- ment (1) The suspect sent the file . . . a. . ..to the lawyer. b. . ..deserved further investigation. can construct a partial parse in at least two dis- tinct ways: In one reading, the verb sent acts as the main verb of the sentence, and the continua- tion of the sentence as an additional argument to sent (as in 1a). In another, less likely, reading, sent the file acts as a modifier in a complex subject, which then requires an additional verb phrase to form a complete sentence (as in 1b). Prior work has demonstrated that regions like deserved further investigation, which disambiguate these temporar- ily ambiguous sentences in favor of the modifier parse (1b), are read slower than those same words would be in an unambiguous version of sentence, such as the following: (2) The suspect who was sent the file deserved further investigation. In (2), the presence of who was signals to the reader that sent the file acts as a modifier (Frazier and Fodor, 1978). One account of this phenomenon, surprisal the- ory (Hale, 2001; Levy, 2008), suggests that read- ers maintain a probabilistic representation of all …

Content truncated to 500 words