Noisy-context surprisal as a human sentence processing cost model

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 688–698, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Noisy-context surprisal as a human sentence processing cost model Richard Futrell and Roger Levy Department of Brain and Cognitive Sciences Massachusetts Institute of Technology {futrell, rplevy}@mit.edu Abstract We use the noisy-channel theory of hu- man sentence comprehension to develop an incremental processing cost model that unifies and extends key features of expectation-based and memory-based models. In this model, which we call noisy-context surprisal, the processing cost of a word is the surprisal of the word given a noisy representation of the preceding context. We show that this model accounts for an outstand- ing puzzle in sentence comprehension, language-dependent structural forgetting effects (Gibson and Thomas, 1999; Va- sishth et al., 2010; Frank et al., 2016), which are previously not well modeled by either expectation-based or memory-based approaches. Additionally, we show that this model derives and generalizes local- ity effects (Gibson, 1998; Demberg and Keller, 2008), a signature prediction of memory-based models. We give corpus- based evidence for a key assumption in this derivation. 1 Introduction Models of human sentence processing difficulty can be divided into two kinds, expectation-based and memory-based. Expectation-based models predict the processing difficulty of a word from the word’s surprisal given previous material in the sentence (Hale, 2001; Levy, 2008a). These models have good coverage: they can account for effects of syntactic construction frequency and resolution of ambiguity on incremental processing difficulty. Memory-based models, on the other hand, explain difficulty resulting from working memory limita- tions during incremental parsing (Gibson, 1998; Lewis and Vasishth, 2005); a major prediction of these models is locality effects, where process- ing a word is difficult when it is far from other words with which it must be syntactically inte- grated. Expectation-based models do not intrin- sically capture this difficulty. Integrating these two approaches at a high level has proven challenging. A major hurdle is that the theories are typically stated at different lev- els of analysis: expectation-based theories are computational-level theories (Marr, 1982) specify- ing what computational problem the human sen- tence processing system is solving—the problem of how update one’s belief about a sentence given a new word—without specifying implementation details. Memory-based theories such as Lewis and Vasishth (2005) are for the most part based in mechanistic algorithmic-level theories describing the actions of a specific incremental parser. Previous theories that capture both surprisal and locality effects have typically done so by aug- menting parsing models with a special prediction- verification operation to capture surprisal effects (Demberg and Keller, 2009; Demberg et al., 2013), or by combining surprisal and memory- based cost derived from a parsing model as sep- arate factors in a linear model (Shain et al., 2016). These models capture surprisal and locality effects at the same time, but they do not clearly capture phenomena involving the interaction of memory and probabilistic expectations such as language- dependent structural forgetting …

Content truncated to 500 words