A large-scale study of the effects of word frequency and predictability in naturalistic reading

Proceedings of NAACL-HLT 2019, pages 4086–4094 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 4086 A Large-Scale Study of the Effects of Word Frequency and Predictability in Naturalistic Reading Cory Shain Department of Linguistics The Ohio State University shain.3@osu.edu Abstract A number of psycholinguistic studies have fac- torially manipulated words’ contextual pre- dictabilities and corpus frequencies and shown separable effects of each on measures of hu- man sentence processing, a pattern which has been used to support distinct mechanisms un- derlying prediction on the one hand and lex- ical retrieval on the other. This paper exam- ines the generalizability of this finding to more realistic conditions of sentence processing by studying effects of frequency and predictabil- ity in three large-scale naturalistic reading cor- pora. Results show significant effects of word frequency and predictability in isolation but no effect of frequency over and above predictabil- ity, and thus do not provide evidence of dis- tinct mechanisms. The non-replication of sep- arable effects in a naturalistic setting raises doubts about the existence of such a distinc- tion in everyday sentence comprehension. In- stead, these results are consistent with previ- ous claims that apparent effects of frequency are underlyingly effects of predictability. 1 Introduction Are there distinct effects of a word’s frequency versus predictability in human sentence compre- hension? Recent evidence implicates prediction as a major organizing principle in cognition (Bubic et al., 2010; Singer et al., 2018; Keller and Mrsic- Flogel, 2018), and psycholinguists have long stud- ied the role of prediction in human sentence pro- cessing and its relation to other comprehension mechanisms (Marslen-Wilson, 1975; Kutas and Hillyard, 1984; MacDonald et al., 1994; Tanen- haus et al., 1995; Hale, 2001; Norris, 2006; Levy, 2008; Frank and Bod, 2011). Some prominent theories of word recognition claim that ease of lexical access is modulated by the strength of a word’s representation in memory, independently of contextual factors that guide prediction (Seiden- berg and McClelland, 1989; Coltheart et al., 2001; Harm and Seidenberg, 2004). Other theories hold that apparent effects of frequency are underlyingly effects of predictability (Norris, 2006; Levy, 2008; Rasmussen and Schuler, 2018). A number of studies using constructed stim- uli that factorially manipulate word frequency and predictability have found separable additive ef- fects of each, suggesting distinct influences on lexical processing (see Staub, 2015 for a re- view). This paper examines the generalizability of these findings to typical sentence comprehension by searching for separable effects of frequency and n-gram predictability using deconvolutional time series regression (DTSR) models (Shain and Schuler, 2018) fitted to three large naturalistic reading corpora: Natural Stories (Futrell et al., 2018), Dundee (Kennedy et al., 2003), and UCL (Frank et al., 2013). While results show evidence of both frequency and predictability effects in iso- lation, they show no effect of frequency over pre- dictability and thus do not support the existence of separable effects. They are instead consistent with either (1) an account of apparent frequency effects as epiphenomena of predictive processing (Norris, …

Content truncated to 500 words