The Natural Stories Corpus

The Natural Stories Corpus Richard Futrell1, Edward Gibson1, Harry J. Tily2, Idan Blank1, Anastasia Vishnevetsky1, Steven T. Piantadosi3, and Evelina Fedorenko4,5 1MIT Department of Brain and Cognitive Sciences 2Netflix, Inc. 3University of Rochester Department of Brain and Cognitive Sciences 4Massachusetts General Hospital Department of Psychiatry 5Harvard Medical School Department of Psychiatry {futrell, egibson, iblank, evelina9}@mit.edu, hal.tily@gmail.com, staseyvi@mail.med.upenn.edu Abstract It is now a common practice to compare models of human language processing by comparing how well they predict behavioral and neural measures of processing difficulty, such as reading times, on corpora of rich naturalistic linguistic materials. However, many of these corpora, which are based on naturally-occurring text, do not contain many of the low-frequency syntactic constructions that are often required to distinguish between processing theories. Here we describe a new corpus consisting of English texts edited to contain many low-frequency syntactic constructions while still sounding fluent to native speakers. The corpus is annotated with hand-corrected Penn Treebank-style parse trees and includes self-paced reading time data and aligned audio recordings. Here we give an overview of the content of the corpus and release the data. Keywords: Cognitive modeling, reading time, psycholinguistics 1. Introduction It is becoming a standard practice to evaluate theories of human language processing by comparing their ability to predict behavioral and neural reactions to fixed standard- ized corpora of naturalistic text. This method has been used to study several dependent variables which are believed to be indicative of human language processing difficulty, in- cluding word fixation time in eyetracking (Kennedy et al., 2013), word reaction time in self-paced reading (Roark et al., 2009; Frank et al., 2013), BOLD signal in fMRI data (Bachrach et al., 2009), and event-related potentials (Dambacher et al., 2006; Frank et al., 2015). The more traditional approach to evaluating psycholinguis- tic models has been to collect psychometric measures on hand-crafted experimental stimuli designed to tease apart detailed model predictions. While this approach makes it easy to compare models on their accuracy for specific con- structions and phenomena, it is hard to get a sense of how models compare on their coverage of a broad range of phe- nomena. Comparing model predictions over standardized texts makes it is easier to evaluate coverage. Although the corpus approach has these advantages, the existing corpora currently used are based on naturally- occurring text, which is unlikely to include the kinds of sentences which can crucially distinguish between theories. Many of the most puzzling phenomena in psycholinguis- tics, and the phenomena which have been used to test mod- els, have only been observed in extremely rare construc- tions, such as multiply nested relative clauses. Corpora of naturally-occurring text are unlikely to contain these con- structions. More generally, models of human language comprehension are more likely to make distinct predictions for sentences that cause difficulty for humans, rather than for sentences that are easy to process. For instance, mod- els of comprehension difficulty based on memory integra- tion cost during parsing (Gibson, 2000; Lewis and Vasishth, 2005) will predict effects when …

Content truncated to 500 words