Surprisal Estimators for Human Reading Times Need Character Models

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3746–3757 August 1–6, 2021. ©2021 Association for Computational Linguistics 3746 Surprisal Estimators for Human Reading Times Need Character Models Byung-Doh Oh Christian Clark Department of LinguisticsWilliam Schuler The Ohio State University {oh.531, clark.3664, schuler.77}@osu.edu Abstract While the use of character models has been popular in NLP applications, it has not been explored much in the context of psycholin- guistic modeling. This paper presents a char- acter model that can be applied to a struc- tural parser-based processing model to calcu- late word generation probabilities. Experimen- tal results show that surprisal estimates from a structural processing model using this char- acter model deliver substantially better fits to self-paced reading, eye-tracking, and fMRI data than those from large-scale language mod- els trained on much more data. This may sug- gest that the proposed processing model pro- vides a more humanlike account of sentence processing, which assumes a larger role of morphology, phonotactics, and orthographic complexity than was previously thought. 1 Introduction and Related Work Expectation-based theories of sentence processing (Hale, 2001; Levy, 2008) posit that processing dif- ficulty is determined by predictability in context. In support of this position, predictability quantified through surprisal has been shown to correlate with behavioral measures of word processing difficulty (Goodkind and Bicknell, 2018; Hale, 2001; Levy, 2008; Shain, 2019; Smith and Levy, 2013). How- ever, surprisal itself makes no representational as- sumptions about sentence processing, leaving open the question of how best to estimate its underlying probability model. In natural language processing (NLP) applica- tions, the use of character models has been popular for several years (Al-Rfou et al., 2019; Kim et al., 2016; Lee et al., 2017). Character models have been shown not only to alleviate problems with out-of-vocabulary words but also to embody mor- phological information available at the subword level. For this reason, they have been extensively used to model morphological processes (Elsner et al., 2019; Kann and Schütze, 2016) or incor- porate morphological information into models of syntactic acquisition (Jin et al., 2019). Nonethe- less, the use of character models has been slow to catch on in psycholinguistic surprisal estimation, which has recently focused on evaluating large- scale language models that make predictions at the word level (e.g. Futrell et al. 2019; Goodkind and Bicknell 2018; Hale et al. 2018; Hao et al. 2020). This raises the question of whether incorporating character-level information into an incremental pro- cessing model will result in surprisal estimates that better characterize predictability in context. To answer this question, this paper presents a character model that can be used to estimate word generation probabilities in a structural parser-based processing model.1 The proposed model defines a process of generating a word from an underlying lemma and a morphological rule, which allows the processing model to capture the predictability of a given word form in a fine-grained manner. Regres- sion analyses on self-paced reading, eye-tracking, and …

Content truncated to 500 words