Context Limitations Make Neural Language Models More Human-Like

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10421 - 10436 December 7-11, 2022 ©2022 Association for Computational Linguistics Context Limitations Make Neural Language Models More Human-Like Tatsuki Kuribayashi1,2 Yohei Oseki3,4 Ana Brassard1,4 Kentaro Inui1,4 1Tohoku University 2Langsmith Inc. 3University of Tokyo 4RIKEN {kuribayashi, inui}@tohoku.ac.jp oseki@g.ecc.u-tokyo.ac.jp ana.brassard@riken.jp Abstract Language models (LMs) have been used in cog- nitive modeling as well as engineering studies— they compute information-theoretic complexity metrics that simulate humans’ cognitive load during reading. This study highlights a lim- itation of modern neural LMs as the model of choice for this purpose: there is a discrep- ancy between their context access capacities and that of humans. Our results showed that constraining the LMs’ context access improved their simulation of human reading behavior. We also showed that LM-human gaps in context access were associated with specific syntactic constructions; incorporating syntactic biases into LMs’ context access might enhance their cognitive plausibility.1 1 Introduction In computational psycholinguistics, human read- ing behavior has been compared with various complexity metrics to understand human sentence processing (Crocker, 2007). Having historically started from simple measures such as word length, surprisal (− log p(word|context)) computed by language models (LMs) has become a common choice (Levy, 2008; Smith and Levy, 2013). On top of this, the next question arises—which model implementation and/or algorithm can compute sur- prisal that successfully simulates human behavior? In this line of research, modern neural LMs such as Transformer (Vaswani et al., 2017) have been analyzed with respect to their cognitive plausibil- ity (Wilcox et al., 2020; Merkx and Frank, 2021; Kuribayashi et al., 2021). Despite their use in cognitive modeling, such modern LM architectures (e.g., self-attention) are, arguably, an unnatural choice when it comes to human cognitive constraints; modern LM architec- tures assume powerful, parallel access to a vast 1Our codes are available at � https://github. com/kuribayashi4/context_limitation_cognitive_ modeling 2 3 5 7 10 20 full words input length LSTM-xs-Wiki GPT2-xs-Wiki GPT2-md-Wiki GPT2-sm GPT2-md GPT2-lg GPT2-xl 2 3 5 7 10 full words input length LSTM-xs-Wiki GPT2-xs-Wiki GPT2-md-Wiki English Japanese PPP PPP Figure 1: Relationship between psychometric predictive power (PPP) of language models (LMs) and their con- text access constraints. LMs with less context access better simulate human reading behavior (higher PPP). The marker color/shape indicates LM settings; colored areas present one standard deviation of PPP. number of context tokens, while humans might have limited and selective context access (Hawkins, 1994; Gibson, 1998, 2000; Lewis et al., 2006). Searching for a computational model that better simulates human sentence processing than previ- ously examined ones, we hypothesized that intro- ducing such context limitations can improve LMs’ estimation of human cognitive load. Specifically, as a starting point, we applied an n-gram-ification trick to neural LMs mimicking loading for long context access (locality effects) and compared their surprisal with human reading behavior data. Despite the simple context limita- tion design, our experiments with 280 settings (40 LM settings×7 noise patterns) showed that the ad- vantage of a shorter context was consistent among …

Content truncated to 500 words