Lower Perplexity is Not Always Human-Like

Lower Perplexity is Not Always Human-Like Tatsuki Kuribayashi1,2, Yohei Oseki3,4, Takumi Ito1,2, Ryo Yoshida3, Masayuki Asahara5, Kentaro Inui1,4 1Tohoku University 2Langsmith Inc. 3University of Tokyo 4RIKEN 5NINJAL {kuribayashi, takumi.ito.c4, inui}@tohoku.ac.jp , {oseki, yoshiryo0617}@g.ecc.u-tokyo.ac.jp , masayu-a@ninjal.ac.jp Abstract In computational psycholinguistics, various language models have been evaluated against human reading behavior (e.g., eye movement) to build human-like computational models. However, most previous efforts have focused almost exclusively on English, despite the re- cent trend towards linguistic universal within the general community. In order to fill the gap, this paper investigates whether the established results in computational psycholinguistics can be generalized across languages. Specifically, we re-examine an established generalization —the lower perplexity a language model has, the more human-like the language model is— in Japanese with typologically different struc- tures from English. Our experiments demon- strate that this established generalization ex- hibits a surprising lack of universality; namely, lower perplexity is not always human-like. Moreover, this discrepancy between English and Japanese is further explored from the perspective of (non-)uniform information den- sity. Overall, our results suggest that a cross- lingual evaluation will be necessary to con- struct human-like computational models. 1 Introduction It is well known that the probability of a word in context (i.e., surprisal) impacts its processing difficulty in incremental human language compre- hension (Hale, 2001; Demberg and Keller, 2008; Levy, 2008; Smith and Levy, 2013). Building on this basis, researchers have compared a vari- ety of language models (LMs) in terms of how well their surprisal correlates with human reading be- havior (Roark et al., 2009; Frank and Bod, 2011; Fossum and Levy, 2012; Hale et al., 2018; Good- kind and Bicknell, 2018; Aurnhammer and Frank, 2019; Merkx and Frank, 2020; Wilcox et al., 2020). Such investigations could provide insights into the development of a general computational model of human language processing. For example, recent studies reported that LMs with better performance for next-word prediction could also better predict the human reading behavior (i.e. more human- like) (Fossum and Levy, 2012; Goodkind and Bick- nell, 2018; Wilcox et al., 2020). In this paper, we re-examine whether the re- cent findings on human-like computational mod- els can be generalized across languages. Despite the community’s ongoing search for a language- independent model (Bender, 2011), existing stud- ies have focused almost exclusively on the English language. Having said that, broad-coverage cross- linguistic evaluation of the existing reports is pro- hibitively difficult. In fact, data on human reading behavior (e.g., eye movement) is available only in limited languages. As an initial foray, this study focuses on the Japanese language as a representa- tive of languages that have typologically different characteristics from the English language. If the ob- servation is different between English and Japanese, the current findings on English data might lack a universality across languages. We specifically revisit the recent report—the lower perplexity a LM has, the more human-like the LM is—in the English and Japanese languages (Fos- sum and Levy, 2012; Goodkind and Bicknell, 2018; Wilcox et al., 2020). In addition to …

Content truncated to 500 words